Hello all!
I am running a Meltano pipeline into R...
# troubleshootingg
gunnar
07/21/2021, 10:09 PMHello all!
I am running a Meltano pipeline into Redshift and it has been running for days now, however, a couple days ago it stopped loading data in. No errors have risen, but it seems as though it's been stalled for some time now.
This pipeline was run through the Meltano UI. Is there a way to stop this pipeline manually and rerun it, or does anyone have any solutions for how I can resolve this issue?
It shows on the Meltano UI has "Running..." and the logs show no sign of error.
d
douwe_maan
07/21/2021, 10:10 PMYou may be seeing https://gitlab.com/meltano/meltano/-/issues/2812
douwe_maan
07/21/2021, 10:11 PMIf you go into the system database, and manually change the
stateRUNNINGFAILg
gunnar
07/21/2021, 10:13 PMI will try that. Thanks for the information
gunnar
07/21/2021, 10:22 PMIt seemed to stop the pipeline, but when I run the pipeline again it does not pick up from its saved state.
I will try manually updating the state now from the last saved
gunnar
07/21/2021, 10:29 PMhm, doesn't seem to be working
gunnar
07/21/2021, 10:30 PM@douwe_maan, do you know how I can still continue the pipeline? Seems as though it keeps starting from the initial start_date now, rather than the state that was saved
d
douwe_maan
07/21/2021, 10:30 PMCan you run
meltano elt … --job_id=<the right job_id> --dump=statedouwe_maan
07/21/2021, 10:30 PMIt should still find state in the system DB, unless you deleted
jobg
gunnar
07/21/2021, 10:37 PMI only ran an update query to change the state from Running... to FAIL
gunnar
07/21/2021, 10:39 PMCould not find state file for this pipelinegunnar
07/21/2021, 11:00 PMAny suggestion?
It also adjusted the UI to show that its never run rather than the time since it terminated
gunnar
07/21/2021, 11:01 PM@douwe_maan
d
douwe_maan
07/21/2021, 11:03 PMCan you show me the
jobg
gunnar
07/21/2021, 11:07 PM Copy code
1051|klaviyo--x|FAIL|2021-07-10 18:51:18.168493||{"singer_state": {"bookmarks": {"subscribe_list": {"since": "2021-02-07T20:10:41Z"}}}}|1|removed|ui|2021-07-11 15:40:29.753572
1250|klaviyo--x|FAIL|2021-07-21 22:27:27.656360||{"singer_state": {"bookmarks": {"subscribe_list": {"since": "2021-02-07T20:10:41Z"}}}}|0|removed|ui|2021-07-21 22:37:11.005708
1251|klaviyo--x|FAIL|2021-07-21 22:28:36.025408||{}|0|removed|ui|2021-07-21 22:37:10.757977gunnar
07/21/2021, 11:07 PMI assume a new row for the job_id is created on each run though, so I assume that is unrelated (number of rows going up)
gunnar
07/21/2021, 11:17 PM@douwe_maan, do you think I should just reset the pipeline and table?
Or is there a way to resolve this state issue?
d
douwe_maan
07/21/2021, 11:32 PMSorry for the delay, was in a call
douwe_maan
07/21/2021, 11:32 PMCan you remove the last row you marked as
FAILdouwe_maan
07/21/2021, 11:32 PMYou’ll want the new run to pick up the state of the last run that actually had some state:
Copy code
1250|klaviyo--x|FAIL|2021-07-21 22:27:27.656360||{"singer_state": {"bookmarks": {"subscribe_list": {"since": "2021-02-07T20:10:41Z"}}}}|0|removed|ui|2021-07-21 22:37:11.005708