Is anyone having trouble stopping an in-progress p...
# troubleshootingc
connor_flynn
02/07/2022, 6:04 PMIs anyone having trouble stopping an in-progress pipeline?
Our usual methods of running “pgrep -f <pipeline_name>” or “ps aux” to identify the PID associated with that pipeline and then killing the PID aren't working.
Thanks in advance!
connor_flynn
02/07/2022, 6:04 PM@edward_ryan @jo_pearson
a
aaronsteers
02/07/2022, 11:45 PMPasting additional context from other thread:
It's several days behind and we are urgently trying to backfill it.
We've tried to :
1. Run ps aux to find the PIDs associated with the problematic pipeline and kill the PIDs
2. Run pgrep -f <pipeline_name> to output the PIDs for the pipeline and then run a kill command
3. Reset the UI
#1 or #2 has always worked in the past, but neither are outputting the associated PID(s).
Are there other ways to kill a pipeline that I'm not think of?
aaronsteers
02/07/2022, 11:45 PMFrom @florian.hines :
Do you have access to any logs, like the meltano elt log file or meltano output, that you can share ?
If you're not see'ing any pid's i'd venture its possible that its crashed or the like.
aaronsteers
02/07/2022, 11:47 PM@connor_flynn and @jo_pearson - Just to confirm on the above point. Are you able to confirm that the EL process is indeed still running and not orphaned? When you mentioned that the process of killing the PID, I'm not sure from context if the PID is found but not killable or not found at all.
c
connor_flynn
02/08/2022, 12:01 AMThank you @aaronsteers !
I will need to confirm with @jo_pearson regarding the status of the EL process and current logs.
j
jo_pearson
02/08/2022, 12:08 AM@aaronsteers the PID is not found at all. The log has not changed in the past 7 hours. Is that the same thing as being orphaned?
a
aaronsteers
02/08/2022, 12:17 AMYeah - there are different things that can happen here, but it sounds like the process may have already been killed/aborted but perhaps the log did not get updated. Could be that the job is already killed but from a logging perspective it appears as if the job is still running.
aaronsteers
02/08/2022, 12:19 AMI wonder if this is worth trying (or if you've already tried it), from the docs...
• Aflag can be passed to force a new run even when a pipeline with the same Job ID is already running, which would result in an error otherwise.--force
j
jo_pearson
02/08/2022, 3:00 AMHaven't tried this yet but I will, I wanted to make sure it actually wasn't running. Thanks @aaronsteers!