I am having this issue where a certain pipeline shows as running, but has no logs and isn't doing anything. It shows it is running in the Meltano UI and in the Airflow UI. I have tried deleting the dag and pipeline and any logs associated, however, when I recreate the scheduled pipeline, both UI's just repopulate what was there before and continue to show it as running. I looked through the issue board and it doesn't seem like there is currently a way to cancel or end a pipelines run.

I just ran into this and got around it by manually deleting the job record in the meltano DB. More context here: https://meltano.slack.com/archives/C01TCRBBJD7/p1620654893103500?thread_ts=1620221119.064500&cid=C01TCRBBJD7

Stale "running" jobs in Meltano (rows in the

job

table in the system database) should automatically be marked as "failed" after 5 minutes: https://gitlab.com/meltano/meltano/-/merge_requests/2000

It wasn't ending after 5 minutes (had been over a day), but @nick_hamlin’s solution seemed to fix the issue! Thank you! The only other thing I was curious about was if Meltano supports multiple pipelines / dags to be run at the same time. I assumed yes and that this would be more related to Airflow. However, I wanted to quickly ask just because I don't want to mess around with anything and break any integrations if it is not yet supported.

It wasn't ending after 5 minutes (had been over a day), but @nick_hamlin’s solution seemed to fix the issue!

I would've loved to see the

job

record before you deleted it so that I could figure out why it wasn't getting marked stale after 5 min 😄

but they must have unique

job_id

values (this was part of my problem)

@nick_hamlin That'd be great, please do.

well, I may have spoken too soon - I’m following the same steps I was over the weekend, but (at least for now), nothing seems to be getting stuck anymore!

Ah sorry! I think I still have the printout in my console, I can check in a second. However, I think the issue was because of multiple changes I made to the project and re-creating pipelines with the same setup and naming convention.

I'm running into the same issue using 1.76. Here's the list of running jobs from the

job

table:

meltano=> select id, run_id, state, started_at, ended_at, trigger, last_heartbeat_at from job where state ='RUNNING';
 id |                run_id                |  state  |         started_at         | ended_at | trigger |     last_heartbeat_at      
----+--------------------------------------+---------+----------------------------+----------+---------+----------------------------
 21 | 6476cd38-f490-4df5-b3e8-e45f1ba324da | RUNNING | 2021-06-18 01:50:24.108557 |          | ui      | 2021-06-18 01:58:49.481304
 20 | 017c90f8-696e-4b6c-8c75-063c0b1e34dc | RUNNING | 2021-06-18 01:50:23.99533  |          | ui      | 2021-06-18 01:58:50.118105
 22 | b3b2d854-acaa-40e5-9888-7c3115126321 | RUNNING | 2021-06-18 01:50:24.231653 |          | ui      | 2021-06-18 01:58:50.224987

@casey If you run

meltano schedule list

, do you see anything logged about stale jobs being marked as such?

Here's the output:

root@ledp-vm:/project# meltano schedule list
[@once] ip-hash: tap-postgres--telemetry → pipelinewise-target-bigquery--telemetry → transforms
[@once] kdp-el: tap-postgres--kdp → pipelinewise-target-bigquery--kdp → transforms
[@once] studio-el: tap-postgres--studio → pipelinewise-target-bigquery--studio → transforms

@casey OK, let’s debug this a little. Can you run

meltano repl

and then:

from meltano.core.job.job_finder import JobFinder
JobFinder.all_stale()

Hmm, it can't find that module.

root@ledp-vm:/project# meltano repl
Python 3.7.10 (default, May 12 2021, 16:05:48) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.24.1 -- An enhanced Interactive Python. Type '?' for help.

Booting import Meltano REPL


In [1]: from meltano.core.job.job_finder import JobFinder
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-d78d2d414e8f> in <module>
----> 1 from meltano.core.job.job_finder import JobFinder

ModuleNotFoundError: No module named 'meltano.core.job.job_finder'

Ah yes, my bad

all_stale()

takes a

session

parameter... what should I provide it with?

That’s what I get for suggesting some code to run without testing it 🙂

hahaha... no worries

Ok, that’s what I’d expect. Let’s see if that query returns any rows:

list(JobFinder.all_stale(session))

Nope...

In [10]: list(JobFinder.all_stale(session))
Out[10]: []

In [11]:

All right, now that is odd!

for reasons I don't understand, Slack won't let me send the results in a message. Let me try and put them in a snippet.

Hmm, so now it can find the stale jobs, but it couldn’t from

JobFinder

do I need to import a module that declares/defines

HEARTBEAT_VALID_MINUTES

?

If you enter those values into the query in https://meltano.slack.com/archives/C01TCRBBJD7/p1624292529338800?thread_ts=1620661818.107900&cid=C01TCRBBJD7, do you get 0 records or more?

This query:

SELECT job.id AS job_id_1, job.job_id AS job_job_id, job.run_id AS job_run_id, job.state AS job_state, job.started_at AS job_started_at, job.last_heartbeat_at AS job_last_heartbeat_at, job.ended_at AS job_ended_at, job.payload AS job_payload, job.payload_flags AS job_payload_flags, job.trigger AS job_trigger FROM job WHERE job.state = 'RUNNING' AND (job.last_heartbeat_at IS NOT NULL AND job.last_heartbeat_at < '2021-06-21 16:50:07.654660' OR job.last_heartbeat_at IS NULL AND job.started_at < '2021-06-20 16:55:07.654660')

returns 3 records

Ok. But when run from Python, SQLAlchemy found 0 😕

Sure

@casey Thanks a lot, I’ve called in @aaronsteers to help debug this. For debug purposes, it would help if you left the DB in the current (broken) state, but to solve the issue you can manually change the state on these jobs from

RUNNING

to

FAIL

.

I'm sorry man. I just deleted the jobs entirely. I did, however, run a sql export of the database before doing so. I'll attach it to the ticket.

No worries, thanks!