Re

Another pipeline is already running which started at 2024-10-10 03:02:10.731573. To ignore this check use the '--force' option.

How long do I have to wait for this job to time out normally? I am running in docker, but not using an external db for my meltano metadata, just the default local sqlite instance (I know, I know). So I can't actually access the db to mark this pipeline run complete. I'm also executing my runs via dagster so I don't have cmd access on my container, or a way to execute a run with the

--force

flag.

I'd modify the dagster job to run with --force myself. You could hop into whereever the sqllite db and connect to it as well it's just in

.meltano/

I use meltano

jobs

where tap and target are pre-specified I can't specify a flag unfortunately, I need to change that. So I use

meltano run facebook

as my command. Also with Azure's amazing implementation in App Service, if you have your own custom container image you can't ssh into it without a tremendous amount of extra setup 🤷‍♂️ Lesson learnt - use a proper db in production for the metadata. But I do appreciate the response, honest 🙂

you can modify the run command to just always run with --force is my point but 🤷

I didn't think that was supported for jobs, but now you mention it that makes sense. You can include

--force

to a task in your job definition, but you can't do

meltano run myjob --force

to modify a job with a flag, because how would meltano know which task to force? Will update, good shout. I have retries on my dagster job, but if 1st one fails for some reason, then retries fail due to the meltano block. That should fix it.