Had some issues with

meltano install .

today. Maybe just the way I'm deploying the jobs right now, but here we go. I have multiple processes running using the same meltano directory. Something like 25 taps/targets in this meltano project (lots of inheritance). When I ran

meltano install . - 4

I had one of the taps/targets get an Access Denied (Another process was using the files in the venv that was getting updated). This lead to a state where that tap/target was not runnable. Maybe this was a

rm -rf tapdirectory/

action? Ideally Meltano would be aware of what's running and not clober the venv for that job unless you told it to try with -force or something. I know the recommendation is to run docker containers which side steps these problems, but I don't have that option today. It's looking like running symlinks on a production directory and stepping in a new version of my Meltano project would be the way to go, but curious about what you guys think about making meltano install not break things if something is running at the same time

So you’re running

meltano install

while

meltano elt

is still/already running? Meltano could place a lockfile in a plugin installation directory while the plugin is running, and have

meltano install

fail if it finds a plugin dir locked, but I worry about Meltano failing to clean up the lockfile if

meltano elt

is killed unceremoniously, which would stop

meltano install

from working until the lockfile is manually removed. We could detect stale lockfiles, but that immediately makes things a lot more complex.

Makes sense to me, but if Meltano doesn't do something like that it really means you shouldn't run more than two taps/targets, and ever run

meltano install .

So basically during development you should only be doing one elt at a time. And even then they should be in their own docker containers. I didn't hit this edge case until I had things running all the time now in my DEV and PROD environment. Didn't think anything of it, ran

meltano install .

and watched a bunch of stuff fail

So basically during development you should only be doing one elt at a time.

Why would it mean this? Multiple ELT pipelines should be able to use the same plugins venv at the same time

Yes it is, as a developer I'd rather not have to think about when I can or cannot run an install. Maybe I'm just being overly needy

I don’t think you are, being able to run

meltano install

whenever would be nice

From this I'll switch to only installing the extractor or loader I need at the specific moment which is fine, I really need to make a strategy for myself but just an idea from some run ins I had

Something else Meltano can do is instead of deleting the venv and then recreating it and installing the package, it could create the new venv in a separate directory, and only then delete the original venv and rename the new one to take the place of the original. That way the window during which the venv is empty is minimized

I'll put one in

Thanks! Left a comment