I’ve got two very similar runs happening in parall...
# plugins-generalj
josh_lloyd
02/10/2023, 10:04 PMI’ve got two very similar runs happening in parallel processes on the same machine (using dagster). The only difference between the two runs are the environment variables, otherwise I’m running the exact same tap, just targeting two different instances of mysql. I’m running tap-mysql with target-snowflake. When I run these two pipelines separately they run fine, but when they run in parallel (starting at the exact same time) I get this error:
Copy code
Cannot start plugin tap-mysql--foo: Catalog discovery failed: invalid catalog: Extra data: line 6544 column 11 (char 157140)v
visch
02/10/2023, 10:31 PMahh yes I have hit this.
Easiest fix is to run everything in a docker container and you'll be good to go.
The issue (I believe) is that meltano accesses the catalog file at the same time as another job is modifying the file. It seems that sometimes this file can be in a state where it's not valid JSON. Not exactly sure why but I have it happen all the time as well. Docker just makes the problem go away as you're not both accessing the same catalog file.
j
josh_lloyd
02/10/2023, 10:33 PMhmmm. I suspected as much, the problem is that Dagster is already running this process in a container, so I’m not entirely sure it’s possible for meltano to kick off a separate container from the one already in play 🤷
a
aaronsteers
02/10/2023, 10:45 PMDefinitely sounds like a bug. Does this also happen if you run discovery first before kicking off both pipelines?
aaronsteers
02/10/2023, 10:46 PMFrom reading the above, it sounds like both are writing out json text to the same file. If you run something like
meltano select tap-mysql --listj
josh_lloyd
02/10/2023, 10:47 PMgood question. Not sure I can test that in the current configuration because I’m using the
dagster_meltanomeltano run--log-leveldebugjosh_lloyd
02/10/2023, 10:49 PMGuess I have to wait for the
dagster_meltanojosh_lloyd
02/10/2023, 10:49 PMmy previous code didn’t have this problem oddly enough but it was calling subprocesses slightly differently
josh_lloyd
02/10/2023, 11:30 PM@aaronsteers I managed to force it to do a
meltano select tap-mysql --lista
aaronsteers
02/10/2023, 11:31 PMSeparate containers or wiped execution workspace between runs perhaps?
aaronsteers
02/10/2023, 11:32 PMThat's very odd
aaronsteers
02/10/2023, 11:33 PMIf separate containers/directories, then I wouldn't expect the file conflict in the first place : 🤔
j
josh_lloyd
02/10/2023, 11:33 PMthey are definitely not running in separate containers
josh_lloyd
02/10/2023, 11:33 PMI’m not sure how to confirm or deny the wiped execution workspace though 🤔
josh_lloyd
02/14/2023, 5:16 PMAs an interesting follow up to this. I have a second dagster pipeline that is nearly identical to this one in that it runs 2 parallel meltano pipelines starting at the exact same time, the only difference is the tap itself. I get no errors when running this other tap-dynamodb based pipeline, but I do with the tap-mysql pipeline …
v
visch
02/14/2023, 5:59 PMWhat I think I "know" about this issue is that it happens "randomly" its caused by something overwriting the catalog file, it happens when jobs are ran at the "same" time. Hard to debug as it doesn't happen all of the time. When I use different containers for each run I don't have this issue. One bandaid fix would be to have meltano create catalog files with
guid-properties.jsonproperties.jsonvisch
02/14/2023, 6:00 PMAn easy fix for you (maybe?) Josh if you can would be to use a different tap name for each of the runs? Depends on how many of the same instance you need though! For me it is 100's so I didn't go that route