https://linen.dev logo
Join Slack
Powered by
I've got Meltano running in a container alongside ...
# troubleshooting
d
I've got Meltano running in a container alongside Airflow and I am getting a frustrating error: 
{"singer_state": {"currently_syncing": "vwProjectMetricRepl", "bookmarks": {"vwProjectRepl": {"replication_key": "updated_at", "version": 1624538306120, "replication_key_value": "2021-06-24T18:47:14.747000+00:00"}}}, "error": "ELT could not be completed: Loader failed"}
I've got both 
meltano
and 
airflow
writing into Postgres, which is also the same DB as my target. The source is mssql, the target is postgres (Amazon Aurora) I've run this pipeline from the CLI with success. The meltano UI is unable to display the logs for this run, and I am unable to SSH into this particular container (running in AWS Fargate). Any ideas?
v
Make a "bastian" host jump into the container? https://stackoverflow.com/questions/52310447/is-it-possible-to-ssh-into-fargate-managed-container-instances Need to see logs. You could possibly get away without sshing in. Next command would be 
meltano --log-level=debug elt tapname targetname
So you can see what's going on.
My guess based on the limited data here is your container doesn't have access to either Postgres or MSSQL
Probably network config
You have me thinking about tricks to get a shell using Airflow I'm sure that's possbile to 🙂
d
New information: If I run the pipeline from the meltano UI, I can see that it is connecting to both tap and target, and I can see the log. I think I need to expose a volume from my 
airflow_scheduler
container to my 
meltano_ui
container so that logs generated by airflow runs can be loaded.
v
nice!
d
Unfortunately it still failed
Copy code
meltano | Loading failed (-9): INFO Writing batch with 204794 records for `vwProjectMetricRepl` with `key_properties`: `['_sdc_primary_key']`
meltano | ELT could not be completed: Loader failed
Bit vague
Hopefully setting 
MELTANO_CLI_LOG_LEVEL=debug
will give me more info
Still waiting on the deployment, might have to pick this up in the morning (11;30pm here in Sydney, AUS) I pulled the container image, and ran the job myself from an EC2 instance and it worked fine. So really hoping the debug mode gives me something
v
It should, whatever the error is that isn't showing up come back and share as hopefully we can make an issue to try to have this type of thing get shown when running elt by default
I think there's a layer of debugging that needs to still be added, it's complicated as you have multiple processes and streams. I am not sure yet but it's almost like a logging level filter on stderr streams from taps/targets 1. tap-stdout(Sometimes not configured to properly show errors in stderr) 2. tap-stderr(Messages are not just stderr since we got rid of stdout. ) 3. target-stdout (Sometimes not configured to properly show errors in stderr) 4. target-stderr (Messages are not just stderr since we got rid of stdout. ) 5. meltanoproc-stdout 6. meltanoproc-stderr
Nice part is you can test individually, like in your case if it's the loader. you can run 
cat testinput | meltano invoke target-postgres
Airflow adds another layer I"m guessing?
d
Airflow more of a red herring I think. I've got 3 containers in my task definition: • Meltano UI • Airflow Scheduler • Airflow Webserver When the scheduler kicks of a job, the execution happens locally (I am using the 
LocalExecutor
), and it writes to the 
/project/.meltano/logs/elt/
folder on that container. I should have realised that the separate container (same task) instance that is running the meltano UI would not see any files written on another container. This is a non-issue, because I can just click the "Run now" button in the meltano UI, and then execution happens from that container instead. My new build/task def defines a volume for the logs path so that containers can share their logs with each other, and setting the debug mode on the Meltano UI container. I have been trawling through the source code and I can't find where loader exit codes are defined. I'm getting 
-9
.
What the....it worked this time...
d
-9
typically means that the OS killed the process because of an out-of-memory issue. Is that possible?
d
Probability = 100%!
I need to give this puppy more juice...
Doubling CPU and Memory. We'll see how that looks tomorrow. Thanks for the help all!
Cranked the resources up and everything is hunky dory now
4vCPU, 8GB RAM seems to be holding nicely. Seems a little overkill but I'll take it for now. @douwe_maan Is there a way to pass some configuration to the gunicorn webservers to only spin up a single worker? We're not likely to have users hitting this too much. It'll mainly do it's thing in the background
v

https://www.youtube.com/watch?v=Ud4HuAzHEUc&t=70sâ–¾

don't listen too long or you'll burn brain cells
I didn't catch the -9 @douwe_maan for the win!
d
2 Views
Open in Slack
PreviousNext
Powered by