https://linen.dev logo
Join Slack
Powered by
Hello everyone I’m trying to detect a success or f...
# troubleshooting
j
Hello everyone I’m trying to detect a success or failure of a pipeline from SSHOperator in Airflow but I don’t know how I’m trying to do 
meltano elt <etc> | grep 'Extract & load complete'
but it doesn’t seam to be the right approach I would also like to detect errors that sometimes meltano (or the singer protocol in general) misses. What do I mean by that ? Sometimes an API sents an error 502 and the pipeline continues and finishes but I would like to detect that issue anyway Anyone could help me by pointing me at the right direction ?
v
The exit code should do what you want! The process will return a 0 code if things are good!
no need to grep, unless I'm missing something
ahh
Copy code
Sometimes an API sents an error 502 and the pipeline continues and finishes but I would like to detect that issue anyway
Anyone could help me by pointing me at the right direction
That will require searching the log. grep would work but I'd run it seperate from the elt process. I'm going to dive into some of this later, but I am a fan of exit codes that have some meaning. 0 = success, 1 is failure, but what about a partial failure. 1337 for partial failures or something would be helpful
taps/targets would have to abide by this though and they don't today ( I think) so your best bet is grepping the logs
j
Thank you very much @visch Yes the exit code is not sufficient in my case unfortunately
v
j
Yes, I saw that article 😄
But I wanted to know if there was maybe a solution directly from meltano instead of bashing my way out haha
v
Yeah maybe Douwe or the Meltano team will be able to point to an issue that's already out there!
2 Views
Open in Slack
PreviousNext
Powered by