Hello all I want suggestion or ideas in how to approach this Meltano #getting-started

Hello all, I want suggestion or ideas in how to ap...

boggdan_barrientos

09/23/2021, 10:23 PM

Hello all, I want suggestion or ideas in how to approach this problem, Someone else is orchestrating Meltano with existing Airflow installation. I have Airflow installed and running, so I'm using the same instance of Airflow to orchestrate meltano jobs. I run my pipelines with ECSOperator. It executes the meltano command. In some cases my elt process is not complete, so for me is a fail or something that I have to solve, but in that cases my task mark as success. As the SS attached, It was an Error but my task was mark as success. I could do something more in Meltano side, to avoid this. Something with the exit code o related to this. Or solve on the Airflow and operator side. Thanks!

edgar_ramirez_mondragon

09/27/2021, 5:33 PM

Hi @boggdan_barrientos. This may be a bug, so let me do a few quick runs to verify...

edgar_ramirez_mondragon

09/27/2021, 5:59 PM

Ok, just checked. It seems like failures in

meltano elt ...

do translate to an exit code of 1:

Copy code

$ meltano elt tap-github target-sqlite
target-sqlite | ERROR Exception in schema_apply() while prrocessing:
target-sqlite | {"type": "SCHEMA", "stream": "repositories", "schema": {"properties": {}, "type": "object"}, "key_properties": ["id"]}
target-sqlite |
target-sqlite | CRITICAL (sqlite3.OperationalError) unable to open database file (Background on this error at: <http://sqlalche.me/e/e3q8>)
meltano       | Loading failed (1): CRITICAL (sqlite3.OperationalError) unable to open database file (Background on this error at: <http://sqlalche.me/e/e3q8>)
meltano       | ELT could not be completed: Loader failed

$ echo $?                                                                                                                                                                                                                             
1

So I'm thinking the docker container is just exiting with code 0. @boggdan_barrientos are you using the official Meltano docker image?

boggdan_barrientos

09/27/2021, 6:05 PM

I think so. This is my dockerfile. How can I verify it?

edgar_ramirez_mondragon

09/27/2021, 7:01 PM

hmm so it seems a Meltano exit code of 1 does translate to the same for docker:

Copy code

$ docker run --name melty-run melty:test elt tap-github target-sqlite
meltano       | Running extract & load...
meltano       | No state was found, complete import.
meltano       | ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/project/.meltano/extractors/tap-github/venv/bin/tap-github', '--config', '/project/.meltano/run/elt/2021-09-27T185857--tap-github--target-sqlite/d2ce9b0a-ace6-4f02-8e5e-a4ecbaa30e25/tap.9279845d-ecb2-4396-8939-41d5d24c0b5b.config.json', '--discover'] returned 1
ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/project/.meltano/extractors/tap-github/venv/bin/tap-github', '--config', '/project/.meltano/run/elt/2021-09-27T185857--tap-github--target-sqlite/d2ce9b0a-ace6-4f02-8e5e-a4ecbaa30e25/tap.9279845d-ecb2-4396-8939-41d5d24c0b5b.config.json', '--discover'] returned 1

$ docker inspect melty-run | jq '.[0].State.ExitCode'
1

boggdan_barrientos

09/27/2021, 7:14 PM

mmm ok, so the bug is not in meltano and docker side. I will check my ECS task, but it has an weird behavior, not makes sense. Thanks @edgar_ramirez_mondragon

edgar_ramirez_mondragon

09/27/2021, 7:51 PM

np @boggdan_barrientos. Let me know if you find something in the ECS console

Open in Slack

Previous Next