Well this seems to be a strange error working with tap sales Meltano #troubleshooting

Well this seems to be a strange error working with...

steve_clarke

10/27/2021, 7:03 AM

Well this seems to be a strange error working with tap-salesforce. I am running this in ECS, it only appears to succeed if I run with a --log-level debug

Copy code

chamber,exec,meltano-dev/tap-salesforce,meltano-dev/target-snowflake,--,meltano,--log-level,debug,elt,tap-salesforce,target-snowflake,--job_id,--job_id=salesforce-border-to-snowflake,--force

If I remove the switch --log-level,debug it hangs and never completes. It appears to get stuck at the discovery stage.

Copy code

2021-10-27 19:00:18
meltano | elt | Found state from 2021-10-27 05:50:24.722952.

2021-10-27 19:00:18
meltano | elt | Running extract & load...

I really don't understand why running it in debug mode would make the elt pipeline work. Note: This problem doesn't affect my other pipeline using tap-spreadsheets-anywhere. Seems to be tap-salesforce that I am having an issue with. I am using an ECS container with no volumes so it is a case of the file system being ephemeral. I also use an Aurora / Postgres database to hold my state. It is designed to spin up and down based on usage - so it should be quite cheap to run. I have messaged @chris_kings-lynne, who is running a similar config to myself. He said his pipeline was working for months but has noticed that it recently started failing like I am experiencing. There was a rebuild of his Docker image recently too. So obviously running in debug mode changes something to allow the job to succeed. I really don't want to run in debug mode as it will be logging all the data to the log. Very strange.

taylor

10/27/2021, 4:53 PM

@florian.hines any thoughts on that since you were touching parts of this code recently? @steve_clarke what version of Meltano are you on?

florian.hines

10/27/2021, 4:56 PM

hrm - this smells like like it could be an async bug that we maybe introduced.

florian.hines

10/27/2021, 4:59 PM

We’ve made logging and discovery related logging changes across a couple of releases lately. So version would help - but i can start poking at the latest version in the mean time.

florian.hines

10/27/2021, 5:16 PM

I pop’d an issue for this https://gitlab.com/meltano/meltano/-/issues/3027 - I have a suspicion on what might be happening

steve_clarke

10/27/2021, 7:57 PM

Hi, in my Docker file I am pulling from meltano/meltano:latest

Copy code

ARG MELTANO_IMAGE=meltano/meltano:latest-python3.8
FROM $MELTANO_IMAGE

For tap-salesforce, I am pulling the latest version as well.

Copy code

plugins:
  extractors:
  - name: tap-salesforce
    variant: meltano
#    pip_url: git+<https://gitlab.com/meltano/tap-salesforce.git@v1.4.27>
    pip_url: git+<https://gitlab.com/meltano/tap-salesforce.git>

@florian.hines, @taylor Should I try going back to a previous version of Meltano?

florian.hines

10/27/2021, 8:28 PM

Going back one release to 1.84.0 i think probably does fix this. We’ll have a fix up for this here in a bit as well, and that should roll out tomorrow as part of the 1.86.0 release.

steve_clarke

10/27/2021, 8:37 PM

Great news, thank you I'll try 1.84.0 now, and will move forward to 1.86.0 when it is released. I will respond back if 1.84.0 resolves the issue.

steve_clarke

10/27/2021, 11:53 PM

Okay, I have had to try a few things. I have a working version now with meltano/meltano:v1.79.1. So I tried the following image and it didn't work

Copy code

ARG MELTANO_IMAGE=meltano/meltano:v1.84.0:latest-python3.8
FROM $MELTANO_IMAGE

Then I tried this image and it didn't work

Copy code

ARG MELTANO_IMAGE=meltano/meltano:v1.80.0:latest-python3.8
FROM $MELTANO_IMAGE

Then I wondered if it had something to do with the Python 3.8 image so went to the default 3.6. I also used the same version that I had installed on my on-premise server which was working 1.79.1, I haven't confirmed whether I need Meltano 1.79.1 or whether it is something to do with the Python 3.8 build.

Copy code

ARG MELTANO_IMAGE=meltano/meltano:v1.79.1
FROM $MELTANO_IMAGE

I hope this information is useful info.

steve_clarke

10/29/2021, 12:12 AM

Hi @florian.hines, I have rebuilt my docker container using. Python 3.6 and have also tried Python 3.8 and can confirm with the latest 1.86.0 release that tap-salesforce is working for me now. So in summary I can use

Copy code

ARG MELTANO_IMAGE=meltano/meltano:v1.86.0-python3.8
FROM $MELTANO_IMAGE

and

Copy code

ARG MELTANO_IMAGE=meltano/meltano:v1.86.0
FROM $MELTANO_IMAGE

Thank you very much for sorting out this issue so quickly. Much appreciate Florian and the Meltano team. 👏🎉

Open in Slack

Previous Next