https://linen.dev logo
Join Slack
Powered by
Trying to build a custom docker image and run a co...
# troubleshooting
j
Trying to build a custom docker image and run a container from it. 
TAP_GITHUB_ACCESS_TOKEN
is not correctly propagated into a running container. Trying to investigate it, but cannot override the entry point to e.g. bash to find out what is happening inside. Cannot find the source code of your Dockefile, from which you build your meltano/meltano image. Please, send the source code or guide me how to override the entry point.
p
you can do docker run meltano -v $PWD:/project <command> to do bind the vol to your lcoal
j
Binding volume works as expected. Specifically, I need to execute something like 
docker run meltano /bin/sh
, resp. I need to test this with docker-compose, where I declare the variable in 
docker-compose.yaml
and need to login into the container executing a shell and validate that the variable is set properly
Because now, when I execute meltano run in the container, it fails:
Copy code
extract_load_1    | 2023-01-05T12:46:00.638363Z [info     ] singer_sdk.exceptions.RetriableAPIError: 401 Client Error: b'{"message":"This endpoint requires you to be authenticated.","documentation_url":"<https://docs.github.com/graphql/guides/forming-calls-with-graphql#authenticating-with-graphql>"}' (Reason: Unauthorized) for path: /graphql cmd_type=elb consumer=False name=tap-github-repo producer=True stdio=stderr string_id=tap-github-repo
extract_load_1    | 2023-01-05T12:46:00.809613Z [error    ] Extractor failed
Which looks like the GITHUB token is not propagated correctly
s
Sounds like you need to exec into a running container. "docker exec -it <container name> sh" should do the trick, unless your docker compose is already down again.
j
Because the entry point is 
meltano
, it does not work. I tried to override the entry point like this:
Copy code
ENTRYPOINT ["bash", "-c"]
Rebuilt the image and ran:
Copy code
docker-compose exec -it extract_load ls -la /
Zero output
s
I usually just override this specific image with entrypoint: sh
(you can overwrite right in docker compose)
j
Copy code
(.venv) jacek@holly:~/work/src/gooddata-data-pipeline$ docker run --entrypoint /bin/sh meltano/meltano ls -la
/bin/sh: 0: cannot open ls: No such file
Using your container
You do not expand PATH at all it seems 😉
This is even weirder:
Copy code
(.venv) jacek@holly:~/work/src/gooddata-data-pipeline$ docker run --entrypoint /bin/sh meltano/meltano /bin/ls /
/bin/ls: 1: Syntax error: ")" unexpected
s
Eh so "docker run -it --entrypoint sh meltano/meltano" works just fine
j
Works. But not with docker-compose
s
Ok took me a moment, and "docker run -it --entrypoint /bin/bash meltano/meltano" should work as well
What does your docker compose yaml look like?
j
OK, this finally works:
Copy code
extract_load:
    build:
      context: src
      dockerfile: Dockerfile_meltano
    entrypoint:
      - bash
      - -c
    command:
      - "echo $TAP_GITHUB_ACCESS_TOKEN: $$TAP_GITHUB_ACCESS_TOKEN"
The variable is set correctly. What could cause the 401 unauthorized error when this variable is set?
Btw it works from localhost without docker with the same value of TAP_GITHUB_ACCESS_TOKEN
p
Where are you setting the env var? In you shell or in .env? If it’s in the .env and mounts are correct then it should work without explicit mapping, I think
j
Copy code
extract_load:
    build:
      context: src
      dockerfile: Dockerfile_meltano
    command:
      - run
      - tap-github-repo
      - target-postgres
      - tap-github-org
      - target-postgres
    environment:
      POSTGRES_HOST: gooddata-cn-ce
      POSTGRES_PORT: 5432
      POSTGRES_USER: demouser
      POSTGRES_PASS: demopass
      POSTGRES_DBNAME: demo
      POSTGRES_INPUT_SCHEMA: cicd_input_stage2
      MELTANO_DATABASE_URI: "<postgresql://demouser:demopass@gooddata-cn-ce:5432/demo?options=-csearch_path%3Dmeltano>"
      TAP_GITHUB_ACCESS_TOKEN: "xxxxxxxxxxxxxxxxxxx"
No .env file
Dockefile:
Copy code
ARG MELTANO_IMAGE=meltano/meltano:v2.12.0-python3.10
FROM $MELTANO_IMAGE

WORKDIR /project

# Copy over Meltano project directory
COPY meltano.yml meltano.yml
RUN meltano install

# Don't allow changes to containerized project files
ENV MELTANO_PROJECT_READONLY 1

# Expose default port used by `meltano ui`
EXPOSE 5000

ENTRYPOINT ["meltano"]
meltano.yml:
s
Did you exec into your container and tried to echo the env variable? Because that should work just fine from the looks of it. Maybe there is a naming problem? You're using TAP_GITHUB_ACCESS_TOKEN, but you're now using the MeltanoLabs variant and that uses the "AUTH_TOKEN" parameter (I know that's confusing, confused me as well when I switchd over between the variants).
(Env variable: TAP_GITHUB_AUTH_TOKEN)
j
Jesus, that's it! Interesting that it somehow worked with TAP_GITHUB_ACCESS_TOKEN outside the docker. Thanks so much! Now I have to make it work in a Gitlab CICD...
s
Oh that's easy. Just use some kind of secrets store and source them into your CI file (TAP_GITHUB_AUTH_TOKEN: ${{ secrets.TAP_GITHUB_AUTH_TOKEN }} ). We got a few setups for that already running, so if you're stuck, just mention me or @pat_nadolny again 😉
And it's gonna be very similar on gitlab CI.
j
Will share the final demo here + going to write an article about the whole story of developing it.
s
We're looking forward to it! Would love to have a GitLab CI powered demo to link to.
Open in Slack
PreviousNext
Powered by