Not very familiar with docker but having an issue here today Meltano #docker

Not very familiar with docker but having an issue ...

Andy Carter

10/09/2023, 1:03 PM

Not very familiar with docker but having an issue here today with

tap-spreadsheets-anywhere

. My tap runs fine locally outside of docker with 2 streams. When trying to deploy in docker on Azure, I am seeing no streams running, and no errors to indicate an obvious issue. I drop into the console of my container, run

cat tap.properties.json

for my tap and see:

Copy code

{
  "streams": []
}

That's bad right? Looking at the local version of my file I can see details of the two expected streams. How can I force that file to get populated at build time? Dockerfile below (pretty much stock):

Andy Carter

10/09/2023, 1:04 PM

Copy code

# <http://registry.gitlab.com/meltano/meltano:latest|registry.gitlab.com/meltano/meltano:latest> is also available in GitLab Registry
ARG MELTANO_IMAGE=meltano/meltano:v2.20.0-python3.10
FROM $MELTANO_IMAGE

WORKDIR /project

# Install any additional requirements
COPY ./requirements.txt .
RUN pip install -r requirements.txt

# Copy over Meltano project directory
COPY . .

# overwrite dagster.yaml with contents of dagster_azure.yaml
RUN rm -rf ./orchestrate/dagster/dagster.yaml
COPY ./orchestrate/dagster/dagster_azure.yaml ./orchestrate/dagster/dagster.yaml

RUN meltano install

# Don't allow changes to containerized project files
ENV MELTANO_PROJECT_READONLY 1

visch

10/09/2023, 1:21 PM

One thing that I dislike about that tap is it doesn't error when something is wrong it just runs and outputs nothing. Makes it hard to narrow this stuff down, it's almost always a config issue

Andy Carter

10/09/2023, 1:22 PM

Yeah, too clever by half. I know a lot of tricks to get things fixed up locally.

Andy Carter

10/09/2023, 1:23 PM

Wait, I should be able to execute meltano commands directly against the container if I can get into the console 🤦‍♂️

Andy Carter

10/09/2023, 1:26 PM

I just don't get containers yet 😞 thanks for the nudge. GMT crew 👋

visch

10/09/2023, 1:33 PM

If you want a run through / get stuck let me know I've been through it with widows and Linux containers happy to have a session with you, you've done tons of good stuff for the community!!

Andy Carter

10/09/2023, 1:38 PM

I think I found it, access issues, IP filtering etc.

Andy Carter

10/09/2023, 1:39 PM

But thanks that is very kind of you to say so

Open in Slack

Previous Next