https://linen.dev logo
Join Slack
Powered by
hey y’all :wave: trying to run meltano within dags...
# troubleshooting
d
hey y’all 👋 trying to run meltano within dagster and hitting a weird issue where meltano seemingly can’t find the executables, even though they are located where meltano seems to expect them to be 🧵
Some logs from within Dagster when trying to run a 
meltano elt
command:
you can see that when running in 
--dry
mode it tells me where meltano is expecting to find the executable at, and you can also see that the executables actually exist (and are owned by dagster) in the locations where they are expected to be
but then when actually trying to run a 
meltano elt
command meltano claims to not be able to locate them
debug logs don’t see to be any more detailed
v
How do you know 
meltano install
ran before this command? Comes down to how you're running meltano, is it a container? If so how's the container setup? etc etc
d
this is in a container. meltano install is run in the docker image
Copy code
FROM base AS builder

RUN python -m venv "$VIRTUAL_ENV" && \
    mkdir -p "$DAGSTER_HOME"

ENV PATH="$VIRTUAL_ENV/bin:$POETRY_HOME/bin:$PATH"

SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN apt-get update && \
    apt-get -y upgrade && \
    apt-get install -y --no-install-recommends build-essential musl-dev g++ gcc git curl && \
    apt-get -y clean && \
    rm -rf /var/lib/apt/lists/* && \
    curl -sSL <https://install.python-poetry.org> | python -

COPY poetry.lock pyproject.toml /
RUN pip install --no-cache-dir --upgrade pip==21.3.1 setuptools==60.2.0 wheel==0.37.1 && \
    poetry config virtualenvs.path "$VIRTUAL_ENV" && \
    poetry install --no-root --no-interaction --no-ansi --no-dev

COPY meltano.yml /
RUN meltano install

FROM base AS runner

ENV PATH="$VIRTUAL_ENV/bin:$POETRY_HOME/bin:$PATH"
ENV MELTANO_PROJECT_READONLY=true

RUN groupadd -r dagster && useradd -m -r -g dagster dagster

COPY --from=builder $VIRTUAL_ENV $VIRTUAL_ENV
COPY --from=builder --chown=dagster $DAGSTER_HOME $DAGSTER_HOME

WORKDIR $DAGSTER_HOME

FROM runner AS ucr
COPY --from=builder --chown=dagster /.meltano $DAGSTER_HOME/.meltano
COPY --from=builder --chown=dagster meltano.yml $DAGSTER_HOME/meltano.yml
COPY --chown=dagster logging.yaml $DAGSTER_HOME/logging.yaml
COPY --chown=dagster dutchie_dagster_meltano/ ./dutchie_dagster_meltano
USER dagster:dagster
EXPOSE 4000
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-m", "dutchie_dagster_meltano"]
some stages removed but this is the gist of it
v
so when does 
meltano install
run?
I'd look into that / debug locally
d
Sorry I'm failing to see why that matters @visch . I've shown in the screenshots and docker image that meltano install has been run and as a result the executable exists in the expected location
Is there a special place that the elt command expects executables to exist? Why would it be different than the locations that are output when it runs in dry mode?
v
Your logs I can't copy, please put them in text so I can help you better. Logs mention a tap doesn't exist. Meltano install, installs them for you, seems likely that's part of the issue. Yes there's tons of details around that, feel free to dive in I'm just trying to help!
Docker doesn't help too much as I don't know when/how the docker build process is ran. When you run that docker file by itself does it work / not work as expected?
What are the logs, and the command ran that you've tried so far in debugging
d
sorry about that @visch — I didn’t realize you couldnt see the logs. I’ve attached them here:
dagster_meltano_logs.txt
at a high-level the steps Im taking to troubleshoot are 1) make sure the executables actually exist in the place that meltano expects to (and also make sure they have the appropriate permissions) 2) run meltano install anyway just to be sure that meltano install is being run before running elt 3) finally running meltano elt
still seems that everytime meltano claims that meltano install hasnt been run and that the executables don’t exist where they should
and here some logs outside of dagster just trying to run the same set of commands inside the container:
Copy code
dagster@d982d5658b17:/opt/dagster/dagster_home$ meltano install
Installing 6 plugins...
Installing loader 'target-snowflake'...
Installing extractor 'tap-launch-darkly'...
Installing extractor 'tap-chargebee'...
Installing extractor 'tap-lessonly'...
Installing extractor 'tap-exchange-rates-api'...
Installing extractor 'tap-ocs'...
Installed extractor 'tap-exchange-rates-api'
Installed extractor 'tap-ocs'
Installed extractor 'tap-chargebee'
Installed extractor 'tap-lessonly'
Installed extractor 'tap-launch-darkly'
Installed loader 'target-snowflake'
Installed 6/6 plugins
dagster@d982d5658b17:/opt/dagster/dagster_home$ meltano elt tap-launch-darkly target-snowflake
{"name": "meltano", "run_id": "7db4c784-9b00-4208-b361-a6d517a7f87b", "job_id": "2022-02-07T043856--tap-launch-darkly--target-snowflake", "event": "Running extract & load...", "level": "info", "timestamp": "2022-02-07T04:38:57.208816Z"}
{"name": "meltano", "run_id": "7db4c784-9b00-4208-b361-a6d517a7f87b", "job_id": "2022-02-07T043856--tap-launch-darkly--target-snowflake", "stdio": "stderr", "cmd_type": "elt", "event": "ELT could not be completed: Cannot start extractor: Executable 'tap-launch-darkly' could not be found. Extractor 'tap-launch-darkly' may not have been installed yet using `meltano install extractor tap-launch-darkly`, or the executable name may be incorrect.", "level": "info", "timestamp": "2022-02-07T04:38:57.277360Z"}
{"name": "meltano", "run_id": "7db4c784-9b00-4208-b361-a6d517a7f87b", "job_id": "2022-02-07T043856--tap-launch-darkly--target-snowflake", "stdio": "stderr", "cmd_type": "elt", "event": "For more detailed log messages, check the generated log file '/opt/dagster/dagster_home/.meltano/logs/elt/2022-02-07T043856--tap-launch-darkly--target-snowflake/7db4c784-9b00-4208-b361-a6d517a7f87b/elt.log' or re-run the command using the '--log-level=debug' CLI flag.", "level": "info", "timestamp": "2022-02-07T04:38:57.277896Z"}
ELT could not be completed: Cannot start extractor: Executable 'tap-launch-darkly' could not be found. Extractor 'tap-launch-darkly' may not have been installed yet using `meltano install extractor tap-launch-darkly`, or the executable name may be incorrect.
For more detailed log messages, check the generated log file '/opt/dagster/dagster_home/.meltano/logs/elt/2022-02-07T043856--tap-launch-darkly--target-snowflake/7db4c784-9b00-4208-b361-a6d517a7f87b/elt.log' or re-run the command using the '--log-level=debug' CLI flag.
v
Nice @david_wallace , full logs are very helpful. Yes great the meltano install is happening, and then 
tap-launch-darkly
cannot be found. 1. So I have to infer that tap-launch-darkly is a custom tap, correct? 2. Can we see the meltano.yml for the custom tap All meltano does is try to run the extractor from the .meltano/extractors/venv/bin/* directory (Could be slightly off on dir names). In that directory is there a 
tap-launch-darkly
executable? Or is it named something else like maybe 
tap-launch
?
Even better if you could point to the code for tap-launch-darkly that would help too.
Troubleshooting wise, try to run the tap locally yourself, does it work?
t
Agreed that seeing the 
meltano.yml
would help debug a bit as well. Thanks @visch for diving in 💜
d
okay so I figured this out and it was a doozy
will circle back with details shortly
tl;dr it was a disconnect between where 
meltano elt --dry
says the executables are located, and what those executables interpreter directives are.
t
@david_wallace any more info you can share about this? Or maybe can you open an issue?
d
Oops yeah sorry I did forget to follow up. Setting a reminder for today
okay so here we go. Running the 
meltano elt
 command in 
--dry
 mode will seemingly output where exactly meltano expects the extractor/loader executables to be located. When I was running it in dry mode, I was seeing the following output:
all well and good so far, I’m able to confirm that the executables do indeed exist at those locations. However, when going to run the 
meltano elt
 command in non-dry mode, I was seeing the following error:
this was totally elusive to me for quite a while. what I didnt think check was the interpreter directive in the executables themselves. Once I checked those, I saw this:
Copy code
dagster@997b6fea8440:/opt/dagster/dagster_home/.meltano/extractors/tap-launch-darkly/venv/bin$ cat tap-launch-darkly
#!/.meltano/extractors/tap-launch-darkly/venv/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from tap_launch_darkly.tap import TapLaunchDarkly
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(TapLaunchDarkly.cli())
ah ha! the interpreter directive in the executables point to a different location, the location where the 
meltano install
 command was initially run.
so tl;dr is that even though we were moving the entire 
.meltano/
 directory during image build that is created as a result of 
meltano install
 to the dagster home directory, the issue is that meltano seems to hardcode the interpreter directives into the executables it creates, and those directives are based on where meltano install was run
v
Got it, I think the "proper" approach is 
.meltano/
is meltano internals, touch at your own risk with little guarantees of backwards compatibility. Curious if the Meltano folks think that! Moving those folders around (Especially things needed at runtime) after 
meltano install
has ran is asking for pain!
d
yeah, that makes sense in hindsight
v
always 20-20 isn't it haha
I feel you!
d
kinda surprised that the dry run elt command outputs the new executable path though
as in, the path where I moved the 
.meltano
dir
v
hmmm
Don't have time to dive in 😕 , curious though
d
yeah no worries! i think thats the only really surprising thing about all of this
t
This may be a bug then! <!subteam^S02BCD9FFEF> any thoughts on this one? Would be worth an issue either way @david_wallace if you have time
e
The shebang is added by 
pip
afaict. It hardcodes whatever the value of 
sys.executable
is when 
pip install
is called. I'm not sure if it can be a relative path or changed after the fact
Open in Slack
PreviousNext
Powered by