https://linen.dev logo
Join Slack
Powered by
So - I have two container apps, one for staging an...
# troubleshooting
i
So - I have two container apps, one for staging and one for production. For some reason, my dbt models (as the second step of each job) are not materializing in production dagster jobs but they ARE materializing in my staging jobs. My extractions are running in both environments but specifically the dbt model build fails, and only in the production container. The docker image is the exact same in both environment (After i deploy to stg environment, my production deployment takes that image and tags it and sends it to prod), and I have run a "dbt-snowflake:run" from the AZ containerapp console successfully so it isn't some sort of env var issue - dbt seems to work fine. Any ideas on what to check next? The only other difference I could think of is the Azure blob store backend - but I'm under the impression that replication key state store is what exists there - which frankly is working fine and I don't believe it has anything to do with dbt. I'm really scratching my head on where to go from here. I don't have the exact error but it had something to do with 
manifests/macros.sql
or something of the sort, which just about always turns out to be something with the dbt version and sort of fixes itself. Again - same exact image being run on both environments.
v
The details are super important here, I'd try to get a shell in production and see exactly what things are/aren't set to. For us to help we have to see things this isn't "hand waveable"
i
I have already done a 
meltano config dbt-snowflake list
on both and the only difference is the db name which is expected.
Is that what you're referring to when you say "set to"?
v
You need to gather more info, maybe someone else has time to disect this for you. I'd throw everything you have above into o1-preview and see if it helps you with next steps.
✅ 1
i
For anyone else, I've run 
meltano invoke dbt-snowflake:debug
which succeeded on both, run`meltano config dbt-snowflake list` 
meltano invoke dbt-snowflake --version
and everything is the same between them. All env var's are the same (except for the db of course
Copy code
FROM meltano/meltano:v3.5.1-python3.10

WORKDIR /project

# Install any additional requirements
COPY ./requirements.txt .
RUN pip install -r requirements.txt

# Copy over Meltano project directory
COPY . .
RUN meltano install

# Set env var to false so DBT can run "clean"
ENV DBT_CLEAN_PROJECT_FILES_ONLY=false

# Don't allow changes to containerized project files
ENV MELTANO_PROJECT_READONLY=1

RUN apt-get update && apt-get install -y dos2unix
COPY entrypoint.sh /project/entrypoint.sh
RUN dos2unix /project/entrypoint.sh
RUN chmod +x /project/entrypoint.sh

ENTRYPOINT ["/bin/bash", "-c", "/project/entrypoint.sh"]
Dockerfile entrypoint.sh parses my dbt manifest then starts dagster
Copy code
utilities:
  - name: dbt-snowflake
    variant: dbt-labs
    pip_url: dbt-core~=1.8.0 dbt-snowflake~=1.8.0 git+<https://github.com/meltano/dbt-ext.git@v0.1.0>
    commands:
      parse_manifest:
        args: parse --quiet
        executable: dbt_invoker
meltano.yml (db config in "environments")
j
when DBT runs/fails to run, doesn't dagster give you the stack trace as to why?
i
Copy code
Errors parsed from dbt logs:

Encountered an error:
'dbt_<snowflake://macros/catalog.sql>'

Traceback (most recent call last):
  File "/project/.meltano/utilities/dagster/venv/lib/python3.10/site-packages/dbt/cli/requires.py", line 138, in wrapper
    result, success = func(*args, **kwargs)
  File "/project/.meltano/utilities/dagster/venv/lib/python3.10/site-packages/dbt/cli/requires.py", line 101, in wrapper
    return func(*args, **kwargs)
  File "/project/.meltano/utilities/dagster/venv/lib/python3.10/site-packages/dbt/cli/requires.py", line 218, in wrapper
    return func(*args, **kwargs)
  File "/project/.meltano/utilities/dagster/venv/lib/python3.10/site-packages/dbt/cli/requires.py", line 247, in wrapper
    return func(*args, **kwargs)
  File "/project/.meltano/utilities/dagster/venv/lib/python3.10/site-packages/dbt/cli/requires.py", line 294, in wrapper
    return func(*args, **kwargs)
  File "/project/.meltano/utilities/dagster/venv/lib/python3.10/site-packages/dbt/cli/requires.py", line 320, in wrapper
    ctx.obj["manifest"] = parse_manifest(
  File "/project/.meltano/utilities/dagster/venv/lib/python3.10/site-packages/dbt/parser/manifest.py", line 1898, in parse_manifest
    manifest = ManifestLoader.get_full_manifest(
  File "/project/.meltano/utilities/dagster/venv/lib/python3.10/site-packages/dbt/parser/manifest.py", line 330, in get_full_manifest
    manifest = loader.load()
  File "/project/.meltano/utilities/dagster/venv/lib/python3.10/site-packages/dbt/parser/manifest.py", line 399, in load
    self.load_and_parse_macros(project_parser_files)
  File "/project/.meltano/utilities/dagster/venv/lib/python3.10/site-packages/dbt/parser/manifest.py", line 676, in load_and_parse_macros
    block = FileBlock(self.manifest.files[file_id])
KeyError: 'dbt_<snowflake://macros/catalog.sql>'
@joshua_janicas I was able to replicate the error on local container. This is what I'm getting - same error as in ACA. Again, when I swap the meltano env over to use prod config and prod snowflake database that's what breaks it. That is the only difference lol. Just tested it with dev and it materialized without failure. At least I know that the failure isnt due to the ACA configuration now since I had the same failure on local.
j
it looks like a bad parsing of the manifest file? can you try to force a rebuild of the manifest?
✅ 1
maybe it's holding on to the old manifest file between envs?
i
ChatGPT is saying to run the docker build with a --no-cache flag
Going to try that
j
as part of my Dockerfile i also run a shell script as my entrypoint that runs this command: commands: parse_manifest: args: parse --quiet executable: dbt_invoker
the command is in a transformers.meltano.yml file under my dbt folder
i
Copy code
commands:
      parse_manifest:
        args: parse --quiet
        executable: dbt_invoker
I use that as well after I saw one of your posts a while back
👍 1
Failed with the --no-cache flag in the docker build as well - same error. I'm just stuck on why the EXACT same image is working when pointed at one db but not the other.
Like, wouldn't some manifest parsing issue show itself when pointed at dev or stg too?
j
stupid idea - case sensitivy somewhere?
i
Valid idea I've had Snowflake throw fits about case sensitive stuff before but that's usually a column name. I'll check
I don't think that's it. The case is consistent in the manifest.json, snowflake db, meltano config. @Edgar Ramírez (Arch.dev) you have any ideas? Otherwise I'll wait for dbt 1.9 and hope it goes away with that lol.
j
'dbt_<snowflake://macros/catalog.sql>'
i assume that macro is nothing you made
i would tear out all the models, make it barebones as possible then try adding components back in? 🤔
✅ 1
i
yea i didnt make that macro. typically when I get some sort of '`macro'`error it ends up being something thats fixed in another dbt version. and the 
parse_manifest
has already been run since thats the preliminary step to serving dagster in my dockerfile - so idk why thats in the stack trace. The manifest has already been parsed for dagster and my model lineage is all the same, etc... I'll try that next though
e
when I swap the meltano env over to use prod config
do you mean literally the 
--environment
command line flag? Or how do you switch envs?
i
MELTANO_ENVIRONMENT='dev' > MELTANO_ENVIRONMENT='prod'
e
Gotcha, do you use 
MELTANO_ENVIRONMENT
anywhere expecting it to be interpolated, i.e. as 
${MELTANO_ENVIRONMENT}
or similar?
i
No I don't believe so
And I'm not getting this behavior with my staging environment in addition to dev
e
So only prod?
i
Yeah only prod. and a typical dbt:run works on the cli (when connected to container console) and a dbt debug returned no issue. I also checked the config with config list. I tried building the docker image with no cache
snowflake loader is working fine, only failing at dbt step in my job
e
i
So, perhaps add a dbt clean before I parse my manifest?
In my dockerfile
j
you can include it as part of your config
dbt_project.yml
This is my current setup
Copy code
clean-targets:
- ../.meltano/transformers/dbt/target/compiled
- ../.meltano/transformers/dbt/target/run
- dbt_packages
- logs
i
I might have found an issue but idk if this is it. One of my asset groups was missing in my dbt_project.yml. models in both groups reference another so perhaps since the group didnt exist in the project it was screwing up the manifest
Copy code
meltano:
  target:
curious - what do you have this default set to in your dbt_profiles.yml?
j
Copy code
meltano:
  target: "{{ env_var('MELTANO_ENVIRONMENT', 'dev') }}"
i
okay same just checking
I fixed it, it was the groups. Both my asset groups referenced each other and I think that was screwing up the manifest. Hopefully won't miss that again lol.
j
i love me some circular references
glad to hear you fixed it!
i
lol its always an obvious thing. still very weird it wasnt failing in dev or stg
red herring
4 Views
Open in Slack
PreviousNext
Powered by