I have a potentially silly question here that I've been trying to come up with for an answer. At this point my current stack is Meltano/DBT/Snowflake/Dagster. I have been able to successfully use Dagster to parallelize run Meltano to E+L my SQL tables from MSSQL into Snowflake, and then load seeds + models into DBT. This all happens in a Docker container. Dagster relies on using DBT's

manifest.json

file to populate the lineage of its assets. However let's say that we are starting with a fresh database to ELT to, every time I try to generate the

manifest.json

with let's say

meltano invoke dbt-snowflake:docs-generate

, I get errors because Snowflake doesn't have the models in place yet. Example:

21:00:37  Encountered an error:
Runtime Error
  Compilation Error in model stg_core_db__attribute_definition_name_localization (models\staging\core_db\localizations\stg_core_db__attribute_definition_name_localization.sql)
    In get_column_values(): relation CAMIS.Camis.stg_core_db__locales does not exist and no default value was provided.

    > in macro statement (macros\etc\statement.sql)
    > called by macro default__get_column_values (macros\sql\get_column_values.sql)
    > called by macro get_column_values (macros\sql\get_column_values.sql)
    > called by macro localization_table_generate (macros\localization_table_generate.sql)
    > called by model stg_core_db__attribute_definition_name_localization (models\staging\core_db\localizations\stg_core_db__attribute_definition_name_localization.sql)

The only way I have been able to generate the manifest on an empty DB with no raw data yet is if I run

meltano invoke dbt-snowflake:seed

, but I want Dagster to do that, not me manually. But if there is no manifest file, Dagster crashes.

You can see an example in the code from https://docs.dagster.io/integrations/dbt/using-dbt-with-dagster/load-dbt-models#step-4-understand-the-python-code-in-your-dagster-project - basically it runs

dbt --quiet parse

to generate a manifest.json which I believe doesn’t even require a database connection

I think that worked!

dbt-snowflake

doesnt have the parse command set up out of the box but I was able to get around that by configuring a custom command

plugins:
  utilities:
  - name: dbt-snowflake
    variant: dbt-labs
    # <https://docs.getdbt.com/docs/dbt-versions/core>
    pip_url: dbt-core~=1.7.0 dbt-snowflake~=1.7.0 git+<https://github.com/meltano/dbt-ext.git@main>
    commands:
      parse_manifest:
        args: parse --quiet
        executable: dbt_invoker