heya- been tinkering with meltano a bit lately to see how it might work for me and was curious about how people are incorporating dbt tests into their projects... so with that I have two questions! #1: first, it seems there's a bug with

meltano invoke dbt

where the wrapper isn't providing enough env vars:

root@1d67acfca30b:/project# meltano invoke dbt test
Running with dbt=0.16.1
Encountered an error:
Compilation Error
  Env var required but not provided: 'DBT_TARGET'

after some hacking, I found that this works... but it's gross:

root@66ebf1065348:/project# DBT_TARGET=postgres PG_ADDRESS=postgres PG_PORT=5432 PG_USERNAME=... PG_PASSWORD=... PG_DATABASE=... meltano invoke dbt test

is this a bug? #2: second, overall none of the docs or examples seem to include any examples of running dbt tests. personally I would want to have

dbt test

invocations running as part of every

meltano schedule run

or something similar... any guidance on what people are doing to ensure sql tests are baked deeply into their pipelines?

Hi @neville_samuell, welcome to the community! As you're finding out, Meltano's dbt integration is not as strong as it could (should?) be. We're tracking a few outstanding issues and opportunities for improvement in https://gitlab.com/groups/meltano/-/epics/82, but unfortunately I haven't been able to get to that epic yet. In https://gitlab.com/meltano/meltano/-/issues/2335, a user wrote up some thoughts on including

dbt snapshot

in

meltano elt

pipelines, and I'd love to see a similar issue for

dbt test

so that we can start discussing what that should look like 🙂 In https://gitlab.com/meltano/meltano/-/issues/2454, we've been talking about including Great Expectations as a

validator

plugin, which could then run at the end of a pipeline, and it would make sense for

dbt test

to also fill that role, but since

dbt

is currently a

transformer

plugin, we'll have to think about how we can make it fill both roles from an architectural perspective. All of that is to say that Meltano doesn't currently have an answer to question #2, but that I'd love to start finding a solution in a new issue! As for #1, the problem is that in order to set the

DBT_TARGET

and

SF_*

env vars, Meltano needs to know the loader (and extractor)

dbt

is running in context of, which is readily available when it runs as part of a

meltano elt

/`meltano schedule run` pipeline: https://meltano.com/docs/integration.html#pipeline-environment-variables, but not when using

meltano invoke

by itself. The cleanest way to resolve this would be to add a native way to run

dbt test

as part of

meltano elt

(as discussed above), but as a workaround, we could also let a schedule be specified on

meltano invoke

to set the pipeline environment:

meltano invoke --schedule=foo-to-postgres dbt ...

. I'd also welcome an issue to discuss that approach some more 🙂

gotcha- thanks that makes sense (those env vars being pipeline-specific)

I'd love to see issues for 1) Native support for running

dbt test

as part of

meltano elt

and 2) Running

meltano invoke

in the context of a pipeline

kk

But starting with a broad issue that I can close in favor of those 2 new ones works too 🙂

and then if you want some broad feedback from a dbt fan I could comment on the dbt epic? I wouldn't want to necessarily discourage the "great expectations" validator, but IMO I would much rather encourage the use of dbt to transform & test

Haha! Yes, please do comment on the dbt epic! I agree that "dbt test" has priority over GE support, but in the end we want to support all the popular plugins and not be too prescriptive 🙂

yeah my bias is too obvious

@neville_samuell Thanks!