Is there a good primer on how to integrate an existing dbt project into Meltano? From what I can tell, it seems to be designed to have a limited set of transformations defined in the Meltano project, then run afterwards. What I’d like to do is perform:

meltano elt tap-salesforce target-snowflake —transform sources.Salesforce+

to run all models downstream of the loaded source, in my (one) dbt project. Is that possible?

You can use the models setting for that. When you run

meltano elt tap-salesforce target-snowflake --transform=run

Meltano will execute behind the scenes

dbt run --models \
    $MELTANO_TRANSFORM__PACKAGE_NAME \
    $MELTANO_EXTRACTOR_NAMESPACE \
    my_meltano_project

So you could tweak the setting in

meltano.yml

like

plugins:
  extractors:
  - name: salesforce
    namespace: Salesforce
  transformers:
  - name: dbt
    pip_url: dbt==0.19.1
    config:
      models: +sources.$MELTANO_EXTRACTOR_NAMESPACE

And how would I add my dbt repo as a package in that case?

you don't need to, since you can simply override the

models

setting as in the above

meltano.yml

example to only use a +something selector. If you still want to make your transform a package, you can make it a local one if you don't want to spin it out into its own repo

does my dbt project need to be nested in the

transforms

directory? the way i have it organized right now is:

.
.. my_dbt_project
.... dbt_project.yml
.... models/
.... etc.
.. my_meltano_project
.... meltano.yml
.... transforms
...... .gitkeep
.... extractors

it’s not clear to me what i should run with

meltano add transform dbt

. Does

my_dbt_project

need to be nested under my_meltano_project? Or do I need to add a

packages.yml

file in the

transforms

directory, and reference

../my_dbt_project

there?

btw I had the same questions and instead of figuring them out I just went with throwing my dbt project in transform/ directory and calling dbt directly. Honestly it'd be easier to use meltano invoke dbt to run the stuff (I think because of venvs) but it just seemed like the easiest route instead of figuring out the meltano route 😕

It seems like it would be useful to support referencing a remote location for dbt. Generally we'd want to have everything nested under meltano, but it might be good to support alternative structures to help with adoption.

should we create an issue?

The answer to that question @edgar_ramirez_mondragon is always yes (unless one already exists!) 😄 Would you mind opening it @edgar_ramirez_mondragon?

I'll do that!

oooo nice,

@visch yup, here are a few examples of what you can select: https://test-next--docs-getdbt-com.netlify.app/reference/model-selection-syntax#specifying-models-to-run

Ok, I'll give this a shot this weekend!

It's not clear to me if that setting requires it still to be w/in the Meltano project. I'm guessing you could give it an absolute path, but I don't know the requirements around having w/in a meltano project specifically