Help! Based on docs <https://share.getcloudapp.com...
# announcements
n
d
@nehil_jain
meltano add
does not directly support adding dbt hub packages by name. Rather, since your project's
transform
directory contains a full-fledged dbt project, you can add a DBT hub package exactly as described on https://hub.getdbt.com/fivetran/github_source/latest/: by adding it to a
packages.yml
file in that directory 🙂 You'll also probably want to add it to your
dbt_project.yml
file under
models
. When you run
meltano add transform <name>
, that specifically refers to the "known" transforms listed here: https://gitlab.com/meltano/meltano/blob/master/src/meltano/core/bundle/discovery.yml#L733 When you add one of those using that command, it will be added to your
meltano.yml
, and it will be added to
transform/packages.yml
and
transform/dbt_project.yml
automatically.
Note though that it's unlikely that the
github_source
dbt model will work with the
tap-github
extractor without some (many?) changes, since it was built specifically for the schema output by Fivetran's GitHub extractor(+loader), not
tap-github
Also note that as part of
meltano elt
,
dbt run
will automatically be invoked with arguments (profile, models, etc) determined by config settings listed here: https://gitlab.com/meltano/meltano/blob/master/src/meltano/core/bundle/discovery.yml#L710 As mentioned under https://meltano.com/docs/command-line-interface.html#pipeline-environment-variables, the value of
models
(which models to run, in your case:
github_source
, or the name of your
tap-github
-compatible fork) is determined automatically based on the value of that setting, which by default has the value
$MELTANO_EXTRACTOR_NAMESPACE my_meltano_model
, which expands to
tap_github my_meltano_model
when the
tap-github
extractor is used. ``my_meltano_project`` refers to models defined project-locally in your
transform/models/my_meltano_project
directory.
In your case, you'll want to override it specifically on a per-pipeline basis using the
DBT_MODELS
environment variable:
DBT_MODELS=github_source meltano elt ...
, or you can override the setting for all pipeline runs in
meltano.yml
:
meltano config dbt set models github_source
@nehil_jain Let me know if that helps! This all deserves to be much better documented, and going through this with you will help me figure out where the gaps in the docs are, and what isn't as obvious or flexible as it could be 🙂