https://linen.dev logo
Join Slack
Powered by
i'm not sure about the good pratices to add a cust...
# plugins-dbt
e
i'm not sure about the good pratices to add a custom "dbt-tap-mycustomtap" with command line
d
If you want to add dbt model-based transformations to your project, you have two options: 1. directly add them under 
transform/models
2. create a separate dbt package in a separate repository, and manually add it as a 
transform
plugin in 
meltano.yml
, or directly to 
transform/packages.yml
Option 1 is easiest, unless you expect wanting to reuse these dbt models in multiple Meltano/dbt projects
Do you have experience with dbt and building custom models?
e
yes. i've already coded a custom dbt package
which is locally available (and in few days , will be on a custom remote git repo)
my question was to add a custom package with 
meltano add transform --custom dbt-my-customtap
d
Ah, OK. 
meltano add --custom transform
is not currently supported. If you've created your own dbt package, the best approach right now is to add it directly in 
transform/packages.yml
e
yup, this is what i've finaly done
also, it found that dbt does receive the target postgres var i've set, i had to add them in a .env file inserted to the docker container via the directive env_file in docker-compose
d
All right, I'm glad you figured it out. We have a few issues outstanding to improve our dbt integration and the documentation around it (e.g. https://gitlab.com/meltano/meltano/-/issues/2703), which we intend to get to later this month: https://meltano.com/docs/#may-2021
also, it found that dbt does receive the target postgres var i've set, i had to add them in a .env file inserted to the docker container via the directive env_file in docker-compose
Hmm, that shouldn't be necessary. How were you running 
dbt
? Just as part of 
meltano elt --transform=run
? The variables in https://gitlab.com/meltano/files-dbt/-/blob/master/bundle/transform/profile/profiles.yml should automatically be populated based on the configuration env vars for the loader: https://meltano.com/docs/integration.html#pipeline-environment-variables
e
for what i've seen, dbt wait for PG_ADDRESS , PG_DATABASE and so on ... and it seem that thoses var are not set from the ones of the loader which are POSTGRES_ADDRESS, POSTGRE_DATABASE ..etc..
i spent manyu hours to fix my setup ... and adding PG_ in .env of docker-compose was the good choice at the end
d
Definitely, that's a valid workaround. I'm just worried there's a bug in Meltano here, since 
PG_ADDRESS
should be populated as well: https://gitlab.com/meltano/meltano/blob/master/src/meltano/core/bundle/discovery.yml#L1404
5 Views
Open in Slack
PreviousNext
Powered by