https://linen.dev logo
Join Slack
Powered by
Hey :wave: I've been using `DBT_SOURCE_SCHEMA` env...
# plugins-general
d
Hey 👋 I've been using 
DBT_SOURCE_SCHEMA
env var in my dbt_project.yml together with target-postgres. But when I switched to using target-bigquery i got stuck since target-bigquery (adswerve) doesn't support schema. I've been eyeing pipelinewise-target-bigquery, but that doesn't support location.. Does anyone have any pointers?
d
@daniel_pettersson As I understand it, BigQuery calls schemas datasets, so as long as target-bigquery and dbt are both configured to look at the same dataset, you should be OK, and you may not need 
DBT_SOURCE_SCHEMA
at all!
Are you using your own dbt models or those provided by Meltano or another package? You'll need to make sure those are compatible with BigQuery as well
If you'd like to use 
DBT_SOURCE_SCHEMA
(which is dynamically set to the extractor namespace, e.g. 
tap_foo
) as the dataset, you'll have to make 2 more changes for it to be wired up correctly:
Copy code
meltano config target-bigquery set dataset_id '$MELTANO_EXTRACT__LOAD_SCHEMA'

meltano config target-bigquery set _target_schema '$TARGET_BIGQUERY_DATASET_ID'
That dynamically sets the BigQuery 
dataset_id
to the extractor's 
load_schema
extra, which matches its namespace by default, and then also dynamically sets the 
target_schema
extra, which will be used to get the value for 
DBT_SOURCE_SCHEMA
, to that same 
dataset_id
setting
d
Ah alright, that makes sense. The reason for using extractor namespace is to avoid clashes between table names, might be a premature worry though..
Thanks, gonna try it tomorrow!
d
Yep, makes sense. If you can confirm that the commands I suggested up there work as expected, I'd like to make setting the dataset dynamically the default behavior for target-bigquery, like we already do with target-postgres and target-snowflake schemas
Sleep well 🙂
d
Will get back to you, thanks again! 🙂
Seems to be working well 💯
d
@daniel_pettersson Awesome! I've created a new issue to make this the default behavior: https://gitlab.com/meltano/meltano/-/issues/2479
@daniel_pettersson I've created an issue to add support for BigQuery to Meltano's dbt integration as well: https://gitlab.com/meltano/files-dbt/-/issues/2 Assuming you've already got that set up, would you mind sharing your config in that issue, or even contributing it straight to 
profiles.yml
? 🙂
d
Great! I created a MR with an example of what I got set up https://gitlab.com/meltano/files-dbt/-/merge_requests/2
Open in Slack
PreviousNext
Powered by