Hi! I have multiple datasources all of which I need to load into BigQuery, each datasource to its own BQ dataset. In pipelinewise target schema (target dataset in BQ) is configured in the same config file as tap. In Meltano it seems that I have to create

target-bigquery-something

for each dataset with overwritten

default_target_schema

I don't like that, because it increases complexity in my opinion. Is it the best way to configure EL in Meltano? Or am I missing something?

Meltano has a special environment variable which passes a "load schema" from the tap to the target. For instance, this would be

tap_github

for tap-github. I'm not at my computer to find the exact docs link, but you might be able to find more info searching "load schema" in the docs.

I found the docs. Seems to be exactly what I need, thanks!

Glad to hear it. Thanks!

@aaronsteers is it possible to override

load_schema

in environment layer? does not seem to work for me

For a single target? Is this because you want the same target schema for each tap, or you just want it to be different from the default mapping? Or is it that the target doesn't seem to do the auto-schema names as expected? Would be helpful to better understand the use case. Thanks.

The setup I need: dev: • mongodb_eu -> dev_raw_mongo_eu • mongodb_bra -> dev_raw_mongo_bra prod: • mongodb_eu -> very_specific_historical_name • mognodb_bra -> raw_mongo_bra

Really? Nice!

So far it seems to do the trick. And I feel that approach is quite flexible. Thanks again!