Hi, (may be this has been asked before) I would like to add second extractor to my setup, but the problem is the loader is still the same (target-bigquery) as I have with first extractor but with only difference being different "dataset_id". does that mean I have to re-add target-bigquery loader with different name? Thank you

I've just set this up. I'm using the load_schema from the extractor, which is the namespace by default eg

tap-extractor

meltano config target-bigquery set dataset_id '$MELTANO_EXTRACT__LOAD_SCHEMA'

meltano config target-bigquery set _target_schema '$TARGET_BIGQUERY_DATASET_ID'

Thank you @ambitious-lamp-95667 , so just to confirm:

meltano config tap-shopify set _load_schema shopify_data
meltano config tap-recharge set _load_schema recharge_data

meltano config target-bigquery set dataset_id '$MELTANO_EXTRACT__LOAD_SCHEMA'

With these settings when i create two pipeline (one for shopify and one for recharge) it will create "shopify_data" and "recharge_data" datasets in bigquery. Also I am not sure, do i need to use?

meltano config target-bigquery set _target_schema '$TARGET_BIGQUERY_DATASET_ID'

Yes exactly, you also need the

_target_schema

on target-bigquery

@ambitious-lamp-95667 thank you for your help. I got it working as I needed it 🙂

Cool! Happy to help

Note that as of https://gitlab.com/meltano/meltano/-/merge_requests/1981, setting

_target_schema

and

dataset_id

as above is already the default behavior 🙂

_load_schema

also doesn't need to be explicitly set on the extractors since it defaults to the namespace (e.g.

tap_shopify

and

tap_recharge

) : https://meltano.com/docs/plugins.html#load-schema-extra Of course, if you prefer the

_data

suffixes, using

load_schema

as you are is great.

A week or two away feels like years, so many new things 🙂 👍