Hey y'all, suspect I'm doing something wrong here: I have N upstream Postgres databases and would like to ETL their outputs into N different schemas in Snowflake. I thought I could use the

inherit_from

trick for each of the extractors/loaders to share some common base config in a single

meltano.yml

file, but that seemed to blow up the size of my install/docker container because Meltano seems to be installing N different copies of the dependent libraries for each of the

inherit_from

extractors/loaders. Am I doing this wrong? Should I just have a single

tap-postgres

extractor and then run N different meltano jobs for each of the different DBs I need it to do extraction from? And how should I dynamically adjust the target schema in Snowflake for each one-- is it like setting the

TARGET_SNOWFLAKE_DEFAULT_TARGET_SCHEMA

differently for each job?

@josh_wills You're doing all the things right. This issue describes our proposed improvement to reuse the install rather than repeat the install per each

inherit_from

instance.

I do what you've described and it looks like this:

- name: target-snowflake
    variant: transferwise
    config:
      account: my_account.us-east-1
      user: my_username
      snowflake_role: LOADER
      dbname: RAW
      warehouse: LOADING
      file_format: RAW.CB_CORE.CSV
      default_target_schema: NO_MANS_LAND
      logging_level: DEBUG
      batch_size_rows: 1000000
      add_metadata_columns: true
      primary_key_required: false
      query_tag: meltano-loader-schema-table
  - name: target-snowflake-cb_chatlog
    inherit_from: target-snowflake
    config:
      default_target_schema: CB_CHATLOG
  - name: target-snowflake-cb_photos
    inherit_from: target-snowflake
    config:
      default_target_schema: CB_PHOTOS

Wanted to close the loop on this one: I ended up implementing my default postgres tap and then writing a shell script to update/override the environment variables that needed to change for the different dbs and then calling

meltano elt

for the tap/target combo; it was an easier hack than either a) blowing up the size of my container by using lots of

inherit_from

blocks or b) fixing Meltano to not duplicate the install of the taps based on the pip_url (sounds like a hella handy feature that I sadly do not have the time to implement at the moment!)