Good morning; I’m just starting to experiment with meltano to replace some in-house tools, and starting with a postgres tap.

Hi @steve_ivy! Meltano supports overriding metadata on a per-table and per-field basis: https://meltano.com/docs/plugins.html#metadata-extra. Is that what you mean?

I’m… not sure? But I’ll check it out

You mean this

tables:
      - table_name: "table_one"
        replication_method: "INCREMENTAL"   # One of INCREMENTAL, LOG_BASED and FULL_TABLE
        replication_key: "last_update"      # Important: Incremental load always needs replication key

        # OPTIONAL: Load time transformations
        #transformations:
        #  - column: "last_name"            # Column to transform
        #    type: "SET-NULL"               # Transformation type

      # You can add as many tables as you need...
      - table_name: "table_two"
        replication_method: "LOG_BASED"     # Important! Log based must be enabled in MySQL

?

yes, rather than the default behavior to try and sync every table in the db. we have a very large db with many tables - and looking for how to create multiple taps with different groups of tables, and have more control over the replication methods and keys

oh so that's what Meltano uses inheritance metadata for. For example, the config above would be added in

meltano.yml

as

plugins:
  extractors:
    # Base tap
    - name: tap-postgres
      load_schema: repl_pg_public
      select:
        - "public.table_one"
        - "public.table_two"
      metadata:
        public-table_one:
          replication_method: INCREMENTAL
          replication_key: last_update
        public-table_two:
          replication_method: LOG_BASED

If you wanted to split the tables in your db between multiple taps with different configuration, the easiest wa is to use plugin inheritance

i did see the inheritance stuff, which makes sense… what you pasted above is new. would that limit that tap to syncing those tables, or just specify config only for those tbales?