I'm trying to simplify my `meltano.yml` and ran in...
# troubleshooting
e
I'm trying to simplify my
meltano.yml
and ran into an unexpected problem in which an inline stream map glob is interfering with the stream selection. My old yaml, which works, is like this (edited for brevity):
Copy code
- name: tap-postgres--my-schema
    inherit_from: tap-postgres
    config:
      filter_schemas:
      - myschema
      stream_maps:
        table_1:
          customer_id: int("${CUSTOMER_ID}")
        ...
        table_999:
          customer_id: int("${CUSTOMER_ID}")
    select:
    - myschema-table_1.col_1
    - myschema-table_1.col_2
    ...
    - myschema-table_999.col_50
    - '!myschema-table_723.*'
    metadata:
      '*':
        replication-method: INCREMENTAL
        replication_key: last_modified_at
      myschema-table_1:
        key_properties: [customer_id, col_1]
      myschema-table_2:
        key_properties: [customer_id, col_1, col_2]
      ...
Note the line the line to deselect table 723. To simplify and save several hundred lines I tried to use a
*
for my stream map like below:
Copy code
- name: tap-postgres--my-schema
    inherit_from: tap-postgres
    config:
      filter_schemas:
      - myschema
      stream_maps:
        '*':
          customer_id: int("${CUSTOMER_ID}")
    select:
    - myschema-table_1.col_1
    - myschema-table_1.col_2
    ...
    - myschema-table_999.col_50
    - '!myschema-table_723.*'
    metadata:
      '*':
        replication-method: INCREMENTAL
        replication_key: last_modified_at
      myschema-table_1:
        key_properties: [customer_id, col_1]
      myschema-table_2:
        key_properties: [customer_id, col_1, col_2]
      ...
With this configuration, Meltano nearly immediately errors with
StreamMapConfigError
due to
Invalid key properties for 'myschema-table_723'
. The docs suggest stream maps are applied only after selection rules are applied but this appears to not be the case. Can anyone confirm whether this is expected behavior? Does anyone know a more elegant way to coerce the meltano.yml into something more DRY?