Ellis Valentiner
07/11/2025, 2:50 PMmeltano.yml
and ran into an unexpected problem in which an inline stream map glob is interfering with the stream selection.
My old yaml, which works, is like this (edited for brevity):
- name: tap-postgres--my-schema
inherit_from: tap-postgres
config:
filter_schemas:
- myschema
stream_maps:
table_1:
customer_id: int("${CUSTOMER_ID}")
...
table_999:
customer_id: int("${CUSTOMER_ID}")
select:
- myschema-table_1.col_1
- myschema-table_1.col_2
...
- myschema-table_999.col_50
- '!myschema-table_723.*'
metadata:
'*':
replication-method: INCREMENTAL
replication_key: last_modified_at
myschema-table_1:
key_properties: [customer_id, col_1]
myschema-table_2:
key_properties: [customer_id, col_1, col_2]
...
Note the line the line to deselect table 723. To simplify and save several hundred lines I tried to use a *
for my stream map like below:
- name: tap-postgres--my-schema
inherit_from: tap-postgres
config:
filter_schemas:
- myschema
stream_maps:
'*':
customer_id: int("${CUSTOMER_ID}")
select:
- myschema-table_1.col_1
- myschema-table_1.col_2
...
- myschema-table_999.col_50
- '!myschema-table_723.*'
metadata:
'*':
replication-method: INCREMENTAL
replication_key: last_modified_at
myschema-table_1:
key_properties: [customer_id, col_1]
myschema-table_2:
key_properties: [customer_id, col_1, col_2]
...
With this configuration, Meltano nearly immediately errors with StreamMapConfigError
due to Invalid key properties for 'myschema-table_723'
. The docs suggest stream maps are applied only after selection rules are applied but this appears to not be the case.
Can anyone confirm whether this is expected behavior? Does anyone know a more elegant way to coerce the meltano.yml into something more DRY?