https://linen.dev logo
Join Slack
Powered by
Hi all! When trying to specify custom PKs in my `...
# troubleshooting
s
Hi all! When trying to specify custom PKs in my 
meltano.yml
by following:
Copy code
metadata:
      foo-bar:
        table-key-properties: [foo1, foo2, foo3]
        replication-key: created_at
        replication-method: INCREMENTAL
I can see that, when I run 
meltano invoke --dump=catalog my-tap  > catalog.json
, I can see that the generated schema has the above PK’s in 
key_properties
, but when I run 
meltano --log-level debug run my-tap my-target
, the generated catalog has empty 
key-properties
. I am using 
tap-postgres
as my sink (meltanolabs variant) and the tap is redshift(sdk variant under dev) More logs in first comment
```2023-07-04T070915.029659Z [debug ] Visiting CatalogNode.STREAM at '.streams[0]'. 2023-07-04T070915.031024Z [debug ] Setting '.streams[0].selected' to 'False' 2023-07-04T070915.031229Z [debug ] Setting '.streams[0].selected' to 'True' 2023-07-04T070915.031504Z [debug ] Setting '.streams[0].table_key_properties' to '['foo1', 'foo2', 'foo3', 'created_at', 'updated_at']' 2023-07-04T070915.031649Z [debug ] Setting '.streams[0].replication_key' to 'created_at' 2023-07-04T070915.031769Z [debug ] Setting '.streams[0].replication_method' to 'INCREMENTAL' 2023-07-04T070915.032085Z [debug ] Skipping node at '.streams[0].tap_stream_id' 2023-07-04T070915.032543Z [debug ] Skipping node at '.streams[0].table_name' 2023-07-04T070915.032768Z [debug ] Skipping node at '.streams[0].replication_method' 2023-07-04T070915.032995Z [debug ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.foo1'. 2023-07-04T070915.033225Z [debug ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.foo2'. 2023-07-04T070915.033410Z [debug ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.foo3'. 2023-07-04T070915.034928Z [debug ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.created_at'. 2023-07-04T070915.035020Z [debug ] Skipping node at '.streams[0].schema.properties.created_at.format' 2023-07-04T070915.035104Z [debug ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.updated_at'. 2023-07-04T070915.035184Z [debug ] Skipping node at '.streams[0].schema.properties.updated_at.format' 2023-07-04T070915.035254Z [debug ] Skipping node at '.streams[0].schema.type' 2023-07-04T070915.035407Z [debug ] Skipping node at '.streams[0].is_view' 2023-07-04T070915.035515Z [debug ] Skipping node at '.streams[0].stream' 2023-07-04T070915.035616Z [debug ] Visiting CatalogNode.METADATA at '.streams[0].metadata[0]'. 2023-07-04T070915.035760Z [debug ] Visiting metadata node for tap_stream_id 'foo-bar', breadcrumb '['properties', 'foo1']' 2023-07-04T070915.035876Z [debug ] Setting '.streams[0].metadata[0].metadata.selected' to 'False' 2023-07-04T070915.035951Z [debug ] Setting '.streams[0].metadata[0].metadata.selected' to 'True' 2023-07-04T070915.036094Z [debug ] Skipping node at '.streams[0].metadata[0].breadcrumb[0]' 2023-07-04T070915.036198Z [debug ] Skipping node at '.streams[0].metadata[0].breadcrumb[1]' 2023-07-04T070915.036285Z [debug ] Skipping node at '.streams[0].metadata[0].metadata.inclusion' 2023-07-04T070915.036353Z [debug ] Skipping node at '.streams[0].metadata[0].metadata.selected' 2023-07-04T070915.036427Z [debug ] Visiting CatalogNode.METADATA at '.streams[0].metadata[1]'. 2023-07-04T070915.036500Z [debug ] Visiting metadata node for tap_stream_id 'foo-bar', breadcrumb '['properties', 'foo2']' 2023-07-04T070915.036638Z [debug ] Setting '.streams[0].metadata[1].metadata.selected' to 'False' 2023-07-04T070915.036724Z [debug ] Setting '.streams[0].metadata[1].metadata.selected' to 'True' 2023-07-04T070915.036839Z [debug ] Skipping node at '.streams[0].metadata[1].breadcrumb[0]' 2023-07-04T070915.036936Z [debug ] Skipping node at '.streams[0].metadata[1].breadcrumb[1]' 2023-07-04T070915.037064Z [debug ] Skipping node at '.streams[0].metadata[1].metadata.inclusion' 2023-07-04T070915.037163Z [debug ] Skipping node at '.streams[0].metadata[1].metadata.selected' 2023-07-04T070915.037266Z [debug ] Visiting CatalogNode.METADATA at '.streams[0].metadata[2]'. 2023-07-04T070915.037355Z [debug ] Visiting metadata node for tap_stream_id 'foo-bar', breadcrumb '['properties', 'foo3']' 2023-07-04T070915.037452Z [debug ] Setting '.streams[0].metadata[2].metadata.selected' to 'False' 2023-07-04T070915.037525Z [debug ] Setting '.streams[0].metadata[2].metadata.selected' to 'True' 2023-07-04T070915.037626Z [debug ] Skip…
Copy code
2023-07-04T07:09:15.050063Z [debug    ] Skipping node at '.streams[0].table_key_properties[0]'
2023-07-04T07:09:15.050131Z [debug    ] Skipping node at '.streams[0].table_key_properties[1]'
2023-07-04T07:09:15.050226Z [debug    ] Skipping node at '.streams[0].table_key_properties[2]'
2023-07-04T07:09:15.050305Z [debug    ] Skipping node at '.streams[0].table_key_properties[3]'
2023-07-04T07:09:15.050421Z [debug    ] Skipping node at '.streams[0].table_key_properties[4]'
2023-07-04T07:09:15.050537Z [debug    ] Skipping node at '.streams[0].table_key_properties[5]'
2023-07-04T07:09:15.050677Z [debug    ] Skipping node at '.streams[0].table_key_properties[6]'
2023-07-04T07:09:15.050873Z [debug    ] Skipping node at '.streams[0].table_key_properties[7]'
2023-07-04T07:09:15.050997Z [debug    ] Skipping node at '.streams[0].table_key_properties[8]'
2023-07-04T07:09:15.051175Z [debug    ] Skipping node at '.streams[0].table_key_properties[9]'
2023-07-04T07:09:15.051303Z [debug    ] Skipping node at '.streams[0].table_key_properties[10]'
2023-07-04T07:09:15.051428Z [debug    ] Skipping node at '.streams[0].table_key_properties[11]'
2023-07-04T07:09:15.051505Z [debug    ] Skipping node at '.streams[0].table_key_properties[12]'
2023-07-04T07:09:15.051609Z [debug    ] Skipping node at '.streams[0].replication_key'
2023-07-04T07:09:15.055954Z [debug    ] Invoking: ['/Users/me/Documents/opensource/tap-redshift/.meltano/extractors/tap-redshift/venv/bin/tap-redshift', '--config', '/Users/me/Documents/opensource/tap-redshift/.meltano/run/tap-redshift/tap.5087c065-6914-4c5a-9ef3-1036b6b7c841.config.json', '--catalog', '/Users/me/Documents/opensource/tap-redshift/.meltano/run/tap-redshift/tap.properties.json']
2023-07-04T07:09:15.121605Z [debug    ] Variable '$TARGET_POSTGRES_SCHEMA' is not set in the provided env dictionary.
2023-07-04T07:09:15.171055Z [debug    ] Variable '$TARGET_POSTGRES_SCHEMA' is not set in the provided env dictionary.
2023-07-04T07:09:15.172427Z [debug    ] Created configuration at /Users/me/Documents/opensource/tap-redshift/.meltano/run/target-postgres/target.338d7179-1b1f-496d-bd0d-4991b0847313.config.json
2023-07-04T07:09:15.175249Z [debug    ] Invoking: ['/Users/me/Documents/opensource/tap-redshift/.meltano/loaders/target-postgres/venv/bin/target-postgres', '--config', '/Users/me/Documents/opensource/tap-redshift/.meltano/run/target-postgres/target.338d7179-1b1f-496d-bd0d-4991b0847313.config.json']
2023-07-04T07:09:15.188743Z [debug    ] waiting for process completion or exception
2023-07-04T07:09:16.010935Z [info     ] 2023-07-04 12:39:16,010 | INFO     | tap-redshift         | Beginning incremental sync of 'foo-bar'... cmd_type=elb consumer=False name=tap-redshift producer=True stdio=stderr string_id=tap-redshift
2023-07-04T07:09:16.011317Z [info     ] 2023-07-04 12:39:16,010 | INFO     | tap-redshift         | Tap has custom mapper. Using 1 provided map(s). cmd_type=elb consumer=False name=tap-redshift producer=True stdio=stderr string_id=tap-redshift
2023-07-04T07:09:16.011654Z [info     ] {"type": "SCHEMA", "stream": "foo-bar", "schema": {"properties": {"foo1": {"type": ["integer", "null"]}, "foo2": {"type": ["integer", "null"]}, "foo3": {"type": ["integer", "null"]}, "created_at": {"format": "date-time", "type": ["string", "null"]}, "updated_at": {"format": "date-time", "type": ["string", "null"]}}, "type": "object"}, "key_properties": [], "bookmark_properties": ["created_at"]} cmd_type=elb consumer=False name=tap-redshift producer=True stdio=stdout string_id=tap-redshift
I have specified around 13 PKs, and trimmed the logs a bit to only show first few Not sure why these are getting skipped, because if I run 
meltano select tap-redshift --list
then I can see every col is selected.
Anybody has any idea on this?
e
Hey @silverbullet1! Can you try manually deleting the cached catalog in 
.meltano/run/tap-redshift/
? Related issues: • https://github.com/meltano/meltano/issues/2848 • https://github.com/meltano/meltano/issues/2856
s
Hi @edgar_ramirez_mondragon, it did not help. I deleted the entire .meltano folder and did a re-install of my extractor and loader. I can see the following line in my debug logs when doing 
meltano run …
2023-07-06T05:23:32.346509Z [debug    ] Cached catalog is outdated, running discovery...
but the generated catalog is same as my initial logs with empty 
key_properties
. 
meltano invoke --dump=catalog tap-redshift  > catalog1.json
is still generating the correct catalog.
Even if I specify the catalog with 
meltano elt tap-redshift target-postgres --catalog catalog.json
, it still doesn’t read the 
key_properties
from 
table_key_properties
in the catalog 🤔 Moreover, this is not a view, so the mapping should be there.
nevermind, i was specifying 
table-key-properties
instead of 
key-properties
, kinda confusing names
15 Views
Open in Slack
PreviousNext
Powered by