Hello guys! G’DAY! just wonder anyone has met this issue (

UndefinedColumn: column "hash_key" named in key does not exist cmd_type=loader

) before, regarding tap-dynamo-totarget-postgres. I got this err from yesterday, relevant chat link here https://meltano.slack.com/archives/C01TCRBBJD7/p1661400699522009. I have check my local dynamodb and found we already have hash_key and range_key configured. Currently stuck here, any clues or guidance would be much appreciated! Thank you 🙏

The dynamo source may require you to override the column schema. Have you done any customizing of the Singer catalog as of yet?

Thanks AJ, no I havent, I am fairly new to meltano do you have any examples or instructions for me to follow? Thanks a lot!

I'm away from my PC right now but I will see what I can dig up!

Thanks mate! 🙏

As tap complexity in the Singer ecosystem goes, DynamoDB is an 11 out of 10, exactly for the reason I mention above. Once you have it set up though, it should be stable without much issue. It's just the first time setup that can be super frustrating.

yeah I hear that, but quite a fun of exploring solutions one by one within this fantastic slack community 😁

AJ is correct. I just reviewed the debug log again that you sent yesterday and I can now see the

SCHEMA

line which just says

{"type":"object"}

...:

2022-08-25T06:26:10.636381Z [info     ] INFO Starting sync.            cmd_type=extractor name=tap-kamprad run_id=8a128b5d-12db-484c-aa61-1d1e894337a8 state_id=2022-08-25T062605--tap-kamprad--target-warehouse-postgres stdio=stderr
2022-08-25T06:26:10.636990Z [debug    ] {"type": "STATE", "value": {}} cmd_type=extractor name=tap-kamprad (out) run_id=8a128b5d-12db-484c-aa61-1d1e894337a8 state_id=2022-08-25T062605--tap-kamprad--target-warehouse-postgres stdio=stdout
2022-08-25T06:26:10.637235Z [debug    ] {"type": "SCHEMA", "stream": "ktest", "schema": {"type": "object"}, "key_properties": ["hash_key", "range_key"]} cmd_type=extractor name=tap-kamprad (out) run_id=8a128b5d-12db-484c-aa61-1d1e894337a8 state_id=2022-08-25T062605--tap-kamprad--target-warehouse-postgres stdio=stdout

The original Gitlab issue is here: https://gitlab.com/meltano/meltano/-/issues/2216 Douwe's implementation of the feature here: https://gitlab.com/meltano/meltano/-/merge_requests/1843/diffs And the relevant Meltano Docs link for the schema extra which is how the source schema of your DynamoDB table needs to be announced to meltano due to the limitation of the tap being able to inspect its source for dynamic catalog (schema) discovery: https://docs.meltano.com/concepts/plugins#schema-extra

Thanks a lot Chris 🙏 I will have a look

In order to manually craft the JSON schema for a DynamoDB source, you can take an amount of sample records from your DynamoDB table as a json file and run it through

genson

to generate a JSON Schema from the sample data: https://github.com/wolverdude/GenSON And then you can you can use that generated JSON schema to configure the

schema

extra for your stream in the meltano.yml

Hey Christoph, do you know why this happened? I now can generate table on target postgres with no data in but got this info, not sure if it is the root cause

[info     ] INFO Unknown replication method: None for stream ktest

, I actually added it (FULL_TABLE)to catalog.json file

My guess would be that the replication method configuration metadata should still remain as per normal in

meltano.yml

and not in the

catalog.json

which is funny, from these logs

2022-08-26T11:00:45.535353Z [debug    ] Setting '.streams[0].metadata[0].metadata.selected' to 'False'

2022-08-26T11:00:45.535419Z [debug    ] Setting '.streams[0].metadata[0].metadata.selected' to 'True'

2022-08-26T11:00:45.535484Z [debug    ] Setting '.streams[0].metadata[0].metadata.replication-method' to 'FULL_TABLE'

2022-08-26T11:00:45.535581Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.table-key-properties[0]'

2022-08-26T11:00:45.535655Z [debug    ] Skipping node at '.streams[0].metadata[0].metadata.table-key-properties[1]'

I found I should put metadata attribute at the position of first index in metadata array, and now it works basically 🤣