https://linen.dev logo
Join Slack
Powered by
Morning! I’m having an issue with implementing par...
# troubleshooting
r
Morning! I’m having an issue with implementing parent-child streams right now. If I look at the output from running 
meltano elt tap-klaviyo target-postgres
, I can see that records are properly being created for each parent-child relationship, but for the child stream, only the last record is being recorded into target-postgres. This issue doesn’t happen with target-jsonl, where I see the entire output being recorded. I saw this thread a few years ago asking the same question, but it was never answered: https://meltano.slack.com/archives/C069CQNHDNF/p1634659098023900. Also, I attached a picture of the setup of my child stream. Anyone have any advice?
✅ 1
v
Sounds like State
How are you running this?
a
Could it be the primary key set on your child stream? Is 
id
the key for the child or the parent? Could you share the json output when using 
target-jsonl
?
r
How are you running this?
What do you mean by this? What command am I running?
➕ 1
Could it be the primary key set on your child stream? Is 
id
the key for the child or the parent?
id
is a column that’s outputted by both the parent and the child. It works like this: • Parent stream is called 
FlowActions
and has an 
id
which gets put into the child context as 
flow_action_id
• Child streams is called 
FlowMessages
and has an 
id
which should be the unique identifier for each its records
Here’s some sample output from running `meltano elt tap-klaviyo target-jsonl`:
Copy code
2024-04-03T15:31:35.068444Z [info     ] 2024-04-03 11:31:35,066 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "http_request_count", "value": 1, "tags": {"stream": "flowmessages", "endpoint": "/flow-actions/{flow_action_id}/flow-messages", "context": {"flow_id": "LC9eM2", "flow_action_id": "8845983"}}} cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.068611Z [info     ] 2024-04-03 11:31:35,066 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.8587641716003418, "tags": {"stream": "flowmessages", "context": {"flow_id": "LC9eM2", "flow_action_id": "8845983"}, "status": "succeeded"}} cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.068767Z [info     ] 2024-04-03 11:31:35,066 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 1, "tags": {"stream": "flowmessages", "context": {"flow_id": "LC9eM2", "flow_action_id": "8845983"}}} cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.068914Z [info     ] 2024-04-03 11:31:35,067 | INFO     | tap-klaviyo.flowmessages | Beginning full_table sync of 'flowmessages' with context: {'flow_id': 'LC9eM2', 'flow_action_id': '8846081'}... cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.069093Z [debug    ] {"type":"RECORD","stream":"flowmessages","record":{"id":"MUTWRu","attributes":{"name":"Sunset Flow: Email #1","channel":"Email","content":{"subject":"FIXD hasn't forgotten about you {{ first_name }}"},"created":"2019-10-18T19:59:40+00:00","updated":"2023-05-11T12:01:18+00:00"},"flow_id":"LC9eM2","flow_action_id":"8845983"},"time_extracted":"2024-04-03T15:31:35.065582+00:00"} cmd_type=extractor name=tap-klaviyo (out) run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stdout
2024-04-03T15:31:35.069299Z [debug    ] {"type":"STATE","value":{"bookmarks":{"flows":{"starting_replication_value":null},"flowactions":{"partitions":[{"context":{"flow_id":"LC9eM2"},"starting_replication_value":null}]},"flowmessages":{"partitions":[{"context":{"flow_id":"LC9eM2","flow_action_id":"8811105"}},{"context":{"flow_id":"LC9eM2","flow_action_id":"8811106"}},{"context":{"flow_id":"LC9eM2","flow_action_id":"8811107"}},{"context":{"flow_id":"LC9eM2","flow_action_id":"8811108"}},{"context":{"flow_id":"LC9eM2","flow_action_id":"8845983"}}]}}}} cmd_type=extractor name=tap-klaviyo (out) run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stdout
2024-04-03T15:31:35.069454Z [debug    ] {"type":"RECORD","stream":"flowactions","record":{"id":"8845983","attributes":{"action_type":"SEND_EMAIL","created":"2019-10-18T19:59:40+00:00","updated":"2024-03-26T06:44:43+00:00"},"flow_id":"LC9eM2"},"time_extracted":"2024-04-03T15:31:35.067048+00:00"} cmd_type=extractor name=tap-klaviyo (out) run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stdout
2024-04-03T15:31:35.069731Z [debug    ] {"type":"SCHEMA","stream":"flowmessages","schema":{"properties":{"id":{"type":["string"]},"flow_id":{"type":["string"]},"flow_action_id":{"type":["string"]},"attributes":{"properties":{"name":{"type":["string","null"]},"channel":{"type":["string","null"]},"created":{"format":"date-time","type":["string","null"]},"updated":{"format":"date-time","type":["string","null"]},"content":{"properties":{"subject":{"type":["string","null"]}},"type":["object","null"]}},"type":["object","null"]}}},"key_properties":["id"]} cmd_type=extractor name=tap-klaviyo (out) run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stdout
2024-04-03T15:31:35.209815Z [info     ] 2024-04-03 11:31:35,209 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.130808, "tags": {"stream": "flowmessages", "endpoint": "/flow-actions/{flow_action_id}/flow-messages", "http_status_code": 200, "status": "succeeded", "context": {"flow_id": "LC9eM2", "flow_action_id": "8846081"}}} cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.210045Z [info     ] 2024-04-03 11:31:35,209 | INFO     | singer_sdk.helpers.jsonpath | JSONPath $[data][*] match count: 1 cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.210130Z [info     ] 2024-04-03 11:31:35,209 | INFO     | tap-klaviyo.flowmessages | record is: {'type': 'flow-message', 'id': 'Qif8ut', 'attributes': {'name': 'Sunset Flow: Email #2', 'channel': 'Email', 'content': {'subject': 'Farewell From FIXD', 'preview_text': "We hope this is more 'see you later' than 'good-bye'", 'from_email': '<mailto:marketing-fixd@fixdapp.com|marketing-fixd@fixdapp.com>', 'from_label': 'FIXD Automotive', 'reply_to_email': '', 'cc_email': '', 'bcc_email': ''}, 'created': '2019-10-18T20:06:26+00:00', 'updated': '2023-05-11T12:01:39+00:00'}, 'relationships': {'flow-action': {'links': {'self': '<https://a.klaviyo.com/api/flow-messages/Qif8ut/relationships/flow-action/>', 'related': '<https://a.klaviyo.com/api/flow-messages/Qif8ut/flow-action/'}}>, 'template': {'data': {'type': 'template', 'id': 'XDY2JL'}, 'links': {'self': '<https://a.klaviyo.com/api/flow-messages/Qif8ut/relationships/template/>', 'related': '<https://a.klaviyo.com/api/flow-messages/Qif8ut/template/'}}}>, 'links': {'self': '<https://a.klaviyo.com/api/flow-messages/Qif8ut/'}}> cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.210282Z [info     ] 2024-04-03 11:31:35,209 | INFO     | tap-klaviyo.flowmessages | Checking the post_process record: {'type': 'flow-message', 'id': 'Qif8ut', 'attributes': {'name': 'Sunset Flow: Email #2', 'channel': 'Email', 'content': {'subject': 'Farewell From FIXD', 'preview_text': "We hope this is more 'see you later' than 'good-bye'", 'from_email': '<mailto:marketing-fixd@fixdapp.com|marketing-fixd@fixdapp.com>', 'from_label': 'FIXD Automotive', 'reply_to_email': '', 'cc_email': '', 'bcc_email': ''}, 'created': '2019-10-18T20:06:26+00:00', 'updated': '2023-05-11T12:01:39+00:00'}, 'relationships': {'flow-action': {'links': {'self': '<https://a.klaviyo.com/api/flow-messages/Qif8ut/relationships/flow-action/>', 'related': '<https://a.klaviyo.com/api/flow-messages/Qif8ut/flow-action/'}}>, 'template': {'data': {'type': 'template', 'id': 'XDY2JL'}, 'links': {'self': '<https://a.klaviyo.com/api/flow-messages/Qif8ut/relationships/template/>', 'related': '<https://a.klaviyo.com/api/flow-messages/Qif8ut/template/'}}}>, 'links': {'self': '<https://a.klaviyo.com/api/flow-messages/Qif8ut/'}}> cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.210496Z [debug    ] {"type":"RECORD","stream":"flowmessages","record":{"id":"Qif8ut","attributes":{"name":"Sunset Flow: Email #2","channel":"Email","content":{"subject":"Farewell From FIXD"},"created":"2019-10-18T20:06:26+00:00","updated":"2023-05-11T12:01:39+00:00"},"flow_id":"LC9eM2","flow_action_id":"8846081"},"time_extracted":"2024-04-03T15:31:35.209939+00:00"} cmd_type=extractor name=tap-klaviyo (out) run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stdout
2024-04-03T15:31:35.210684Z [info     ] 2024-04-03 11:31:35,210 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "http_request_count", "value": 1, "tags": {"stream": "flowmessages", "endpoint": "/flow-actions/{flow_action_id}/flow-messages", "context": {"flow_id": "LC9eM2", "flow_action_id": "8846081"}}} cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.210776Z [info     ] 2024-04-03 11:31:35,210 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.14228105545043945, "tags": {"stream": "flowmessages", "context": {"flow_id": "LC9eM2", "flow_action_id": "8846081"}, "status": "succeeded"}} cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.210846Z [info     ] 2024-04-03 11:31:35,210 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 1, "tags": {"stream": "flowmessages", "context": {"flow_id": "LC9eM2", "flow_action_id": "8846081"}}} cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.210914Z [info     ] 2024-04-03 11:31:35,210 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "http_request_count", "value": 1, "tags": {"stream": "flowactions", "endpoint": "/flows/{flow_id}/flow-actions", "context": {"flow_id": "LC9eM2"}}} cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.210977Z [info     ] 2024-04-03 11:31:35,210 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 1.936324119567871, "tags": {"stream": "flowactions", "context": {"flow_id": "LC9eM2"}, "status": "succeeded"}} cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.211037Z [info     ] 2024-04-03 11:31:35,210 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 6, "tags": {"stream": "flowactions", "context": {"flow_id": "LC9eM2"}}} cmd_type=extractor name=tap-klaviyo run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stderr
2024-04-03T15:31:35.211129Z [debug    ] {"type":"STATE","value":{"bookmarks":{"flows":{"starting_replication_value":null},"flowactions":{"partitions":[{"context":{"flow_id":"LC9eM2"},"starting_replication_value":null}]},"flowmessages":{"partitions":[{"context":{"flow_id":"LC9eM2","flow_action_id":"8811105"}},{"context":{"flow_id":"LC9eM2","flow_action_id":"8811106"}},{"context":{"flow_id":"LC9eM2","flow_action_id":"8811107"}},{"context":{"flow_id":"LC9eM2","flow_action_id":"8811108"}},{"context":{"flow_id":"LC9eM2","flow_action_id":"8845983"}},{"context":{"flow_id":"LC9eM2","flow_action_id":"8846081"}}]}}}} cmd_type=extractor name=tap-klaviyo (out) run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stdout
a
Sorry I meant the actual output into the .json file from running, but this is sufficient. So I can see two child record ids in here: 
Qif8ut
and 
MUTWRu
, what happens on the postgres side? Do you only see one of those?
The easiest way to compare would be a 
--full-refresh
on both, and share the output jsonl file here, together with a screenshot of the postgres output or similar. Worth checking the ddl for your postgres table (or drop the target table and start again) to make sure you don't have any remnants. Given that you're passing the parent id, it's possible your postgres table still has this has your PK, which would explain the 'keep the last one' behaviour. But I could be completely wrong of course!
v
The logs are very helpful
Copy code
2024-04-03T15:31:35.069731Z [debug    ] {"type":"SCHEMA","stream":"flowmessages","schema":{"properties":{"id":{"type":["string"]},"flow_id":{"type":["string"]},"flow_action_id":{"type":["string"]},"attributes":{"properties":{"name":{"type":["string","null"]},"channel":{"type":["string","null"]},"created":{"format":"date-time","type":["string","null"]},"updated":{"format":"date-time","type":["string","null"]},"content":{"properties":{"subject":{"type":["string","null"]}},"type":["object","null"]}},"type":["object","null"]}}},"key_properties":["id"]} cmd_type=extractor name=tap-klaviyo (out) run_id=3e8117d4-e5a4-4e54-9e93-e6801ec736fd state_id=2024-04-03T153106--tap-klaviyo--target-jsonl stdio=stdout
Lays out the issue. Specifically 
"key_properties":["id"]
Target-postgres uses that as your primary key and will upsert that data into your system. The tap should either provide an empty list of primary keys (if you want to append only) or you should override the primary key for that stream you care about to be empty. Or you should switch to a primary key that is actually unique for every record. I think you could probably find a better primary key from the tap but I"m not exactly sure which one you'd use. Easiest thing would probably be to just not specify a primary key I made an issue here for us to document this better https://github.com/MeltanoLabs/target-postgres/issues/329 . The context I think is missing here:
I'm not certain that's the issue but it smells like it
I'd have to deep dive more. If you could show the difference between that two it'd be helpful. I like to do 
meltano invoke tap-klavio > out
cat out | meltano invoke target-postgres
and 
cat out | meltano invoke target-jsonl
you'll figure out from that if it's state or the key properties thing we're talking about.
r
@visch I would think that 
id
is a unique key for each flow message. The non-unique keys would be 
flow_id
or 
flow_action_id
I’m going to try those commands now though
➕ 1
Just for clarity’s sake, here’s an example of two records outputted by the child stream (neither of these end up making it to the postgres DB):
Copy code
{'type': 'flow-message', 'id': 'Yw8vwt', 'attributes': {'name': 'Buying & Selling Used Cars: Email 4', 'channel': 'Email', 'content': {'subject': 'Ends TONIGHT at midnight!', 'preview_text': "Don't miss the #1 tool for used car shoppers", 'from_email': '<mailto:marketing-fixd@fixdapp.com|marketing-fixd@fixdapp.com>', 'from_label': 'FIXD Automotive', 'reply_to_email': '', 'cc_email': '', 'bcc_email': ''}, 'created': '2023-08-16T15:36:31+00:00', 'updated': '2023-08-16T18:46:52+00:00'}, 'relationships': {'flow-action': {'links': {'self': '<https://a.klaviyo.com/api/flow-messages/Yw8vwt/relationships/flow-action/>', 'related': '<https://a.klaviyo.com/api/flow-messages/Yw8vwt/flow-action/'}}>, 'template': {'data': {'type': 'template', 'id': 'SvPNnp'}, 'links': {'self': '<https://a.klaviyo.com/api/flow-messages/Yw8vwt/relationships/template/>', 'related': '<https://a.klaviyo.com/api/flow-messages/Yw8vwt/template/'}}}>, 'links': {'self': '<https://a.klaviyo.com/api/flow-messages/Yw8vwt/'}}>
---
Copy code
{'type': 'flow-message', 'id': 'VyhM88', 'attributes': {'name': 'FMH Q&A: Email #1', 'channel': 'Email', 'content': {'subject': 'Buying & Selling Used Cars in 2023', 'preview_text': 'Here's the video you requested', 'from_email': '<mailto:marketing-fixd@fixdapp.com|marketing-fixd@fixdapp.com>', 'from_label': 'FIXD Automotive', 'reply_to_email': '', 'cc_email': '', 'bcc_email': ''}, 'created': '2023-08-16T15:36:29+00:00', 'updated': '2023-08-16T16:35:51+00:00'}, 'relationships': {'flow-action': {'links': {'self': '<https://a.klaviyo.com/api/flow-messages/VyhM88/relationships/flow-action/>', 'related': '<https://a.klaviyo.com/api/flow-messages/VyhM88/flow-action/'}}>, 'template': {'data': {'type': 'template', 'id': 'XkHECB'}, 'links': {'self': '<https://a.klaviyo.com/api/flow-messages/VyhM88/relationships/template/>', 'related': '<https://a.klaviyo.com/api/flow-messages/VyhM88/template/'}}}>, 'links': {'self': '<https://a.klaviyo.com/api/flow-messages/VyhM88/'}}>
I’m a little confused, are you asking me to run this command as one: 
meltano invoke tap-klaviyo > out   cat out | meltano invoke target-postgres
v
1. meltano invoke tap-klaviyo > out 2. cat out | meltano invoke target-jsonl (Does the record make it to jsonl?) 3. cat out | meltano invoke target-postgres (Does the record make it to target-postgres?)
🫡 1
r
@Andy Carter I’m attaching the postgres and json output below
flowmessages.jsonlflowmessages_postgres_output.csv
v
really it still isn't getting inserted*? Can you shoot me the output file I'll run it on my side that doesn't make any sense
r
I’m running the command for the output file rn so I’ll send that in 5-10 One seeming red herring that I’m seeing is that every time we run the child stream again (i.e. a new 
flow_action_id
is passed down) it says `Beginning full table sync`:
Copy code
2024-04-04 10:05:06,103 | INFO     | tap-klaviyo.flowmessages | Beginning full_table sync of 'flowmessages' with context: {'flow_id': 'TMXqEa', 'flow_action_id': '51405799'}...
2024-04-04 10:05:06,237 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.121799, "tags": {"stream": "flowmessages", "endpoint": "/flow-actions/{flow_action_id}/flow-messages", "http_status_code": 200, "status": "succeeded", "context": {"flow_id": "TMXqEa", "flow_action_id": "51405799"}}}
2024-04-04 10:05:06,238 | INFO     | singer_sdk.helpers.jsonpath | JSONPath $[data][*] match count: 0
2024-04-04 10:05:06,238 | INFO     | tap-klaviyo.flowmessages | Pagination stopped after 0 pages because no records were found in the last response
2024-04-04 10:05:06,238 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "http_request_count", "value": 1, "tags": {"stream": "flowmessages", "endpoint": "/flow-actions/{flow_action_id}/flow-messages", "context": {"flow_id": "TMXqEa", "flow_action_id": "51405799"}}}
2024-04-04 10:05:06,239 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.13499212265014648, "tags": {"stream": "flowmessages", "context": {"flow_id": "TMXqEa", "flow_action_id": "51405799"}, "status": "succeeded"}}
2024-04-04 10:05:06,239 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"stream": "flowmessages", "context": {"flow_id": "TMXqEa", "flow_action_id": "51405799"}}}
2024-04-04 10:05:06,248 | INFO     | tap-klaviyo.flowmessages | Beginning full_table sync of 'flowmessages' with context: {'flow_id': 'TMXqEa', 'flow_action_id': '51405800'}...
2024-04-04 10:05:06,353 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.094187, "tags": {"stream": "flowmessages", "endpoint": "/flow-actions/{flow_action_id}/flow-messages", "http_status_code": 429, "status": "failed", "context": {"flow_id": "TMXqEa", "flow_action_id": "51405800"}}}
Another thing I’m going to try is making the primary key both the 
id
for the flow message and the 
name
. Maybe that’ll work better, though I don’t know why it’d produce a different result than just 
id
v
Based on your flow messages.jsonl I don't think it'll make a difference either
You'd be seeing a lot more messages in your table in postgres if that was the case
r
I feel like I must be just implementing parent-child streams incorrectly
Lemme just paste the full code for my parent-child streams
v
Please share the out file
I don't want to see the code (yet 😄 )
r
Code’s still running. One sec
out
out
👀 1
v
cat out | grep '{"type":"RECORD","stream":"flowmessages"' | wc
shows 206 lines which matches the postgres database, this just got cut off as I took like 6 min to do it instead of 5 lol
Here's me fumbling through this so you can see my imperfectness
tldr; it works for me
👍 1
a
@ryan_bell have you tried dropping the postgres table and running again from scratch to let meltano recreate the target table? Or if you have dbeaver viewing the ddl for the table? That should tell you if the PK(s) are configured properly, perhaps you changed the table config at some point?
r
I followed what you did and somehow still got the one record output in postgres 😂 here’s the 
target_output
file I created. You can see at the bottom it doesn’t the single record insert into 
flowmessages
target_output
@Andy Carter I have tried dropping it but ran into the same problem. Let me try wiping my venv and trying a fresh install
➕ 1
v
If that still doesn't work then share your meltano file and we can try it somewhere else, it's clear we're at a target-postgres related thing now (unless there's some other context missing)
r
😮‍💨 still no luck. You’re saying share my meltano.yml file?
v
Yeah I mean it's pretty clear now it's target-postgres, your output file is fine so the tap is good
So we have to figure out what's different between mine and yours, so it's either the target-postgres version your on, postgres, postgres state etc etc
r
That makes sense. My company has our own fork of target-postgres, so it might just be that it needs some updating. Here’s the github repo: https://github.com/fixdauto/target-postgres-fixd/tree/changeMeltano
v
ahh got it, yeah I"m using the meltano labs variant https://github.com/MeltanoLabs/target-postgres
r
Let me see if I can either figure out what changes I need to implement, or maybe I can just switch our project back over to using the MeltanoLabs variant
➕ 1
v
another option is to just add the meltano labs target-postgres as another target and use it just for klavio for now so you don't break any other things
r
Great idea
6 Views
Open in Slack
PreviousNext
Powered by