https://linen.dev logo
Join Slack
Powered by
Hi <@U06C5FMMCQN> ! I was running my Meltano proje...
# plugins-general
r
Hi @peter_kosztolanyi ! I was running my Meltano project on Friday just fine, but today when I am trying to run a pipeline I am getting this error from the target-postgres target:
Copy code
target-postgres                 | Traceback (most recent call last):
target-postgres                 |   File "/project/.meltano/loaders/target-postgres/venv/bin/target-postgres", line 8, in <module>
target-postgres                 |     sys.exit(main())
target-postgres                 |   File "/project/.meltano/loaders/target-postgres/venv/lib/python3.6/site-packages/target_postgres/__init__.py", line 373, in main
target-postgres                 |     persist_lines(config, singer_messages)
target-postgres                 |   File "/project/.meltano/loaders/target-postgres/venv/lib/python3.6/site-packages/target_postgres/__init__.py", line 133, in persist_lines
target-postgres                 |     primary_key_string = stream_to_sync[stream].record_primary_key_string(o['record'])
target-postgres                 |   File "/project/.meltano/loaders/target-postgres/venv/lib/python3.6/site-packages/target_postgres/db_sync.py", line 337, in record_primary_key_string
target-postgres                 |     flatten = flatten_record(record, self.flatten_schema, max_level=self.data_flattening_max_level)
target-postgres                 |   File "/project/.meltano/loaders/target-postgres/venv/lib/python3.6/site-packages/target_postgres/db_sync.py", line 148, in flatten_record
target-postgres                 |     for k, v in d.items():
target-postgres                 | AttributeError: 'list' object has no attribute 'items'
Have you ever seen this before? Any chance you'd be able to help me out? Thank you!
d
@ricky_renner The error suggests that 
o['record']
is a 
list
rather than a 
dict
here: https://github.com/transferwise/pipelinewise-target-postgres/blob/1f4f47b64748c9c79e741bec8af605aec8fa7b20/target_postgres/__init__.py#L133 That indicates that the target received a 
RECORD
message from the tap with an invalid 
record
value. What tap are you running with?
r
I'm running it with the 
tap-spreadsheets-anywhere
d
@ricky_renner If you run in debug mode (https://meltano.com/docs/command-line-interface.html#debugging), can you identify the RECORD message that is triggering the error?
r
Copy code
tap-spreadsheets-anywhere (out) | {"type": "RECORD", "stream": "invoice", "record": [{"number": 133343, "ticketnum": 194979, "freight": 0.0, "stotal": 26235.24, "misc": 0.0, "total": 26235.24, "idate": "2019-01-03", "tax": 0.0, "itype": "Inv_Stock", "purchitems": 0.0, "taxamount_total": 0.0, "customernumber": 468, "_smart_source_bucket": "<gs://brook-and-whittle>", "_smart_source_file": "data-warehouse/data-extraction/connecticut/invoice/invoice.csv", "_smart_source_lineno": 89}]}
Might be this one? This is the last message logged
d
Yep, that's invalid since the 
record
needs to be a single dict, not an array of dict(s)
I suggest debugging tap-spreadsheets-anywhere to figure out where the list is coming from
r
Okay
@douwe_maan that would be in the same exact pipeline log, correct?
@eric_simmerman have you ever seen this before from the 
tap-spreadsheets-anywhere
?
e
I haven’t seen that one. If you can reduce it to a testcase in a PR - that’d be great. I see you’re parsing a CSV so at first guess I’d look at the delimiters and newlines for that line (line 89) - anything look off?
r
No nothing looks off 😕
Just so I understand - the 
tap-spreadsheets-anywhere
is spitting out a line from the CSV as a list even though each line in the CSV file is just one record? And it must be because of some format in the CSV? It's just weird because these same CSV's were working on Friday...
d
@ricky_renner Some changes were made to the tap over the weekend: https://github.com/ets/tap-spreadsheets-anywhere/commits/master. Is it possible the tap got updated (reinstalled) in your project and the behavior you're seeing is a bug caused by these changes?
r
Potentially. I actually just targeted the last commit before Friday to check that
d
r
I'm wondering if it's this line
d
@ricky_renner Ah yes, that's definitely it!
I think we should keep 
write_record
(singular) and drop the 
[]
r
@eric_simmerman can you confirm this when you get a sec? I can open up a PR
e
Sure I’ll take a look. If it’s really what you’ve pointed to then my test suite has certainly failed me here and needs to be made more robust.
oic - backing out the Transformer work that caused issues with Smart columns…I accidentally dropped the ‘s’ I’ll roll with Douwe’s recommendation.
@ricky_renner I just pushed a fix which includes a test for the singer.write output. That was a bit tricky because I had to capture stdout for the verification - but it’s working now. The test fails with the error you encountered but passes now that I’ve made the change @douwe_maan recommended.
r
Thank you @eric_simmerman!
Open in Slack
PreviousNext
Powered by