Wondering if anyone has run into this issue. The t...
# troubleshootinga
adam_roderick
10/28/2021, 4:21 AMWondering if anyone has run into this issue. The tap has a nested object, but not all records have all properties for the object. The target is target-csv, which flattens all nested objects. The result is inconsistent field counts in the output!
adam_roderick
10/28/2021, 4:21 AMfor example
adam_roderick
10/28/2021, 4:22 AMtap's output is something like
Copy code
// record 1
{
"id": 123,
"nestedObj": {
"A": "aaa",
"B": "bbb"
}
}
// record 2
{
"id": 987,
"nestedObj": {
"B": "bbb"
}
}adam_roderick
10/28/2021, 4:23 AMtarget-csv's output is
Copy code
id,nestedObj__A,nestedObj__B
123,aaa,bbb
987,bbbadam_roderick
10/28/2021, 4:23 AMThey seem incompatible. Anyone have suggestions on how to resolve something like this?
adam_roderick
10/28/2021, 4:24 AM@chris_marchetti
v
visch
10/28/2021, 12:47 PMHit something similar to this here https://github.com/MeltanoLabs/target-athena/issues/27 . @nicholas_degiacomo
We ended up going with jsonl instead https://github.com/MeltanoLabs/target-athena/issues/27#issuecomment-952144716
visch
10/28/2021, 12:48 PMNot the same issue though, but I think we're talking about the same target-s3 as target-athena inherits from https://github.com/transferwise/pipelinewise-target-s3-csv
visch
10/28/2021, 12:51 PMThe real issue is there's a bug in the denesting for target-s3's csv "serialization" portion.
I personally try to stay way away from csv's
re https://garrens.com/blog/2017/10/09/spark-file-format-showdown-csv-vs-json-vs-parquet/
a
adam_roderick
10/28/2021, 1:36 PMYes, I believe this is the same issue