Good day (again). Has anyone ever delt with nested...
# singer-tap-developments
Stéphane Burwash
04/14/2022, 2:42 PMGood day (again). Has anyone ever delt with nested data that contains the id, but we also want to access the rest of the data? something in the form of:
Copy code
{
identifiers{ id: 12345}
data: {all_that_good_shit: ...}
}visch
04/14/2022, 3:04 PMNested primary_keys / replication_keys and singer don't go together (maybe I'm wrong about this but that's my experience thus far). So you have to bring them to the top.
It'd be curious someday to have Singer's primary and replication keys use json paths as that'd solve this without having to touch the data
e
edgar_ramirez_mondragon
04/14/2022, 3:10 PMIt'd be curious someday to have Singer's primary and replication keys use json paths as that'd solve this without having to touch the data@visch That may be relatively easy to implement in the SDK 🙂. Would you mind filing an issue? I think the only place we'd need to update is https://gitlab.com/meltano/sdk/-/blob/241b38e8b6cdfa6f3397d9c70e7b9082b4594554/singer_sdk/helpers/_state.py#L219
v
visch
04/14/2022, 3:14 PMvisch
04/14/2022, 3:15 PMHow'd down stream targets know where to look? Are you thinking of having the sdk generate
_sdc_primary_keye
edgar_ramirez_mondragon
04/14/2022, 3:29 PMAh I completely missed the primary key bit 🤔.
Are you thinking of having the SDK generateThat is a good option. The SDK already appends context keys in child streams, for example.?_sdc_primary_key
s
Stéphane Burwash
04/14/2022, 4:02 PM@visch yours is much prettier than mine. I'm just trying to get some data out atm, so look at this baby 😉 :
Copy code
def parse_response(self, response: requests.Response) -> Iterable[dict]:
"""Parse the response and return an iterator of result rows."""
response.raise_for_status()
input = response.json()
for elem in input["results"]:
elem["id"] = elem["engagement"]["id"]
elem["lastUpdated"] = elem["engagement"]["lastUpdated"]
yield from extract_jsonpath(self.records_jsonpath, input=input)v
visch
04/14/2022, 4:04 PMcredit to @josh_lloyd