When extracting data from mongo, can we map the so...
# getting-started
m
When extracting data from mongo, can we map the source schema and the result we want to get? For example, I have a nested field
{level1: {level2: {location: [latitude, longitude]}, ...},. ...}
I want to get latitude and longitude as separate columns. Tried
flattening_enabled
but then it returns location and an array. Is it possible to map the format I want somehow or should I just extract and load as jsonb then do the processing with dbt?
a
Arrays are a different data structure. Implicit in an array is that it is variable-length. So it would need to be unpacked by bespoke business specific code. There is no logical flattening that unpacks an N-length array (in a way that works for all users generically w/o risk of janky outcomes). Personally I recommend transformation after the fact, with or without dbt. Flattening can simplify the sql but keeping the document as-is is nice too.
m
I have been trying stream maps but couldn't make it work. This type of mapping would be cool:
Copy code
latitude: level1['level2']['location'][0]
a
Ya I feel like that should work but on the flip side, under the hood stream maps are transforming BOTH the record and the schema message. So I am not sure if it is choking on how to transform the schema with this. I dunno.