is there a reason that the meltano variant of target-snowflake chooses to flatten objects instead of loading them into a variant column?

I’m not sure. @douwe_maan do you have any insight into that?

https://gitlab.com/meltano/target-snowflake#implementation-notes says:

We unnest Nested JSON Data Structures and follow a

[object_name]__[property_name]

approach similar to what Stitch platform also does.

interesting, okay. it feels a little more natural to me to leverage the semi-structured data support in snowflake. what do y’all think?

Yeah if a DB supports object types natively (like Postgres json(b)) I think that makes sense. Would need to be configurable for backward compatibility of course

cool, will look into it

@david_wallace - This is also something we likely would want to add into the SDK as a core feature, so any target based on it would get the functionality. I’ve seen some targets have a

flatten_records

bool setting, or

max_flattening_level

configurable setting.

oh interesting. that makes sense