Join Slack
Powered by
Hello! I am using target-snowflake as loader to st...
# singer-targets
e
Eni S
11/13/2024, 1:15 PM
Hello! I am using target-snowflake as loader to store some data with flattening enabled. Is there a way to skip flattening on arrays? When flattening is enabled these arrays are converted to string. If no, how do you handle this problem? Thnaks!
v
visch
11/13/2024, 2:14 PM
I always keep flattening off in the targets 🤷
visch
11/13/2024, 2:14 PM
Flatten in transformation after
e
Edgar RamÃrez (Arch.dev)
11/13/2024, 3:59 PM
Hi
@Eni S
. I agree with Derek that at the moment, flattening during downstream transformation would be the only way to preserve those fields as arrays. If you're curious, this is happening because the Singer SDK, which is the framework target-snowflake and others build upon, dumps complex values at the max flattening depth:
https://github.com/meltano/sdk/blob/72c9a19ac55ec1c8e86222aa49ebc09add981166/singer_sdk/helpers/_flattening.py#L444-L451
I can imagine that is something we could tweak, so do
log a feature request
.
3
Views
Open in Slack
Previous
Next