Hey! Trying to run tap-facebook along with target-...
# troubleshootingg
gary_james
09/08/2021, 5:16 PMHey! Trying to run tap-facebook along with target-snowflake, but can't seem to stop this behaviour where JSON columns are expanded into their own tables 😬 has anyone else come across this/ know how to stop it happening? maybe one of the non-DataMill variants of target-snowflake? dbt-tap-facebook doesn't appreciate this setup 😂 thanks!
a
aaronsteers
09/08/2021, 5:17 PMThis is called "flattening" and the Pipelinewise variant makes it configurable using the
data_flattening_max_levelaaronsteers
09/08/2021, 5:19 PMSorry - I read this through too quickly. My note relates to column flattening but I'm not sure about the entirely new tables that are being created there.
aaronsteers
09/08/2021, 5:19 PMI do think it's worth trying the Pipelinewise fork to see if it has similar/same behavior.
aaronsteers
09/08/2021, 5:19 PMI assume you've confirmed already that the tap isn't sending them as separate streams?
g
gary_james
09/08/2021, 5:38 PMrunning the tap alone they're returning within the stream as just json fields, so that seems ok
gary_james
09/08/2021, 5:38 PMbut sure thing, will give the pipelinewise one a shot and come back 👌
p
pablo_seibelt
09/09/2021, 3:30 PMIf you can modify the tap, you can make those JSON columns be stringified so they are not expanded like that, it's what i did with some data that i didn't want to create new tables. Usually these tables are created when it detects arrays, if they are just objects they are insterted into the original table. I'd love for it to use the VARIANT type instead
g
gary_james
09/10/2021, 9:49 AMyeah switching to the pipelinewise version seems to use variants instead, so no extra tables, just seemed weird that it would behave that way for a snowflake target anyway where json/ arrays are easily handled.
now I have "Numeric value '' is not recognized" on every tap instead, the plot thickens 😂
gary_james
09/10/2021, 10:27 AMWill update my file formats and will hopefully be all good here, thanks!