I'm getting data from

tap-recurly

and pushing into

pipelinewise-target-snowflake

. There's a stream,

coupons

that has a field

plans

. 98% of the time, this field is an array. 2% of the time, this field is a string. Now, the way

pipelinewise-target-snowflake

works is that it has understood this field as VARIANT and will try to json_parse() it. When a row from an S3 stage where this field is a string, json_parse() on that field will fail.

Error parsing JSON: unknown keyword "all", pos 4

Now my question is: What is the root issue? What is the right solution? Should I fix the

tap-recurly

to always pass an array of objects? Should I fix the

pipelinewise-target-snowflake

so that if a string is passed into a variant column, something happens? There isn't a good universal solution to that IMO, but I could just NULL it instead of letting it fail like this. Then again, maybe it should fail. Some other solution that would be better?

You could also use a mapper to convert the data on the fly. Easiest would be to cast it all to a string and then convert in Snowflake. https://sdk.meltano.com/en/latest/stream_maps.html#automatic-schema-detection