Hey everyone .. having an issue when extracting data from mysql db using

tap-mysql

and dumping it in S3 using

target-s3-csv

then -> manually importing it in Snowflake .. The issue is that Snowflake is complaining about the importation / data schema because of the csv file dumped in the s3 has different columns order compared with the table defined in Snowflake .. My question, how can I define the order of the extracted columns the way I want .. not letting meltano decides the order (every time it does that in a different order). Or, is

targe-s3-csv

that changes the order, if so .. then how can I modify the order?

I believe the target determines the column order. The data it receives from the tap is simply JSON, which has no inherent order, so the target can decide what it wants to do on its own. target-postgres and target-snowflake (at least the transferwise variants) sort the columns in alpha order, which is fine because they also create the destination tables that way.

Yeah, Thomas is correct. It’s the target that decides what to do with the arbitrary field order it gets from the tap. I’d say it’s mostly the responsibility of the target, but taps should also try to be consistent in the order of fields they output in the schema messages. There’s a good discussion in https://github.com/transferwise/pipelinewise-target-snowflake/issues/222.