Hi, I am working on a pipeline where I am (step 1) exporting data from MySql to Parquet files stored on S3, then (step 2) transforming the data using dbt Redshift external tables.

meltano tap-mysql target-s3

There is a problem though with aligning the parquet fields for dbt to ingest. I am seeing multiple Spectrum errors relating to the order of fields in the Parquet file not being aligned to the definition in the dbt external schema definition. Is there anyone who has done similar who can guide me in the right direction?

Hi Ian, are you using https://github.com/crowemi/target-s3?

I'm using

tap-mysql

-->

target-s3

to create parquet files stored in AWS S3 I cannot find a Meltano/Singer tap that will then allow me to load the parquet files to AWS Redshift. So my current option seems to be to create external tables in Redshift. The problem here is that the order of the fields in the parquet files does not always seem to be consistent - it is this that causes issues when trying to load the data.