Hey! I'm trying to connect tap_shopify with target-bigquery and running into some known issues I couldn't find the solution for, here's the error I get:

INFO Load t_orders_f2bb4a8fa8a3451bb7dfb3f1555d29cf by FULL_TABLE (truncate) cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery
2023-01-25T00:23:01.766684Z [info ] INFO loading t_orders_f2bb4a8fa8a3451bb7dfb3f1555d29cf to BigQuery cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery
2023-01-25T00:23:02.497689Z [info ] ERROR failed to load table t_orders_f2bb4a8fa8a3451bb7dfb3f1555d29cf from file: 400 POST <https://bigquery.googleapis.com/upload/bigquery/v2/projects/exp/jobs?uploadType=resumable>: Field subtotal_price_set is type RECORD but has no schema cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery
2023-01-25T00:23:02.498182Z [info ] CRITICAL 400 POST <https://bigquery.googleapis.com/upload/bigquery/v2/projects/exp/jobs?uploadType=resumable>: Field subtotal_price_set is type RECORD but has no schema cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery

I tried using a table config json for the bigquery loader to set the data types. Also weird note, I didn't include subtotal_price_set in my yml shopify parameters or the bigquery table config, I'm confused why that's popping up. I'm able to do tap-shopify to target-jsonl with no issues. Any help on this would be really appreciated 🙏

Hi @enzo_garofalo in my experience these issues tend to be in the ‘target’. I'm guessing there is a type emitted from the shopify tap that the big query target doesn't understand. Unfortunately I havent had to use big query in anger, so can't offer much more specific help. In general, you might be able to supply your own catalog and finesse the type to something big query can understand. Using the catalog you could exclude the problematic column.

Thanks @aaron_phethean! How do you supply your own catalog?

You supply the catalog as a setting to the tap. Meltano provides this to the tap as config when run. e.g.

export TAP_MSSQL__CATALOG="catalog.json"
meltano run tap-mssql target-postgres--transferwise

https://docs.meltano.com/guide/integration#extractor-catalog-generation The catalog can deal with type (hints to the target), selection of tables and columns.