My pipeline is failing after loading hubspot for ~10 minutes. how do I debug this? preferably without having to wait 10 minutes each time?

target-bigquery | INFO loading t_test-contact_lists_d646fe67efc5443ea356ec5e05fb6327 to BigQuery

target-bigquery | ERROR failed to load table t_test-contact_lists_d646fe67efc5443ea356ec5e05fb6327 from file: 400 POST <https://bigquery.googleapis.com/upload/bigquery/v2/projects/warehouse-310706/jobs?uploadType=resumable>: Invalid value for type: OBJECT is not a valid value

target-bigquery | CRITICAL 400 POST <https://bigquery.googleapis.com/upload/bigquery/v2/projects/warehouse-310706/jobs?uploadType=resumable>: Invalid value for type: OBJECT is not a valid value

pipe your tap data to a file so you can keep sending it to the target. The beauty of singer

meltano invoke tap-hubspot > hubspotdata

cat hubspotdata | meltano invoke target-bigquery

thank you. i am learning a lot here.

😄 welcome to the party!

why do you prefer this over

target-csv

or something similar?

I suspect @visch recommend that because dumping it to a file preserves the raw records (potentially aiding in debugging). Then if theres a failure on the

meltano invoke target-bigquery

you can both inspect the raw records and also potentially retry your load without having to extract everything from the hubspot api all over again.