Hi Hi, I've posted this question a while ago and tried a few things however I'm not too sure what I'm doing wrong I still can't seem to get the target-Bigquary to work. I'm not too sure if its a config error on my part/most likely. I'm using the dog API by Kinduff just for testing it out however for a separate project I'm making a pipeline that uses it and need to figure out why i can't seem to get the loader to work. I've tried playing around with the config a bit however I'm still stuck. Any help would be appreciated 🙏

IIRC you said this worked with

target-jsonl

? Maybe best to isolate the target process and debug from there:

meltano invoke tap-rest-api-msdk > tap.out
cat tap.out | meltano invoke target-bigquery

yes it did

It looks like it almost worked

Target 'target-bigquery' completed reading 13 lines of input (10 records, (0 batch manifests, 2 state messages)

but then this right after doesn't fill me with hope

RPC termination has signaled streaming pull manager shutdown

yeah thats what i was thinking aswell it pick up and reads the records but doesn't insert it 🤔

https://github.com/z3z1ma/target-bigquery/issues/73 seems relevant. Basically says to try

method: gcs_stage

I'll give it a try and see if it changes anything hopefully 🤞

Yeah, I won't pretend to know anything about how it's actually working behind the scenes. 😅