Hi all, sorry I feel like keep coming back with a different problem every week haha, I'm trying to do a test run with the Rest API extractor and load it into Bigquary, however when loading the data into Bigquary it gets stuck on: google.api_core.bidi | Thread-ConsumeBidirectionalStream exiting cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=6210ab7f-32c7-420b-a913-3d84248aa340 stdio=stderr string_id=target-bigquery (as the last line). I will add the rest down below in a screenshot along with my yml: Any help on where I might have gone wrong will be appreciated, I am still learning about all of this. 🙏

Thanks for providing the detail. What

meltano

command are you running

meltano run tap-rest-api-msdk target-bigquery

And please can you try

meltano invoke tap-rest-api-msdk

and see what the output is? Do you get the expected rows etc?

yes seems to be running all fine:

The jsonPath matches =0 is a bit odd, can you try

meltano run tap-rest-api-msdk target-jsonl

No I only have target-bigquery installed

OK can you install target-jsonl and test that you are getting the expected data out?

yes, just did it it generated the jsonl file with the correct output

OK great, just checking. So did you manage to get any data loaded into bigquery? Did you try without the where clause in your sql?

yes I tested with the where and without the where clause I'll post the screen shot showing both results 🤔

Can you try the suggestions here? Sorry I dont use bq but this looks similar?

Aaah okay, thank you I'm going to try their suggestion of changing the method 👍

Tap / target repos closed issues can sometimes be helpful, good luck. If not someone else might know more

thank you that fixed the problem for me changed method to batch 🥳