Hi all, sorry I feel like keep coming back with a ...
# getting-started
h
Hi all, sorry I feel like keep coming back with a different problem every week haha, I'm trying to do a test run with the Rest API extractor and load it into Bigquary, however when loading the data into Bigquary it gets stuck on: google.api_core.bidi | Thread-ConsumeBidirectionalStream exiting cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=6210ab7f-32c7-420b-a913-3d84248aa340 stdio=stderr string_id=target-bigquery (as the last line). I will add the rest down below in a screenshot along with my yml: Any help on where I might have gone wrong will be appreciated, I am still learning about all of this. 🙏
1
In Bigquary itself "dogs" does get created however there is no data inside of it 🤔.
a
Thanks for providing the detail. What
meltano
command are you running
h
meltano run tap-rest-api-msdk target-bigquery
a
And please can you try
meltano invoke tap-rest-api-msdk
and see what the output is? Do you get the expected rows etc?
h
yes seems to be running all fine:
would this mean that it is the loader itself that I might have misconfigured? 🤔
a
The jsonPath matches =0 is a bit odd, can you try
meltano run tap-rest-api-msdk target-jsonl
assuming you have target-jsonl installed?
h
No I only have target-bigquery installed
a
OK can you install target-jsonl and test that you are getting the expected data out?
h
yes, just did it it generated the jsonl file with the correct output
a
OK great, just checking. So did you manage to get any data loaded into bigquery? Did you try without the where clause in your sql?
h
yes I tested with the where and without the where clause I'll post the screen shot showing both results 🤔
a
Can you try the suggestions here? Sorry I dont use bq but this looks similar?
h
Aaah okay, thank you I'm going to try their suggestion of changing the method 👍
a
Tap / target repos closed issues can sometimes be helpful, good luck. If not someone else might know more
🙌 1
h
thank you that fixed the problem for me changed method to batch 🥳