https://linen.dev logo
Join Slack
Powered by
Hi Hi, wondering if someone might be able to help ...
# troubleshooting
h
Hi Hi, wondering if someone might be able to help me. I seem to be having a bit of trouble with the target-bigquery I'm on windows, but i am using WSL I'm using the dog api by kinduff for this example. I'm using the tap-rest-api-msdk everything seems to be running however i can get any records to display in the dataset. I believe i configure the tap correct as when i use target-jasonl i get the attached output. However i'm wondering if I might have misconfigure big query per chance?
👀 1
Copy code
version: 1
default_environment: dev
project_id: 1c83f15b-75f1-4cc6-96bb-7f8fffe1bf76
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-rest-api-msdk
    variant: widen
    pip_url: tap-rest-api-msdk
    config:
      api_url: <https://dogapi.dog/api/v2>
      streams:
      - name: dog_breeds
        path: /breeds
        primary_keys:
        - id
      records_path: $.data[*]
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
    config:
      custom_name: Testoutput
  - name: target-bigquery
    variant: z3z1ma
    pip_url: git+<https://github.com/z3z1ma/target-bigquery.git>
    config:
      credentials_path: test-dataset-422210-a91b0cc6d6ae.json
      dataset: dogs_dataset
      fail_fast: false
      project: test-dataset-422210
v
What are you running (I'd assume 
meltano run tap-rest-api-msdk target-bigquery)
? if so can you post that , and the logs
h
Copy code
hanroux@Morty:/mnt/c/Users/Mortus/meltano-projects/DogsApi$ meltano run tap-rest-api-msdk target-bigquery
2024-09-11T14:19:25.813948Z [info     ] Environment 'dev' is active   
2024-09-11T14:19:31.266611Z [info     ] 2024-09-11 15:19:31,266 | INFO     | tap-rest-api-msdk    | No schema found. Inferring schema from API call. cmd_type=elb consumer=False job_name=dev:tap-rest-api-msdk-to-target-bigquery
 name=tap-rest-api-msdk producer=True run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=tap-rest-api-msdk
2024-09-11T14:19:31.467887Z [info     ] 2024-09-11 15:19:31,467 | INFO     | singer_sdk.helpers.jsonpath | JSONPath $.data[*] match count: 10 cmd_type=elb consumer=False job_name=dev:tap-rest-api-msdk-to-target-bigquery name=tap-rest-api-msdk producer=True run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=tap-rest-api-msdk
2024-09-11T14:19:31.472281Z [info     ] 2024-09-11 15:19:31,472 | INFO     | tap-rest-api-msdk.tap-rest-api-msdk | Beginning full_table sync of 'dog_breeds'... cmd_type=elb consumer=False job_name=dev:tap-rest-api-msdk-to-target-bigquery name=tap-rest-api-msdk producer=True run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=tap-rest-api-msdk
2024-09-11T14:19:31.472884Z [info     ] 2024-09-11 15:19:31,472 | INFO     | tap-rest-api-msdk.tap-rest-api-msdk | Tap has custom mapper. Using 1 provided map(s). cmd_type=elb consumer=False job_name=dev:tap-rest-api-msdk-to-target-bigquery name=tap-rest-api-msdk producer=True run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=tap-rest-api-msdk
2024-09-11T14:19:31.473263Z [info     ] 2024-09-11 15:19:31,472 | INFO     | tap-rest-api-msdk.tap-rest-api-msdk | the next_page_token_jsonpath = $.next_page. cmd_type=elb consumer=False job_name=dev:tap-rest-api-msdk-to-target-bigquery name=tap-rest-api-msdk producer=True run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=tap-rest-api-msdk
2024-09-11T14:19:31.612523Z [info     ] 2024-09-11 15:19:31,612 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.13845, "tags": {"stream": "dog_breeds", "endpoint": "/breeds", "http_status_code": 200, "status": "succeeded"}} cmd_type=elb consumer=False job_name=dev:tap-rest-api-msdk-to-target-bigquery name=tap-rest-api-msdk producer=True run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=tap-rest-api-msdk
2024-09-11T14:19:31.612980Z [info     ] 2024-09-11 15:19:31,612 | INFO     | singer_sdk.helpers.jsonpath | JSONPath $.data[*] match count: 10 cmd_type=elb consumer=False job_name=dev:tap-rest-api-msdk-to-target-bigquery name=tap-rest-api-msdk producer=True run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=tap-rest-api-msdk
2024-09-11T14:19:31.668063Z [info     ] 2024-09-11 15:19:31,667 | INFO     | singer_sdk.helpers.jsonpath | JSONPath $.next_page match count: 0 cmd_type=elb consumer=False job_name=dev:tap-rest-api-msdk-to-target-bigquery name=tap-rest-api-msdk producer=True run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=tap-rest-api-msdk
2024-09-11T14:19:31.668511Z [info     ] 2024-09-11 15:19:31,667 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "http_request_count", "value": 1, "tags": {"stream": "dog_breeds", "endpoint": "/breeds", "pid": 892}} cmd_type=elb consumer=False job_name=dev:tap-rest-api-msdk-to-target-bigquery name=tap-rest-api-msdk producer=True run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=tap-rest-api-msdk
2024-09-11T14:19:31.668831Z [info     ] 2024-09-11 15:19:31,668 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.19555187225341797, "tags": {"stream": "dog_breeds", "pid": 892, "context": {}, "status": "succeeded"}} cmd_type=elb consumer=False job_name=dev:tap-rest-api-msdk-to-target-bigquery name=tap-rest-api-msdk producer=True run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=tap-rest-api-msdk
2024-09-11T14:19:31.669138Z [info     ] 2024-09-11 15:19:31,668 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 10, "tags": {"stream": "dog_breeds", "pid": 892, "context": {}}} cmd_type=elb consumer=False job_name=dev:tap-rest-api-msdk-to-target-bigquery name=tap-rest-api-msdk producer=True run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=tap-rest-api-msdk
2024-09-11T14:19:34.164029Z [info     ] 2024-09-11 15:19:34,163 | INFO     | target-bigquery      | Using thread-based parallelism cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:34.164464Z [info     ] 2024-09-11 15:19:34,163 | INFO     | target-bigquery      | Target 'target-bigquery' is listening for input from tap. cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:34.164744Z [info     ] 2024-09-11 15:19:34,164 | INFO     | target-bigquery      | Initializing 'target-bigquery' target sink... cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:34.165037Z [info     ] 2024-09-11 15:19:34,164 | INFO     | target-bigquery      | Initializing target sink for stream 'dog_breeds'... cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:34.804934Z [info     ] /mnt/c/Users/Mortus/meltano-projects/DogsApi/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/target_bigquery/proto_gen.py:97: UserWarning: MessageFactory class is deprecated. Please use GetMessageClass() instead of MessageFactory.GetPrototype. MessageFactory class will be removed after 2024. cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:34.805366Z [info     ]   proto_cls = factory.GetPrototype(proto_descriptor) cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:34.805729Z [info     ] 2024-09-11 15:19:34,804 | INFO     | target-bigquery      | Setting up dog_breeds cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:34.806758Z [info     ] 2024-09-11 15:19:34,806 | INFO     | target-bigquery      | Target 'target-bigquery' completed reading 13 lines of input (10 records, (0 batch manifests, 2 state messages). cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:34.807059Z [info     ] 2024-09-11 15:19:34,806 | INFO     | target-bigquery      | Adding worker f5c4873aa13b4fdb9d7c0ed863c9c0fa cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:35.103376Z [info     ] 2024-09-11 15:19:35,103 | INFO     | google.cloud.bigquery_storage_v1.writer | RPC termination has signaled streaming pull manager shutdown. cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:35.103884Z [info     ] 2024-09-11 15:19:35,103 | INFO     | google.api_core.bidi | Thread-ConsumeBidirectionalStream exiting cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:43.162019Z [info     ] 2024-09-11 15:19:43,161 | INFO     | target_bigquery.storage_write | job.attempts : 2 cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:43.162605Z [info     ] 2024-09-11 15:19:43,161 | INFO     | target_bigquery.storage_write | [f5c4873aa13b4fdb9d7c0ed863c9c0fa] : {'projects/test-dataset-422210/datasets/dogs_dataset/tables/dog_breeds': 0} cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:43.162964Z [info     ] 2024-09-11 15:19:43,161 | INFO     | target_bigquery.storage_write | Worker process exiting. cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:43.163358Z [info     ] 2024-09-11 15:19:43,162 | INFO     | target-bigquery      | Emitting completed target state {"bookmarks": {"dog_breeds": {}}} cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:43.193221Z [info     ] Incremental state has been updated at 2024-09-11 14:19:43.193200+00:00.
2024-09-11T14:19:43.301174Z [info     ] Block run completed.           block_type=ExtractLoadBlocks err=None set_number=0 success=True
i believe this is the log
elt.log
v
Copy code
2024-09-11T14:19:34.807059Z [info     ] 2024-09-11 15:19:34,806 | INFO     | target-bigquery      | Adding worker f5c4873aa13b4fdb9d7c0ed863c9c0fa cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:35.103376Z [info     ] 2024-09-11 15:19:35,103 | INFO     | google.cloud.bigquery_storage_v1.writer | RPC termination has signaled streaming pull manager shutdown. cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:35.103884Z [info     ] 2024-09-11 15:19:35,103 | INFO     | google.api_core.bidi | Thread-ConsumeBidirectionalStream exiting cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:43.162019Z [info     ] 2024-09-11 15:19:43,161 | INFO     | target_bigquery.storage_write | job.attempts : 2 cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:43.162605Z [info     ] 2024-09-11 15:19:43,161 | INFO     | target_bigquery.storage_write | [f5c4873aa13b4fdb9d7c0ed863c9c0fa] : {'projects/test-dataset-422210/datasets/dogs_dataset/tables/dog_breeds': 0} cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:43.162964Z [info     ] 2024-09-11 15:19:43,161 | INFO     | target_bigquery.storage_write | Worker process exiting. cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
2024-09-11T14:19:43.163358Z [info     ] 2024-09-11 15:19:43,162 | INFO     | target-bigquery      | Emitting completed target state {"bookmarks": {"dog_breeds": {}}} cmd_type=elb consumer=True job_name=dev:tap-rest-api-msdk-to-target-bigquery name=target-bigquery producer=False run_id=223861aa-f130-450c-a491-d50dbf7169e5 stdio=stderr string_id=target-bigquery
There's something up with the target, this thing I hope is failing with a non zero exit code, but this error isn't super helpful. There's been multiple complaints about this target recently I wonder if there's a consistent failure out there. The error messages should be a bit better too 😮
h
yeah I'm not too sure. I'm going to keep playing around with it until something changes or if i can find something. 🤔
2 Views
Open in Slack
PreviousNext
Powered by