https://linen.dev logo
Join Slack
Powered by
hey all, another noob to Meltano here - running in...
# plugins-general
c
hey all, another noob to Meltano here - running into an error when using 
tap-hellobaton
target-bigquery
. has anyone seen this before? I followed instructions exactly as in the READMEs for both the tap and target. 😕 ```2022-02-09T213319.248121Z [info ] INFO activity schema: {'properties': {'id': {'type': 'integer'}, '_self': {'type': 'string'}, 'type': {'type': ['string', 'null']}, 'group': {'type': 'string'}, 'parent': {'type': ['string', 'null']}, 'child': {'type': ['string', 'null']}, 'actor': {'type': ['string', 'null']}, 'project': {'type': 'string'}, 'parent_type': {'type': ['string', 'null']}, 'child_type': {'type': ['string', 'null']}, 'meta': {'type': ['object', 'null']}, 'created': {'format': 'date-time', 'type': 'string'}, 'modified': {'format': 'date-time', 'type': 'string'}}} cmd_type=loader job_id=2022-02-09T213315--tap-hellobaton--target-bigquery name=target-bigquery run_id=6bffdd0c-cd23-4d24-8716-8ff330f4691f stdio=stderr 2022-02-09T213319.758702Z [info ] time=2022-02-09 163319 name=tap-hellobaton level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 1.049385, 'tags': {'endpoint': '/activity/', 'http_status_code': 200, 'status': 'succeeded'}} cmd_type=extractor job_id=2022-02-09T213315--tap-hellobaton--target-bigquery name=tap-hellobaton run_id=6bffdd0c-cd23-4d24-8716-8ff330f4691f stdio=stderr 2022-02-09T213319.759769Z [info ] CRITICAL 'RECORD' cmd_type=loader job_id=2022-02-09T213315--tap-hellobaton--target-bigquery name=target-bigquery run_id=6bffdd0c-cd23-4d24-8716-8ff330f4691f stdio=stderr 2022-02-09T213319.761067Z [info ] CRITICAL ['Traceback (most recent call last):\n', ' File "/Users/chack/Documents/engineering/meltano-projects/baton/.meltano/loaders/target-bigquery/venv/lib/python3.8/site-packages/target_bigquery/__init__.py", line 129, in main\n for state in state_iterator:\n', ' File "/Users/chack/Documents/engineering/meltano-projects/baton/.meltano/loaders/target-bigquery/venv/lib/python3.8/site-packages/target_bigquery/process.py", line 54, in process\n for s in handler.handle_record_message(msg):\n', ' File "/Users/chack/Documents/engineering/meltano-projects/baton/.meltano/loaders/target-bigquery/venv/lib/python3.8/site-packages/target_bigquery/processhandler.py", line 179, in handle_record_message\n nr = format_record_to_schema(nr, self.bq_schema_dicts[stream])\n', ' File "/Users/chack/Documents/engineering/meltano-projects/baton/.meltano/loaders/target-bigquery/venv/lib/python3.8/site-packages/target_bigquery/schema.py", line 359, in format_record_to_schema\n record[k] = conversion_dict[bq_schema[k]["type"]](v)\n', "KeyError: 'RECORD'\n"] cmd_type=loader job_id=2022-02-09T213315--tap-hellobaton--target-bigquery name=target-bigquery run_id=6bffdd0c-cd23-4d24-8716-8ff330f4691f stdio=stderr 2022-02-09T213319.837095Z [error ] Loading failed code=2 job_id=2022-02-09T213315--tap-hellobaton--target-bigquery message=CRITICAL ['Traceback (most recent call last):\n', ' File "/Users/chack/Documents/engineering/meltano-projects/baton/.meltano/loaders/target-bigquery/venv/lib/python3.8/site-packages/target_bigquery/__init__.py", line 129, in main\n for state in state_iterator:\n', ' File "/Users/chack/Documents/engineering/meltano-projects/baton/.meltano/loaders/target-bigquery/venv/lib/python3.8/site-packages/target_bigquery/process.py", line 54, in process\n for s in handler.handle_record_message(msg):\n', ' File "/Users/chack/Documents/engineering/meltano-projects/baton/.meltano/loaders/target-bigquery/venv/lib/python3.8/site-packages/target_bigquery/processhandler.py", line 179, in handle_record_message\n nr = format_record_to_schema(nr, self.bq_schema_dicts[stream])\n', ' File "/Users/chack/Documents/engineering/meltano-projects/baton/.meltano/loaders/target-bigquery/venv/lib/python3.8/site-packages/target_bigquery/schema.py", line 359, in format_record_to_schema\…
m
I’m not entirely sure, but you should try rerunning with the 
--log-level=debug
flag if you haven’t already. If that doesn’t help, can you share your 
meltano.yml
file?
c
that flag doesn’t seem to work
Copy code
(base) ➜  baton git:(master) ✗ meltano elt tap-hellobaton target-bigquery --log-level=debug                                                                                 
Usage: meltano elt [OPTIONS] EXTRACTOR LOADER
Try 'meltano elt --help' for help.

Error: No such option: --log-level
Copy code
(base) ➜  baton git:(master) ✗ cat meltano.yml 
version: 1
send_anonymous_usage_stats: true
project_id: d5be6888-f203-4fb0-a9c9-fb0165632a1a
plugins:
  extractors:
  - name: tap-hellobaton
    namespace: tap_hellobaton
    pip_url: git+<https://github.com/dluftspring/tap-hellobaton.git>
    executable: tap-hellobaton
    capabilities:
    - catalog
    - discover
    - state
    - about
    - stream-maps
    settings:
    - name: company
    - name: api_key
    - name: user_agent
    config:
      api_key: REDACTED
      company: mantl
  loaders:
  - name: target-bigquery
    variant: adswerve
    pip_url: git+<https://github.com/adswerve/target-bigquery.git@0.11.3>
    config:
      project_id: mantl-edw-datalake-production
      dataset_id: baton
environments:
- name: prod
cc: @dishant_bhatt
e
hi @charles_hack, the flag should go immediately after 
meltano
simple smile , so
Copy code
meltano --log-level=debug elt tap-hellobaton target-bigquery
c
thanks @edgar_ramirez_mondragon! it looks like the bigquery target is complaining about a data type that the baton tap is outputting
```[...] 2022-02-15T163850.576774Z [debug ] {"type": "RECORD", "stream": "activity", "record": {"id": 4623, "_self": "https://mantl.hellobaton.com/api/activity/4623/", "type": "task-create", "group": "task", "parent": "https://mantl.hellobaton.com/api/tasks/4671/", "child": null, "actor": "https://mantl.hellobaton.com/api/users/7078/", "project": "https://mantl.hellobaton.com/api/projects/423/", "parent_type": "task", "child_type": null, "meta": {}, "created": "2020-08-14T151810.630110Z", "modified": "2020-08-14T151810.630145Z"}, "time_extracted": "2022-02-15T163850.566283Z"} cmd_type=extractor job_id=2022-02-15T163846--tap-hellobaton--target-bigquery name=tap-hellobaton (out) run_id=81839c70-6017-4084-9e4a-a994858667d1 stdio=stdout 2022-02-15T163850.576947Z [debug ] {"type": "RECORD", "stream": "activity", "record": {"id": 4624, "_self": "https://mantl.hellobaton.com/api/activity/4624/", "type": "task-create", "group": "task", "parent": "https://mantl.hellobaton.com/api/tasks/4672/", "child": null, "actor": "https://mantl.hellobaton.com/api/users/7078/", "project": "https://mantl.hellobaton.com/api/projects/423/", "parent_type": "task", "child_type": null, "meta": {}, "created": "2020-08-14T151810.684685Z", "modified": "2020-08-14T151810.684786Z"}, "time_extracted": "2022-02-15T163850.566374Z"} cmd_type=extractor job_id=2022-02-15T163846--tap-hellobaton--target-bigquery name=tap-hellobaton (out) run_id=81839c70-6017-4084-9e4a-a994858667d1 stdio=stdout 2022-02-15T163850.577110Z [debug ] {"type": "RECORD", "stream": "activity", "record": {"id": 4625, "_self": "https://mantl.hellobaton.com/api/activity/4625/", "type": "task-create", "group": "task", "parent": "https://mantl.hellobaton.com/api/tasks/4673/", "child": null, "actor": "https://mantl.hellobaton.com/api/users/7078/", "project": "https://mantl.hellobaton.com/api/projects/423/", "parent_type": "task", "child_type": null, "meta": {}, "created": "2020-08-14T151810.769963Z", "modified": "2020-08-14T151810.770001Z"}, "time_extracted": "2022-02-15T163850.566465Z"} cmd_type=extractor job_id=2022-02-15T163846--tap-hellobaton--target-bigquery name=tap-hellobaton (out) run_id=81839c70-6017-4084-9e4a-a994858667d1 stdio=stdout 2022-02-15T163850.577273Z [debug ] {"type": "RECORD", "stream": "activity", "record": {"id": 4626, "_self": "https://mantl.hellobaton.com/api/activity/4626/", "type": "task-create", "group": "task", "parent": "https://mantl.hellobaton.com/api/tasks/4674/", "child": null, "actor": "https://mantl.hellobaton.com/api/users/7078/", "project": "https://mantl.hellobaton.com/api/projects/423/", "parent_type": "task", "child_type": null, "meta": {}, "created": "2020-08-14T151810.857084Z", "modified": "2020-08-14T151810.857122Z"}, "time_extracted": "2022-02-15T163850.566556Z"} cmd_type=extractor job_id=2022-02-15T163846--tap-hellobaton--target-bigquery name=tap-hellobaton (out) run_id=81839c70-6017-4084-9e4a-a994858667d1 stdio=stdout 2022-02-15T163850.627666Z [debug ] Deleted configuration at /Users/chack/Documents/engineering/meltano-projects/baton/.meltano/run/elt/2022-02-15T163846--tap-hellobaton--target-bigquery/81839c70-6017-4084-9e4a-a994858667d1/target.0695d689-9d0f-4323-8c9e-5b8eb7c32382.config.json 2022-02-15T163850.628411Z [debug ] Deleted configuration at /Users/chack/Documents/engineering/meltano-projects/baton/.meltano/run/elt/2022-02-15T163846--tap-hellobaton--target-bigquery/81839c70-6017-4084-9e4a-a994858667d1/tap.b64fb904-5854-49a5-868b-86254150725a.config.json 2022-02-15T163850.628556Z [error ] Loading failed code=2 job_id=2022-02-15T163846--tap-hellobaton--target-bigquery message=CRITICAL ['Traceback (most recent call last):\n', ' File "/Users/chack/Documents/engineering/meltano-projects/baton/.meltano/loaders/target-bigquery/venv/lib/python3.8/site-packages/target_bigquery/__init__.py", line 129, in main\n for state in state_iterator:\n', '…
i think bigquery is unable to understand these types with 
['string','null']]
Copy code
2022-02-15T16:38:49.501376Z [info     ] INFO activity schema: {'properties': {'id': {'type': 'integer'}, '_self': {'type': 'string'}, 'type': {'type': ['string', 'null']}, 'group': {'type': 'string'}, 'parent': {'type': ['string', 'null']}, 'child': {'type': ['string', 'null']}, 'actor': {'type': ['string', 'null']}, 'project': {'type': 'string'}, 'parent_type': {'type': ['string', 'null']}, 'child_type': {'type': ['string', 'null']}, 'meta': {'type': ['object', 'null']}, 'created': {'format': 'date-time', 'type': 'string'}, 'modified': {'format': 'date-time', 'type': 'string'}}} cmd_type=loader job_id=2022-02-15T163846--tap-hellobaton--target-bigquery name=target-bigquery run_id=81839c70-6017-4084-9e4a-a994858667d1 stdio=stderr
Open in Slack
PreviousNext
Powered by