Hello Trying to add big query loader but gives err...
# troubleshooting
p
Hello Trying to add big query loader but gives error in installation of some libraries, Please check and let me know if anyone can help
Copy code
Specify the plugin's namespace, which will serve as the:
- identifier to find related/compatible plugins
- default target database dialect (`dialect` extra),
  for use by transformers that connect with the database

Hit Return to accept the default: plugin name with underscores instead of dashes

(namespace) [target_bigquery]:

Specify the plugin's `pip install` argument, for example:
- PyPI package name:
        target-bigquery
- Git repository URL:
        git+<https://gitlab.com/meltano/target-bigquery.git>
- local directory, in editable/development mode:
        -e extract/target-bigquery
- 'n' if using a local executable (nothing to install)

Default: plugin name as PyPI package name

(pip_url) [target-bigquery]:

Specify the plugin's executable name

Default: name derived from `pip_url`

(executable) [target-bigquery]:

Specify the target's supported settings (`config.json` keys)

Multiple setting names (keys) can be separated using commas.

A setting kind can be specified alongside the name (key) by using the `:` delimiter,
e.g. `port:integer` to set the kind `integer` for the name `port`

Supported setting kinds:
string | integer | boolean | date_iso8601 | email | password | oauth | options | file | array | object | hidden

- Credentials and other sensitive setting types should use the password kind.
- If not specified, setting kind defaults to string.
- Nested properties can be represented using the `.` separator, e.g. `auth.username` for `{ "auth": { "username": value } }`.
- To find out what settings a target supports, reference its documentation.

Default: no settings

(settings) [[]]: port:integer
Loader 'target-bigquery' already exists in your Meltano project
To add it to your project another time so that each can be configured differently,
add a new plugin inheriting from the existing one with its own unique name:
        meltano add loader target-bigquery--new --inherit-from target-bigquery

Installing loader 'target-bigquery'...
Loader 'target-bigquery' could not be installed: failed to install plugin 'target-bigquery'.
  Running command git clone --filter=blob:none --quiet <https://github.com/adswerve/target-bigquery.git> /tmp/pip-req-build-_qz6v1w9
  Running command git checkout -q 837c140cdbc017c32939371e2a374877523853cb
ERROR: Cannot install setuptools==57.5.0 and target-bigquery==0.11.0 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>
c
đź‘‹ Hi Pranit. Could you could you share some additional info here? It would help to know: 1. The specific CLI command that you ran that had this output 2. The contents of your
meltano.yml
file 3. The output of
pip3 freeze
p
@cody_hanson I was able to surpass that issue with updating git version of loader, but bigquery loader fetches below error wile running the pipe ```meltano elt tap-klaviyo target-bigquery 2022-04-08T043046.426348Z [info ] Running extract & load... job_id=2022-04-08T043045--tap-klaviyo--target-bigquery name=meltano run_id=fcd0ba3f-39f5-4b11-a099-539497ef1018 2022-04-08T043046.485168Z [warning ] No state was found, complete import. 2022-04-08T043047.681744Z [info ] INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.3120112419128418, "tags": {"endpoint": "bounce", "http_status_code": 200, "status": "succeeded"}} cmd_type=extractor job_id=2022-04-08T043045--tap-klaviyo--target-bigquery name=tap-klaviyo run_id=fcd0ba3f-39f5-4b11-a099-539497ef1018 stdio=stderr 2022-04-08T043047.702765Z [info ] INFO Replicated bounce up to {'since': '2021-07-04T170952Z'} cmd_type=extractor job_id=2022-04-08T043045--tap-klaviyo--target-bigquery name=tap-klaviyo run_id=fcd0ba3f-39f5-4b11-a099-539497ef1018 stdio=stderr 2022-04-08T043047.971356Z [info ] INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.26142120361328125, "tags": {"endpoint": "bounce", "http_status_code": 200, "status": "succeeded"}} cmd_type=extractor job_id=2022-04-08T043045--tap-klaviyo--target-bigquery name=tap-klaviyo run_id=fcd0ba3f-39f5-4b11-a099-539497ef1018 stdio=stderr 2022-04-08T043047.983650Z [info ] INFO Replicated bounce up to {'since': '2021-07-08T184559Z'} cmd_type=extractor job_id=2022-04-08T043045--tap-klaviyo--target-bigquery name=tap-klaviyo run_id=fcd0ba3f-39f5-4b11-a099-539497ef1018 stdio=stderr 2022-04-08T043048.217630Z [info ] INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.22844743728637695, "tags": {"endpoint": "bounce", "http_status_code": 200, "status": "succeeded"}} cmd_type=extractor job_id=2022-04-08T043045--tap-klaviyo--target-bigquery name=tap-klaviyo run_id=fcd0ba3f-39f5-4b11-a099-539497ef1018 stdio=stderr 2022-04-08T043048.234302Z [info ] INFO Replicated bounce up to {'since': '2021-07-13T130524Z'} cmd_type=extractor job_id=2022-04-08T043045--tap-klaviyo--target-bigquery name=tap-klaviyo run_id=fcd0ba3f-39f5-4b11-a099-539497ef1018 stdio=stderr 2022-04-08T043048.469848Z [info ] INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.22894620895385742, "tags": {"endpoint": "bounce", "http_status_code": 200, "status": "succeeded"}} cmd_type=extractor job_id=2022-04-08T043045--tap-klaviyo--target-bigquery name=tap-klaviyo run_id=fcd0ba3f-39f5-4b11-a099-539497ef1018 stdio=stderr 2022-04-08T043048.483690Z [info ] INFO Replicated bounce up to {'since': '2021-07-14T110120Z'} cmd_type=extractor job_id=2022-04-08T043045--tap-klaviyo--target-bigquery name=tap-klaviyo run_id=fcd0ba3f-39f5-4b11-a099-539497ef1018 stdio=stderr 2022-04-08T043048.787897Z [info ] INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.29814720153808594, "tags": {"endpoint": "bounce", "http_status_code": 200, "status": "succeeded"}} cmd_type=extractor job_id=2022-04-08T043045--tap-klaviyo--target-bigquery name=tap-klaviyo run_id=fcd0ba3f-39f5-4b11-a099-539497ef1018 stdio=stderr 2022-04-08T043048.797466Z [info ] INFO Replicated bounce up to {'since': '2021-07-14T120808Z'} cmd_type=extractor job_id=2022-04-08T043045--tap-klaviyo--target-bigquery name=tap-klaviyo run_id=fcd0ba3f-39f5-4b11-a099-539497ef1018 stdio=stderr 2022-04-08T043048.865089Z [info ] INFO bounce schema: {'properties': {'uuid': {'type': 'string'}, 'event_name': {'type': 'string'}, 'timestamp': {'type': 'integer'}, 'object': {'type': 'string'}, 'datetime': {'type': 'string'}, 'statistic_id': {'type': 'string'}, 'id': {'type': 'string'}, 'event_properties': {'properties': {}, 'type': 'object'}, 'person': {'properties': {}, 'type': 'object'}}, 'type': 'object'} cmd_type=loader job_id=2022-04-08T043045--tap-klav…
specifically
bigquery/venv/lib/python3.8/site-packages/target_bigquery/schema.py", line 359, in format_record_to_schema\n    record[k] = conversion_dict[bq_schema[k]["type"]](v)\n', "KeyError: 'RECORD'\n"] name=meltano run_id=fcd0ba3f-39f5-4b11-a099-539497ef1018
I have created taps catalog file using discover command.
I think the issue is with
table_config
file , how do we generate that?
c
Could you share the pip url for the
target-bigquery
variant that you're using? If I read your initial message correctly, it seems like you installed it as a custom plugin, is that correct?
p
Hi Cody, I have came long way from that. Installation is sorted tagging you in other thread where there are two more like me