sam_daniel
08/18/2022, 2:45 AMversion: 1
default_environment: dev
environments:
- name: dev
- name: staging
- name: prod
plugins:
extractors:
- name: tap-bigquery
variant: anelendata
pip_url: tap-bigquery
config:
project_id: 24af770b-361d-4eea-8546-825bf8c8b144
name: bq_exct
table: "`etl-development-sam.Dummy_Data.Transact`"
columns: [order_date]
credentials_path: cred.json
datetime_key: order_date
start_datetime: '2018-05-15'
end_datetime: '2018-06-01'
limit: 1234
location: EU
streams: ['Transact']
loaders:
- name: target-csv
variant: hotgluexyz
pip_url: git+<https://github.com/hotgluexyz/target-csv.git>
config:
destination_path: test.csv
This is the error I got when I tried to run meltano invoke tap-bigquery
Catalog discovery failed: command ['/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/extractors/tap-bigquery/venv/bin/tap-bigquery', '--config', '/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/run/tap-bigquery/tap.619fa9c9-23ec-462d-946b-f8cb79dffa87.config.json', '--discover'] returned 1 with stderr:
CRITICAL string indices must be integers
Traceback (most recent call last):
File "/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/extractors/tap-bigquery/venv/bin/tap-bigquery", line 8, in <module>
sys.exit(main())
File "/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/extractors/tap-bigquery/venv/lib/python3.9/site-packages/singer/utils.py", line 229, in wrapped
return fnc(*args, **kwargs)
File "/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/extractors/tap-bigquery/venv/lib/python3.9/site-packages/tap_bigquery/__init__.py", line 175, in main
catalog = discover(CONFIG)
File "/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/extractors/tap-bigquery/venv/lib/python3.9/site-packages/tap_bigquery/__init__.py", line 45, in discover
stream_metadata, stream_key_properties, schema = source.do_discover(
File "/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/extractors/tap-bigquery/venv/lib/python3.9/site-packages/tap_bigquery/sync_bigquery.py", line 76, in do_discover
keys = {"table": stream["table"],
TypeError: string indices must be integerschristoph
08/18/2022, 4:32 AMstreams config looks different from the example provided by the tap author https://github.com/anelendata/tap-bigquery#step-1-configurechristoph
08/18/2022, 4:34 AMaaron_phethean
08/18/2022, 7:11 AMjuan_luis_cano_rodriguez
08/18/2022, 12:49 PMplugins:
extractors:
- name: tap-bigquery
variant: anelendata
pip_url: tap-bigquery
- name: tap-bigquery--ga4-data
inherit_from: tap-bigquery
select:
- analytics_257360217.*
- analytics_257428027.*
- analytics_266231070.*
metadata:
'*':
replication-method: INCREMENTAL
replication-key: event_timestamp
environments:
- name: dev
config:
plugins:
extractors:
- name: tap-bigquery--ga4-data
config:
start_datetime: '2022-08-10T00:00:00Z'
streams:
- name: analytics_257360217
table: orchest-ga4-data.analytics_257360217.all_events
columns:
- '*'
datetime_key: event_timestamp
hope it helps