Hi, I am new to Meltano. I am trying to set up a B...
# troubleshooting
s
Hi, I am new to Meltano. I am trying to set up a BigQuery extractor. Below is my yml file created to configure the extractor. But not sure why I am getting this type error. Can someone help me please?
Copy code
version: 1
default_environment: dev
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-bigquery
    variant: anelendata
    pip_url: tap-bigquery
    config:
      project_id: 24af770b-361d-4eea-8546-825bf8c8b144
      name: bq_exct
      table: "`etl-development-sam.Dummy_Data.Transact`"
      columns: [order_date]
      credentials_path: cred.json
      datetime_key: order_date
      start_datetime: '2018-05-15'
      end_datetime: '2018-06-01'
      limit: 1234
      location: EU
      streams: ['Transact']
  loaders:
  - name: target-csv
    variant: hotgluexyz
    pip_url: git+<https://github.com/hotgluexyz/target-csv.git>
    config:
      destination_path: test.csv
This is the error I got when I tried to run meltano invoke tap-bigquery
Copy code
Catalog discovery failed: command ['/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/extractors/tap-bigquery/venv/bin/tap-bigquery', '--config', '/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/run/tap-bigquery/tap.619fa9c9-23ec-462d-946b-f8cb79dffa87.config.json', '--discover'] returned 1 with stderr:
 CRITICAL string indices must be integers
Traceback (most recent call last):
  File "/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/extractors/tap-bigquery/venv/bin/tap-bigquery", line 8, in <module>
    sys.exit(main())
  File "/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/extractors/tap-bigquery/venv/lib/python3.9/site-packages/singer/utils.py", line 229, in wrapped
    return fnc(*args, **kwargs)
  File "/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/extractors/tap-bigquery/venv/lib/python3.9/site-packages/tap_bigquery/__init__.py", line 175, in main
    catalog = discover(CONFIG)
  File "/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/extractors/tap-bigquery/venv/lib/python3.9/site-packages/tap_bigquery/__init__.py", line 45, in discover
    stream_metadata, stream_key_properties, schema = source.do_discover(
  File "/home/samdaniel/meltano-projects/meltano-projects/my_bq_project/.meltano/extractors/tap-bigquery/venv/lib/python3.9/site-packages/tap_bigquery/sync_bigquery.py", line 76, in do_discover
    keys = {"table": stream["table"],
TypeError: string indices must be integers
c
Your
streams
config looks different from the example provided by the tap author https://github.com/anelendata/tap-bigquery#step-1-configure
a
Don't like the look of those backticks in the table: configuration.
j
I successfully configured the YAML like this:
Copy code
plugins:
  extractors:
  - name: tap-bigquery
    variant: anelendata
    pip_url: tap-bigquery
  - name: tap-bigquery--ga4-data
    inherit_from: tap-bigquery
    select:
    - analytics_257360217.*
    - analytics_257428027.*
    - analytics_266231070.*
    metadata:
      '*':
        replication-method: INCREMENTAL
        replication-key: event_timestamp
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-bigquery--ga4-data
        config:
          start_datetime: '2022-08-10T00:00:00Z'
          streams:
          - name: analytics_257360217
            table: orchest-ga4-data.analytics_257360217.all_events
            columns:
            - '*'
            datetime_key: event_timestamp
hope it helps