Hi all I am looking to use tap bigquery anelendata variant H Meltano #troubleshooting

Hi all! I am looking to use tap-bigquery (anelend...

ella_wherritt

06/28/2023, 3:30 AM

Hi all! I am looking to use tap-bigquery (anelendata variant). However, I want to do two things differently: • Instead of using a

client_secrets.json

file, I want to express the credential as a JSON string in the

GOOGLE_APPLICATION_CREDENTIALS_STRING

environment variable. In the documentation, it does support this and says "JSON string should contain the same contents and be formatted the same way as the contents of a

client_secrets.json

file." • I'd like to express the streams in the configuration itself, rather than in a tap_config.json. However, I have run into this error message with the current configuration (as seen in the comments):

Copy code

Plugin configuration is invalid
Catalog discovery failed: command ['....config.json', '--discover'] returned -11 with stderr:
 INFO Running query:
    SELECT * FROM <project>.<dataset>.<table> LIMIT 10

Any guidance would be appreciated! 🙏

ella_wherritt

06/28/2023, 3:30 AM

Copy code

plugins:
  extractors:
  - name: tap-bigquery
    namespace: tap_bigquery
    pip_url: git+<https://github.com/anelendata/tap-bigquery.git@v0.3.7>
    executable: tap-bigquery
    capabilities:
    - state
    - catalog
    - discover
    settings:
    - name: credentials_path
      env: GOOGLE_APPLICATION_CREDENTIALS_STRING
      value: '{"type": "X", "project_id": "X "auth_uri":...}'
    - name: start_datetime
      kind: date_iso8601
    - name: start_always_inclusive
      kind: boolean
    - name: GOOGLE_CLOUD_PROJECT
      value: xxxx
    config:
      streams:
      - name: table_name
        table: <project>.<dataset>.<table>
        columns:
        - col_1
        datetime_key: updated_at
      start_datetime: $BQ_START_DATETIME
      start_always_inclusive: true
      limit: 10
  - name: tap-example
    inherit_from: tap-bigquery
    metadata:
      '*':
        replication-method: INCREMENTAL
        replication-key: updated_at

Andy Carter

06/28/2023, 9:18 AM

@ella_wherritt are you able to separate out the two changes you made and test them separately? Assuming you had a working meltano.yml before, which change caused the error?

haleemur_ali

06/29/2023, 6:32 PM

👋 @ella_wherritt

Open in Slack

Previous Next