https://linen.dev logo
#plugins-general
Title
# plugins-general
m

michael_cooper

10/07/2020, 4:52 PM
So I'm following the pipeline specific guide to try and override a specific environment variable for this pipeline, but it doesn't seem to be passing value to the pipeline. Here's what I've got so far:
Copy code
# meltano.yml
...
  loaders:
  - name: target-bigquery
    namespace: bigquery
    pip_url: git+<https://github.com/Mashey/target-bigquery.git>
    executable: target-bigquery
    settings:
    - name: project_id
      value: vibrant-bonus-233400
    - name: dataset_id
    - name: validate_records
      value: false
    - name: stream_data
      value: true
    - name: disable_collection
      value: true
    - name: add_metadata_columns
      value: false
    - name: location
      value: US
schedules:
- name: shipstation-to-bigquery
  extractor: tap-shipstation
  loader: target-bigquery
  transform: skip
  interval: '@daily'
  start_date: 2020-10-07 16:24:54.035456
  env:
    TARGET_BIGQUERY_DATASET_ID: raw_shipstation
and when I run
meltano elt tap-shipstation target-bigquery --job_id shipstation-to-bigquery --dump=loader-config
I get the following output:
Copy code
{
  "project_id": "vibrant-bonus-233400",
  "dataset_id": null,
  "validate_records": false,
  "stream_data": true,
  "disable_collection": true,
  "add_metadata_columns": false,
  "location": "US"
}
If I add
TARGET_BIGQUERY_DATASET_ID
to the
.env
file, it will take that value, but
env:
doesn't seem to be overriding it for this specific pipeline.