michael_cooper
10/07/2020, 4:52 PM# meltano.yml
...
loaders:
- name: target-bigquery
namespace: bigquery
pip_url: git+<https://github.com/Mashey/target-bigquery.git>
executable: target-bigquery
settings:
- name: project_id
value: vibrant-bonus-233400
- name: dataset_id
- name: validate_records
value: false
- name: stream_data
value: true
- name: disable_collection
value: true
- name: add_metadata_columns
value: false
- name: location
value: US
schedules:
- name: shipstation-to-bigquery
extractor: tap-shipstation
loader: target-bigquery
transform: skip
interval: '@daily'
start_date: 2020-10-07 16:24:54.035456
env:
TARGET_BIGQUERY_DATASET_ID: raw_shipstation
and when I run meltano elt tap-shipstation target-bigquery --job_id shipstation-to-bigquery --dump=loader-config
I get the following output:
{
"project_id": "vibrant-bonus-233400",
"dataset_id": null,
"validate_records": false,
"stream_data": true,
"disable_collection": true,
"add_metadata_columns": false,
"location": "US"
}
If I add TARGET_BIGQUERY_DATASET_ID
to the .env
file, it will take that value, but env:
doesn't seem to be overriding it for this specific pipeline.