Love Eklund
09/11/2024, 2:24 PMOne or more records have exceeded the max age of 5.0 minutes. Draining all sinks.
This leads to data being written and it also updates the state which normally only happens at the end of a run. This becomes a problem when I try to run with upsert in bigquery as it will only write to a temporary staging table and the data is not written into the "real" target table until the job finishes. So if the job crashes for any reason, I get a state that is not aligned with my "real" target table.
Anyone have any idea on how to solve this ?
See bellow for my meltano.yaml file
plugins:
extractors:
- name: tap-oracle
variant: s7clarke10
pip_url:
git+<https://github.com/s7clarke10/pipelinewise-tap-oracle.git@53bb75ed27d7796d2f492e74cec87f10f1bce4d4>
config:
use_ora_rowscn: true
ora_python_driver_type: thick
use_singer_decimal: true
default_replication_method: FULL_TABLE
- name: tap-oracle-<redacted>
inherit_from: tap-oracle
config:
filter_tables:
- <redacted>
offset_value: $TAP_ORACLE_OFFSET_VALUE
metadata:
<redacted>:
replication-method: INCREMENTAL
replication-key: CREATION_TIME
primary-key: ID
select:
- <redacted>
loaders:
- name: target-bigquery
variant: z3z1ma
pip_url:
git+<https://github.com/z3z1ma/target-bigquery.git@1737738613c1b7f0c08069d2cc12ea7e40f5a55c>
config:
location: $DATA_LOCATION
project: $GOOGLE_CLOUD_PROJECT
batch_size: 1000
method: batch_job
denormalized: true
- name: target-bigquery-test
inherit_from: target-bigquery
config:
overwrite: false
dataset: $MELTANO_TEST_DATASET
denormalized: true
upsert: true
I've tried with both method batch_job, storage_write_api and gcs_stage. I also get the same behavior if I don't do upsert, but in those cases it is not really a problem.Love Eklund
09/11/2024, 5:23 PM