Hello, I'm running a tap-oracle to target-bigquer...
# troubleshooting
l
Hello, I'm running a tap-oracle to target-bigquery. After running for a while I get this message
One or more records have exceeded the max age of 5.0 minutes. Draining all sinks.
This leads to data being written and it also updates the state which normally only happens at the end of a run. This becomes a problem when I try to run with upsert in bigquery as it will only write to a temporary staging table and the data is not written into the "real" target table until the job finishes. So if the job crashes for any reason, I get a state that is not aligned with my "real" target table. Anyone have any idea on how to solve this ? See bellow for my meltano.yaml file
Copy code
plugins:
  extractors:
  - name: tap-oracle
    variant: s7clarke10
    pip_url:
      git+<https://github.com/s7clarke10/pipelinewise-tap-oracle.git@53bb75ed27d7796d2f492e74cec87f10f1bce4d4>
    config:
      use_ora_rowscn: true
      ora_python_driver_type: thick
      use_singer_decimal: true
      default_replication_method: FULL_TABLE
  - name: tap-oracle-<redacted>
    inherit_from: tap-oracle
    config:
      filter_tables:
      - <redacted>
      offset_value: $TAP_ORACLE_OFFSET_VALUE
      metadata:
         <redacted>:
          replication-method: INCREMENTAL
          replication-key: CREATION_TIME
          primary-key: ID
    select:
    - <redacted>


loaders:
  - name: target-bigquery
    variant: z3z1ma
    pip_url:
      git+<https://github.com/z3z1ma/target-bigquery.git@1737738613c1b7f0c08069d2cc12ea7e40f5a55c>
    config:
      location: $DATA_LOCATION
      project: $GOOGLE_CLOUD_PROJECT
      batch_size: 1000
      method: batch_job
      denormalized: true

  - name: target-bigquery-test
    inherit_from: target-bigquery
    config:
      overwrite: false
      dataset: $MELTANO_TEST_DATASET
      denormalized: true
      upsert: true
I've tried with both method batch_job, storage_write_api and gcs_stage. I also get the same behavior if I don't do upsert, but in those cases it is not really a problem.