We have a scenario where both the source and target DB is Bigquery.We are trying to extract data from BigQuery(source DB) and load to BigQuery (target DB), the approach we are using is Upsert: true and we have set denormalized: true also. When we insert new records in source DB and run the pipeline then those records are correctly populated in target DB, but the issue is when we update any existing records in source DB, it does not update that in target DB instead it inserts as new records (making it as a duplicate record in target DB). Given below is meltano yml used in this case. Please share your thoughts on how to resolve it. Sample YML: extractors: - name: tap-bigquery-load-bronze variant: anelendata pip_url: tap-bigquery config: streams: - name: ** table: ** columns: - '*' datetime_key: _sdc_extracted_at credentials_path: **.json start_datetime: 2019-03-26 153339.217317 UTC start_always_inclusive: true inherit_from: tap-bigquery loaders: - name: target-bigquery-load-silver variant: z3z1ma pip_url: git+https://github.com/z3z1ma/target-bigquery.git config: location: us-central1 project: ** dataset: ** denormalized: true upsert: true method: batch_job credentials_path: json inherit_from: target-bigquery

Does your source stream define a primary key? https://github.com/z3z1ma/target-bigquery?tab=readme-ov-file#upsert-aka-merge

yes

Can you share the ddl for the source and target table?

create table br-dw-np.bronze.test( EmployeeID INT64 primary key not enforced, FirstName STRING, LastName STRINg, _sdc_extracted_at TIMESTAMP); --target table create table br-dw-np.silver.test( EmployeeID INT64 primary key not enforced, FirstName STRING, LastName STRINg, _sdc_extracted_at TIMESTAMP);