I have added `replication-method: INCREMENTAL` in ...
# troubleshooting
f
I have added
replication-method: INCREMENTAL
in my meltano.yaml file. But it always shows 'Beginning full_table sync of 'Meltano_Data_Source'' in logs. Here is my meltano.yaml file:
Copy code
version: 1
default_environment: dev
project_id: add13831-5899-498c-b871-85e655dbe1e0
environments:
- name: dev
plugins:
  extractors:
  - name: tap-google-sheets
    variant: matatika
    pip_url: git+<https://github.com/Matatika/tap-google-sheets.git>
    config:
      replication-method: INCREMENTAL
      replication_key: Id
      child_sheet_name: Sheet1
      key_properties:
      - Id
  loaders:
  - name: target-gsheet
    variant: singer-io
    pip_url: target-gsheet
    config:
      spreadsheet_id: 1yyVsEGVfdfdfdfdfdfdfdfdfdfdf
  - name: target-csv
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/target-csv.git>
v
not all taps support incremental replication. For google sheets, I'd stick to full tables. You're almost certainly talking about less than 10 million records
💯 1
âž• 2
r
FWIW I think it's possible to implement if you want to open an issue @Feroze Ahmad. In theory, this could be done fairly easily with respect to the file last modified date, but that would be more like "only sync the file if there are changes" (i.e. sync the entire file, but only when we know something has changed). I think it could also be done with sheet data, but that is definitely more of a lift.
âž• 2