Hey I’m running into a weird issue with incremental state for a

tap-zendesk

to

target-bigquery

pipeline. I followed the docs to add Meltano to an existing Airflow installation and the generated DAG sets the number of retries to 1 (maybe this should be 0 instead?). One of the streams fails to sync, for reasons not relevant to the issue here, but that means the retry logic gets triggered. On each run, Meltano successfully finds previous state for the first try and fetches only recent data, but on the retry it can’t seem to find the state and does a complete import. See logs below for when it does a full import:

INFO - Temporary script location: /tmp/airflowtmpcpi5tttv/extract_to_bqh43uwc2b
INFO - Running command: meltano schedule run zendesk-to-bigquery
INFO - Output:
INFO - meltano         | Running extract & load...
INFO - meltano         | Found state from 2021-04-09 04:51:00.688030.
INFO - meltano         | No state was found, complete import.

Yea, I think I saw this in testing, hopefully it gets fixed soon. You can try pointing to the hotfix branch in the meantime: https://github.com/adswerve/target-bigquery/tree/hotfix/issue9 http://meltano.com/docs/plugin-management.html#using-a-custom-fork-of-a-plugin

oh yeah good idea! i’ll give that a shot