Hey everyone :smile: Happy friday! Quick question...
# troubleshooting
s
Hey everyone 😄 Happy friday! Quick question - what is the value for get_starting_timestamp when the replication method is full-table? I'm trying to implement a backfilling stream that would run once a week. This stream is setup in the tap to take the value from get_starting_timestamp and pass it as a parameter to the API. This is to perform an incremental sync only on the latest values. Initially (on the first run) this tap performed a full backfill, but then started behaving incrementally. Which brings me back to my questions: • Does
get_starting_timestamp
take into account the replication method? • How can I ensure that the starting timestamp for my backfilling stream is always
start_date
? • Where in the code is the metadata from the catalog applied to the tap? Can the catalog be overwritten by the tap itself? Thanks! (Update) When running with these settings:
Copy code
metadata:
        "*":
          # replication_key: updated_at
          replication_method: FULL_TABLE
Stream still seems to be registered as incremental
Copy code
Beginning incremental sync of '<stream_name>'...
Is this normal? Note: I do manually set the replication method in my tap:
Copy code
replication_key = "updated_at"
    replication_method = "INCREMENTAL"
With the expectation that I can overwrite this in the metadata