quick question regarding testing incremental streams… I am testing my tap locally using the default jsonl target. I configured a

replication_key

in the stream I am testing and ran an elt on that one stream. I checked the state of that run after it was done and all looked good. Then I re-ran it using the same

job_id

and the state was seemingly picked up by the elt, but the elt job output to the jsonl file anyway, effectively appending the same data from the first run. Am I correct in my reasoning here that the incremental elt is not performing as expected?

Hi @giancarlo_perrone! You probably still have to make sure the state bookmark is used by the tap so only new data is retrieved. For example: https://github.com/MeltanoLabs/tap-stackexchange/blob/main/tap_stackexchange/client.py#L117-L118

So I am trying to see if our internal API supports that sort of query param, but in the case it doesnt… is there another way?

is there another way?

Not really in any meaningful way. You could still try to filter what the tap outputs using the bookmark so at least only new-ish data is presented to the target, but the volume of data traveling through the wire as http responses would be the same