Hi all, I am wondering if any one ever done incremental tap and write a a different file csv each time ‘meltano elt’ ? This thought just cross my mind and I a not it the state that can test this. Can anyone share some insight on this ? Thank you

Yes, this sounds pretty standard. The target-csv (or a similar target like

target-s3-csv

) would generally add timestamps into the file names, and would only write new records received from the tap. If naming conventions on the CSV files are timestamped, the only other thing to do is to make sure you’re passing in state to the tap and enabling the incremental behavior at the tap level.

Yes that pretty standard, that what usually is. My connection was bad so i cant test with s3 really at all so i tested with jsonl tap. Which later on when i review the file, the overall pipeline seems off

For jsonl, I think you just need to turn on the

do_timestamp_file

setting (set to True)

Oh cool, thanks

The docs say that is on by default but when I tried it, it seemed to require being manually set to true.