Hi all! I have some fundamental questions regarding meltano. 1. From what I have seen, the state is written at the very end when batch record is written to target, so what if after extracting a large no. of records, it fails? My state is persisted in s3. 2. Do we have the capability to define

start_date

per table level in my

meltano.yml

for initial snapshotting ? 3. Is overriding batch size supported on SDK based tap sides by default ?

1. State "should" be written "often" as it's supposed to keep track of where you're currently at so that you don't have to start from scratch on your next import. Behavior depends on the target 2. Depends on the tap, generally no, but you could just use meltano's inheirt_from and specify a different start date for that "instance" of the tap 3. Not sure

Thanks Derek, do we have any examples of 2nd point ?

@silverbullet1 take a look at Multiple plugin configurations

I'd also add that my understanding is that taps that dont explicitly define that they have sorted streams will wait until the end of the sync to emit state. So partial run state isnt saved if it fails midway. If the stream isnt sorted then the tap cant be sure that all records up to that state bookmark were synced until its fully complete so its unsafe to emit state before then or data loss could occur.