Hi, I've been getting some problems due to this function _handle_max_record_age https://github.com/meltano/sdk/blob/6708cb995c68ab6f74d4874dfc8f978c3b054ceb/singer_sdk/target_base.py#L284 So I thought it would be interesting to understand why it works like it does, if someone knows. My problem is cause by the fact that it causes targets to write out states in the middle of a run. Is it there to "checkpoint" runs in that way or what is the reason for this? and what is the reason drain_all() (https://github.com/meltano/sdk/blob/6708cb995c68ab6f74d4874dfc8f978c3b054ceb/singer_sdk/target_base.py#L477) writes out a state?

Is it there to "checkpoint" runs in that way or what is the reason for this?

Yes https://hub.meltano.com/singer/spec#state-files-1 https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md#state

and what is the reason drain_all() (https://github.com/meltano/sdk/blob/6708cb995c68ab6f74d4874dfc8f978c3b054ceb/singer_sdk/target_base.py#L477) writes out a state?

For the above reason. Not sure any of that helps, but happy to answer any other questions you have

It's there to ensure targets commit records and update state frequently, rather than hoarding tons of data in memory which would be lost if the target crashes for any reason

Thanks for the links and answer. It makes a lot of sense. I think the target I'm working with handles it in a somewhat unexpected way when upserting. It writes to a temporary staging table and merges in the data only if/when all the data has been written to the staging table. This leads to the state being moved forward without any data being written to the target table.