Singer spec thought: Could we add a message when a tap is finished outputting all the records for a stream? Most targets periodically flush rows when certain size limits are reached. But for streams which are smaller than the buffer limit, they won’t be flushed until the whole pipeline is complete, as the target is still waiting to see if more records are coming. Consider what I think is a common case for taps: a dozen small streams and one really big stream. You want to set the buffer size to be relatively large to get good behavior for the large stream, but the small streams might fall completely under that threshold. If the big stream craps out, e.g. on the first load/a full refresh, all of the smaller streams are lost and will have to start over from the beginning. Sure these are small but if you’re dealing with a rate-limiting API you really want to avoid making duplicate API calls as much as possible. Such a message could be added without breaking compatibility. Targets could optionally choose to flush streams when they see the message. Thoughts?

I like this idea. Kind of an "end of stream" version of EOF that taps could optionally emit. This topic has come up in regards to target development on the SDK. I think this thread sums it up nicely: https://gitlab.com/meltano/sdk/-/issues/172#note_638059462

I think this is a great idea! I was also thinking through ways that we could hint that the stream is done. I had a few use cases where that would be ideal

We are also using the pipelinewise

target-snowflake

, which is what got me thinking about this

@julian_knight and @pnadolny - To solve the "dangling records" issue, we finally added a max-time-elapsed (or "max record age") in the SDK for targets. We started with I think 30 minutes, and I am considering bumping it up to 5 minutes. It seems like possibly having an extra flush of records every 4 or 5 minutes would have minimal impact on performance - and would make scenarios like AWS Lambda more feasible.