https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
j

julian_knight

09/14/2021, 7:11 PM
Singer spec thought: Could we add a message when a tap is finished outputting all the records for a stream? Most targets periodically flush rows when certain size limits are reached. But for streams which are smaller than the buffer limit, they won’t be flushed until the whole pipeline is complete, as the target is still waiting to see if more records are coming. Consider what I think is a common case for taps: a dozen small streams and one really big stream. You want to set the buffer size to be relatively large to get good behavior for the large stream, but the small streams might fall completely under that threshold. If the big stream craps out, e.g. on the first load/a full refresh, all of the smaller streams are lost and will have to start over from the beginning. Sure these are small but if you’re dealing with a rate-limiting API you really want to avoid making duplicate API calls as much as possible. Such a message could be added without breaking compatibility. Targets could optionally choose to flush streams when they see the message. Thoughts?