Meltano

Is there a way to know when a stream is finished loading? I have a situation where we have an open stream handle that we write to, but I don't know which batch is the last batch i.e. when the tap is finished sending records

Or can I just leave the batch method overrides empty and write each record as it is received in `process_record`?

My understanding is this

1. If the tap sends an EOF (ie tap stops with an exit code of 0) you know the stream is done loading
2. If the tap sends a State message you should flush all currently open streams
For the SDK yeah you can just write them each in `process_record`

SDK makes it so you don't have to think about it (I think) I haven't thought about it and I just use process_record and it hasn't bitten me yet

Writing a sql target or something though I'd think about it so :shrug:

&gt; Or can I just leave the batch method overrides empty and write each record as it is received in `process_record`?
Yes - you can certainly do _just_ the process_record() implementation if your target is fine with singleton writes. (Some targets are really bad+slow at singleton writes, which is why the batch implementation is helpful in those cases.)

To <@U06CCB0EUBC>'s point, the SDK is designed to take care of it for you but if you have use cases where you need more control/visibility, those use cases would be helpful to discuss/understand.

Thanks for all these tips! I ended up using the RecordSink as the base class, which cleared out some of the noise around batches. Great work, as usual, on the SDK. This was my first target