Happy Friday! If I wanted to use the new batch mes...
# singer-tap-developmenth
hunter_kuffel
11/04/2022, 7:19 PMHappy Friday! If I wanted to use the new batch messaging capability for one stream within a tap, but not the other streams, is that configurable? I have a tap for our email service provider where most of the streams are just standard rest APIs and aren’t problem children, but one stream generates a downloadable CSV file that could definitely benefit from batching
a
aaronsteers
11/04/2022, 7:23 PMHi, @hunter_kuffel. As of now, no, stream-level opt-in or opt-out of batch messaging is not possible. Although we would be happy to support anyone who is interested in contributing the feature.
The recommended workaround is to declare a second instance of the tap using
inherits_fromselectaaronsteers
11/04/2022, 7:25 PMOften users will do something similar for 'large tables' vs 'small/fast tables' - because this gives better retryability for the 80% of streams that finish much faster than the smaller-number of very large tables.
aaronsteers
11/04/2022, 7:25 PMLet us know if this sounds like a decent solution for now, or not.
e
edgar_ramirez_mondragon
11/04/2022, 7:44 PM+1 to @aaronsteers’s comments above. Also, if this your own custom tap, you could look into overriding get_batches for the one stream.
h
hunter_kuffel
11/04/2022, 9:02 PMThank you both! I think for now I will just have separate instances, as that seems easiest. I would definitely be interested in potentially contributing the feature though, if you have any initial thoughts around it I am all ears!
a
aaronsteers
11/04/2022, 9:26 PMSure thing! I've put some ideas here in a new issue; feel free to propose alternative approaches.