Meltano

with `meltano run my-tap my-target`, if the target is writing records in batches of 10,000, where does that batch get built up? Is it all in-memory or does it get written to disk?

theoretically it should all be in memory in the target

We’re running meltano in kubernetes and “on paper” I think the size of the batch would exceed the memory available to the Pod :thinking_face:

Paging and things tends to hide a lot of things, so it's possible that's what's happening?

It's also possible that the batch isn't getting filled before something else tells the batch to drain