For folks deploying meltano on Azure App Jobs or A...
# dockerh
haleemur_ali
09/20/2024, 1:52 PMFor folks deploying meltano on Azure App Jobs or AWS batch (or comparable service). How do you handle scaling up your container if it encounters an OOM error?
v
visch
09/20/2024, 2:01 PMfor aws batch I up the memory
visch
09/20/2024, 2:02 PMGenerally if you need to do that though there's something up with your tap/target as the design of taps/targets and singer is not memory heavy
h
haleemur_ali
09/20/2024, 2:46 PMI see. would the memory consumption be roughly constant even if the volume of data fluctuates?
I would assume that the memory usage would fluctuate
ā¢ with # of rows transferred (e.g. if on some days the source system sees an abnormal amount of updates or a full refresh is triggered). i'm basing this assumption because bulk loading would require rows to be buffered on the target side, and I'm not sure if the tap will generally fetch each "page" of data, pipe it over and then free that memory before fetching another "page" of data.
ā¢ as additional fields are added to the stream (since each record now needs more memory)
v
visch
09/20/2024, 2:52 PMreally it should only max at the buffer size for the target
visch
09/20/2024, 2:52 PMPersonally I'd just bump the memory and not worry about it until it becomes obnoxious or too costly
ā
2
visch
09/20/2024, 2:52 PMServers are cheap people are expensive
ā
1
l
Love Eklund
09/22/2024, 3:04 PMWe run in cloudrun jobs and do one tap one table. It makes config and scheduling a bit more verbose. But we don't have to think about things like scaling containers when adding more tables etc. I also think it makes state management a bit easier as you have one state file one table.
Love Eklund
09/22/2024, 3:11 PMworth mentioning is that we use a target that writes out for every X thousand records recieved. So we shouldn't get problem with a lot of records for one day.
We could of course experience problems if someone add a lot of new fields. But as we do one table one tap it would only impact that table and then we can create a new upscaled version of a cloudrun job to use for that table. But haven't run into that issue yet.
h
haleemur_ali
09/22/2024, 3:33 PMyes. im a big fan of splitting containers by stream. i do this through tap inheritance, and the meltano invocation looks like meltano run tap-source-streamname target-destination
š 1