have other folks encountered OOM errors. We contai...
# singer-tapsh
haleemur_ali
05/05/2023, 3:33 PMhave other folks encountered OOM errors. We containerize our meltano workloads and encounted an OOM error when doing a full load of marketo activities.
what strategies have you employed to work around this error.
would we benefit if meltano provided some sort of flushing & state tracking so that memory is cleared before the machine crashes?
haleemur_ali
05/05/2023, 3:33 PMdo you need a volunteer to implement said functionality?
k
kk
05/05/2023, 3:46 PMA easy solution we came up with is to have different catalog configs per stream.. deploy steams with huge data to containers with bigger specs
t
thomas_briggs
05/05/2023, 4:04 PMWhat target are you loading to? Some of the targets I've used (target-postgres, for example) batch data for all streams and flush them at the end so if you have a lot of streams, and especially if you have a lot of data, you can use a lot of memory.
h
haleemur_ali
05/05/2023, 4:05 PMyeah, that's what we ended up doing. run the big streams on the big machines. we're doing target-s3 for this case, but for a different one, we'll likely do target-redshift.
haleemur_ali
05/05/2023, 4:06 PMbut, the solution is still sub-optimal & inelegant in my view
e
edgar_ramirez_mondragon
05/05/2023, 4:08 PMI agree that taps should be able to receive an OOM signal and stop gracefully (i.e. emit a final state message). @haleemur_ali would you like to log an issue in the meltano repo?
h
haleemur_ali
05/05/2023, 4:08 PMyes. i'll log it & we can continue the discussion there.