Hi Guys! I’m using meltano el command to extract and save data, but we’re running it in parallel, and noticed that each

meltano el

run consumes lot’s of memory, is there a way to optimize it ? maybe other command or something ? Thanks!

noticed that each

meltano el

run consumes lot’s of memory

I bet either the tap or target either is buffering a large amount of data in memory. Which loader are you using? It may have an option to increase the frequency with which data is committed and removed from memory.

I’m useng custom elasticsearch target, I’ve checked memory it’s not like e single item is consuming a lot, but we’re running multiple meltano el and together it’s get pretty big, we have a 2gig ram instance and it can only run 7 meltano etl at a time, the amount of data is like 100 records in each, it’s tiny it’s a break down for 7 meltano el running in parallel

target-elasticsearch: 878MB (40.4%) - Your biggest memory consumer
meltano orchestrators: 825MB (37.9%) - High overhead from process management
tap extractors: 458MB (21.1%) - Data extraction processes
shell wrappers: 13MB (0.6%) - Minimal overhead