Hi there,
When running meltano with tap-hubspot an...
# troubleshootings
steven_herweijer
10/11/2022, 7:32 AMHi there,
When running meltano with tap-hubspot and target-postgres, the target-postgres keeps gobbling up more RAM (gigabytes of it) until the entire system runs out of RAM and becomes unresponsive. Is there something I can do to prevent this?
t
thomas_briggs
10/11/2022, 12:45 PMAssuming you're using the pipelinewise target: I believe each destination table gets its own DB connection, and each DB connection gets its own copy of the DB schema. Plus the last batch of rows for each table is held in memory until the end of the process, when they're all flushed together, So if you have lots of tables that all adds up to lots (and lots) of memory.
thomas_briggs
10/11/2022, 12:47 PMI know I ran into that when I first started working with Meltano, and I know I spent some time investigating it, but that was like... 8 months ago. So I don't remember the details precisely but I'm pretty sure the problem is ultimately caused by loading into lots of tables.
a
aaron_phethean
10/12/2022, 9:16 PMMight be a long shot, try reducing the batch size on the pipelinewise target. Default is 100,000 from memory and I suspect HubSpot rows are pretty big.