I have just spent time doing a dagster upgrade with the dagster_ext only to find I'm having more memory issues and jobs failing on my most intensive workload 😞. It does involve opening some large CSV files. Running the job in the container directly seems fine, although error is intermittent. I am already setting a small batch size which I will look to reduce:

- name: target-postgres-small-batch
    inherit_from: target-postgres
    config:
      batch_size_rows: 5000

Are there any other quick wins I could use to limit memory usage for just this one job? I'm not too bothered about run time but maxing out on memory on my 4GB container which I can't increase without a quota request to Azure.

It's probably the tap you're using, I'd look at the processes as it's running and verify that. Then look at a different tap or the reason why. my guess with the tap is the files are all sticking in memory. tap-universal-file might be an ok over to look at to but I haven't ever had your issue so I don't know if it's optimized for that or not

Thanks, I moved to a different docker deployment method (App service, not container apps) and that gave me increased memory quota