Does anyone have recommendations re: resource allocation? For context, I'm running tap-postgres -> target-redshift in AWS Fargate -- not Meltano, just raw Singer tap/target. Incremental runs are picking up ~10k records and completing in ~30 seconds. In testing larger replication by modifying the state that I pass in to the job, I'm finding that replicating a larger volume of data is taking a significantly longer time: over an hour for ~200k records. Based on the 30 seconds for ~10k records, I'd expect a roughly 10 minute runtime assuming the increase is linear. My Fargate task at the moment is running with 2048 MiB (2GB) of memory and 512 CPU (0.5 vCPU). I have the batch size at 20,000 records on both the tap and target, so I can also adjust that and see if it has any effect, but any input is appreciated. Thank you!

Taps and Targets most often lend themselves to higher CPU low-med Mem machines due to the fact taps typically dont keep data in memory, its pushed into a pipe which itself has a limited capacity. The target will keep data in memory enough to perform a batch upload (this varies by target) so in 90% of cases your CPU constrained with the only chance of memory constraints being on the target side dependent on implementation. I imagine there is contention for the CPU causing the processes to run inefficiently because the kubernetes scheduler. From the k8s docs

The CPU limit defines a hard ceiling on how much CPU time that the container can use. During each scheduling interval (time slice), the Linux kernel checks to see if this limit is exceeded; if so, the kernel waits before allowing that cgroup to resume execution.

This is still conjecture without knowing much about the specific tap and target implementations but is a safe ground rule.