I'm still learning a lot about Meltano and Singer. But trying to gauge a full refresh/initial load and how much compute resources I'd need to do so. Does it depend on the connectors to handle batching/streaming? Or is there some default behavior? For context I'm using

tap-zendesk -> target-athena

Is there a resource somewhere that someone can point me to, that describes how data is extracted and loaded. i.e. is it one stream/table loaded into memory then loaded and forgotten etc. TIA

I think this will depend on the target. The SQL targets I’ve used (meltanolabs target-Postgres and transferwise target-redshift) read streams into memory up until a configurable batch size or time limit, then write that batch in one operation

Thank you