does meltano have to go through the full sync of a stream before loading begins by the target? for some reason I thought that meltano would pipe to the loader while streaming. (using a custom tap from the SDK)

meltano streams the data, yes! What are you seeing?

im just seeing the tap request logs for each pagination loop for the past 1.5 hours -- no loading information

So it does depend, a way to tell for sure is to just run somethign like

meltano invoke tap-braintree > out

and then tail the output file to be sure data is flowing. But generally yes the tap will continue pushing "record" data to the target. The target also depends on your target and settings. Most of the time the target will have some kind of "batch records" count. The targets will also "flush" the current data into the target (normally) if a "State" message is sent from the Tap as that (should) trigger the target to write everything, then send its own state message

amazing, thanks derek. im guessing it's the fact that

batch_size_rows

in my snowflake target is 100k