Have we ever thought about notifying about backpressure issues like the target getting behind? Here's what I did a while back when I was messing with all of this (snippet of some of my own tap->target magic)

async def _outstream_handler_tap(self, handle, name, procname, queue):
        data = "initial"
        i = 0
        queue_max_size = queue.maxsize

        while data:
            data = await handle.readline()
            line = data.decode()
            if data: 
                await queue.put(line)
            i = i + 1
            if (i >= (queue_max_size*.1)):
                i = 0
                if (queue.qsize() >= queue_max_size*.1):
                    <http://logging.info|logging.info>(f"Queue Size is >10% currently at: {queue.qsize()}. Max Queue size is: {queue_max_size}. Target is behind")

I'm having some back pressure issues and haven't dove in, but it'd be nice to know if the target wasn't keeping up with the tap

Also interested in this idea

Yeah it's fairly common when you're using a tap with lots of data like a database tap, most of the time I am "just" trying to get a feel for what to put the batch_row configurations to ie should we wait for 100k rows before pushing to target-snowflake or 10k or 1M

I have Meltano loading both a MongoDB cluster and an AWS DocumentDB cluster Change Stream into Snowflake. We’re actively migrating from MongoDB to DocumentDB, and the AWS DMS backfills into DocumentDB for that migration effort result in a flood of inserts that our meltano pipeline can’t keep up with