Hello, I'm developing a tap (Rest API) what I would like to use with pipelinewise-target-redshift loader. The problem is that when my tap emits STATE message the loader is still waiting in the loop to get the expected (batch_size_rows) amount of lines from the tap. However I cannot see how to close/flush/etc explicitly the stream what would allow the loader to load the arrived records after the STATE message. For my tap I used cookiecutter and followed the recent tutorial. However in an another project I use tap-jira with the same pipelinewise-target-redshift loader and there is no such problem. But as I see that tap-jira is a different (old?) implementation where the developer maintains and sends the STATE message. Can you please advise? Thanks

Hi Peter! So, to confirm: do you need the target to process record batches more frequently, i.e. as soon as it receives any target message?

Hi, thanks for the quick reply! So not exactly I just don't know why it hangs when it got records than a state message. Target is still waiting while there is no more records from the tap. How can I (explicitly) close the ceremony and let the target write the records? Of course if I set the batch_size_rows to e.g. 10 then the target loads every 10 records and hangs up when there is no more. While this doesn't happen with the tap-jira which has a different implementation comparing to the cookiecutter template and functions. I tested my tap with the test_core and everything looks fine. Also if I load the output of my tap as a text file the loader works fine.