Can I use Meltano to stream real-time data into something like a Kinesis firehose stream? It seems that the Singer target 'target-kinesis' could load the data into the firehose stream, but I was wondering what the extraction side of things would look like 🤔 would Meltano just have a Singer tap sit on the data source I'd wish to extract and write the data to stdout until I tell it to stop? Big fan of this project btw 😁

would Meltano just have a Singer tap sit on the data source I'd wish to extract and write the data to stdout until I tell it to stop?

That'd work! You could just stop

meltano elt

entirely using

SIGINT

(

Ctrl-C

), which will cause the job to fail from Meltano's perspective, but your data will still have been imported, or you could have your tap look for some external signal so that it can fail gracefully.

Thanks for the quick reply! This is great news. As for how I'd be receiving/accessing the real-time data, I was just planning on tapping various tables that get more rows added to them over time

@joey All right! DB taps will just keep running until they run out of data to extract, so you should be good 🙂

Interesting... so if my DB tap extracts all the rows in a table before more rows are added, would it stop running? Or would it wait "at the end" for more rows to be added and extract them as they get added? I appreciate the help! 🙂

@joey It'd be up to the implementation of the tap, but I'd expect most to stop when they're out of rows without waiting for new ones. It's more typical to have a pipeline run on a schedule, where each next run picks up where the last one left off and syncs any rows added in the mean time. You could set the schedule interval to something very low (like a minute), so that realtime syncing will only be interrupted for a moment if the tap catches up with the data source.

Wow, thank you so much! This is super helpful yay

@joey Good luck!