hello everyone, does meltano supports steaming data as well, like replicating mysql to mysql. Similar to Streamsets?
t
thomas_briggs
03/22/2022, 12:40 PM
@vaibhav_kumar I think you're looking for the LOG_BASED option in the MySQL tap. Each time the pipeline runs the tap will read the changes to the tables you've selected from the replication log send them to the target.
p
pat_nadolny
03/22/2022, 1:43 PM
@vaibhav_kumar Meltano doesnt currently support streaming in the traditional sense of real time long running replication but if you dont have a strict requirements on timing you can just run very frequent incremental replication jobs, but it definitely depends on your volume of data. If thats what youre after then using LOG_BASED replication (reading the binary logs for changes) on your mysql would be most efficient. Here are two links to get more info https://docs.meltano.com/guide/integration#log-based-incremental-replication and https://docs.meltano.com/guide/integration#log-based-incremental-replication
pat_nadolny
03/22/2022, 1:44 PM
Also if its relevant theres a proposal to add a similar long running feature to Singer using a