hello everyone, does meltano supports steaming dat...
# getting-started
v
hello everyone, does meltano supports steaming data as well, like replicating mysql to mysql. Similar to Streamsets?
t
@vaibhav_kumar I think you're looking for the LOG_BASED option in the MySQL tap. Each time the pipeline runs the tap will read the changes to the tables you've selected from the replication log send them to the target.
p
@vaibhav_kumar Meltano doesnt currently support streaming in the traditional sense of real time long running replication but if you dont have a strict requirements on timing you can just run very frequent incremental replication jobs, but it definitely depends on your volume of data. If thats what youre after then using LOG_BASED replication (reading the binary logs for changes) on your mysql would be most efficient. Here are two links to get more info https://docs.meltano.com/guide/integration#log-based-incremental-replication and https://docs.meltano.com/guide/integration#log-based-incremental-replication
Also if its relevant theres a proposal to add a similar long running feature to Singer using a
tail
function https://github.com/MeltanoLabs/Singer-Working-Group/issues/7