I am using LOG_BASED replication to ingest data into clickhouse. One thing I notice was that only new changes are ingested to clickhouse such as insert/update/delete after setting up replication. To bring already existing data to do i need to first use FULL_TABLE replication ?

Log based reads the logs to push only changes by design, maybe you could run an inital full table replication and changes right after that, though it can be though to find the exact point at which you took the snapshot to start the CDC from that point forward

@pablo_seibelt, do you think running with --full-refresh might do the trick ?

Only if your binary logs have the whole history of your dataset, which i doubt, most likely you have some retention policy that has just a fraction of it

Yup metlano failed to live up to my expectations Things don't work as expected most of the times

Meltano is amazing for replicating data from external services (e.g. APIs), i don't think database CDC replication is the best usecase, specially if you have a big database

I would definitely agree that not a good tool for database replication it have definitely given me alot of pain.