Hi , I am planning to use log-based replication method for ingesting data from mysql to redshift. The issue that I am facing right now in , wherever there is a major maintenance is scheduled for mysql they create an new RDS instance and switch over it and later rename the end point to the existing ones and dumping the previous instance , but since log-based replication keeps a state based on log-file position . What we can do to avoid error in our pipeline ? @visch , @edgar_ramirez_mondragon

super curious situation. I don't know Places I"d look are the binlog dogs for MySQL, AWS RDS documentation, and I'd probably throw in a ticket to AWS. If AWS is migrating the DB for maintenance I'd think they'd provide a migration plan?

In my experience, the upgraded database ends up in a mixed configuration state after upgrading; config set by connecting to the database directly is persisted during the upgrade, but AWS applies a new default parameter group to the instance to match the new major version, overriding any parameter group changes previously set. This caused me a week of pipeline stalls in my last role as it was difficult to debug and find the root cause. However if you know in advanced that this is going to happen (it isn't well documented at all) it is possible to i) pause your replication pipelines and disable any applications writing to the database, ii) perform the major upgrade, iii) apply a new parameter group matching the target version (requires a restart) to re-configure replication on the new major version and finally iv) re-enable your application and pipelines 🙂