I'm looking to determine if tap-mysql is suitable for my use case. I have a 1TB MySQL database with multiple large tables, and I'm aiming to develop an EL pipeline using TransferWise's tap-mysql and target-postgres. While setting this up, I noticed that tap-mysql is taking a significant amount of time to read from the biglog file. For instance, it took 29 minutes to process 40,233 records. Proccessing time veries if tap-mysql has to read changes from biglog file & change log event is for very large table. Could anyone provide suggestions or insights regarding this use case?

The first thing I would check is if the bottleneck is on the tap or target.

@Conner Panarella (SpaceCondor) I checked that. Loding data is taking few seconds only. I put Python profiler to see where the tap was spending its time. The vast majority of the runtime for the tap was serializing the data to JSON.

SDK-based variants might benefit from https://github.com/meltano/sdk/pull/2541 for faster serialization in a future release.