Hi,
I noticed tap-mysql is slow at reading changes...
# plugins-generala
Anita Bajariya
06/03/2024, 5:51 AMHi,
I noticed tap-mysql is slow at reading changes from binlog. Tap-mysql is taking 33.91min to read changes from binlog (record count = 12119).
I noticed that the vast majority of the runtime for the tap was serializing the data to JSON.
From what I’ve read Python is notoriously slow at serializing/deserializing JSON.
So how to make tap-mysql more efficient to read mysql binlog changes?
e
Edgar Ramírez (Arch.dev)
06/03/2024, 4:54 PMHi Anita.
Which variant of tap-mysql is this?
a
Anita Bajariya
06/06/2024, 7:45 AMHi @Edgar Ramírez (Arch.dev)
Copy code
variant: transferwise
pip_url: pipelinewise-tap-mysqlAnita Bajariya
06/21/2024, 6:21 AMHi @Edgar Ramírez (Arch.dev) do you know why tap-mysql cab be time taking to read changes from bin log?
e
Edgar Ramírez (Arch.dev)
06/21/2024, 11:27 AMNot sure, the best approach would be to get a flamegraph of tap-mysql running in binlog mode and figure out where the bottlenecks are
3 Views