I’m using transferwise’s variant of

tap-mysql

with

LOG_BASED

replication. I’ve got about a dozen meltano pipelines running in a staggered fashion such that at any one time there are no more than 7

elt

pipelines running at a given time. Each of these pipelines is accessing a distinct database on the same AWS RDS MySQL instance. The first time I ran this series of pipelines, most pipelines completely successfully (some errorred for unrelated reasons). The second and third days I ran this series of pipelines most of the pipelines error with the following

meltano | Extraction failed (1): pymysql.err.InternalError: (1236, "A slave with the same server_uuid/server_id as this slave has connected to the master; the first event 'mysql-bin-changelog.418935' at 777596, the last event read from '/rdsdbdata/log/binlog/mysql-bin-changelog.418937' at 737078, the last byte read from '/rdsdbdata/log/binlog/mysql-bin-changelog.418937' at 737078.")

So, several were running in parallel at a given time and completed successfully, but most produced some variation of the error above at various points during the extraction. There are several recommendations online for how to solve this, but I’m not sure which one to attempt. Some recommend changing server configuration which may not be possible since I down’t manage these source dbs. Some suggest that the server_id needs to change for each concurrent call, which doesn’t seem to be controllable using this tap and much of the calls seem to be controlled by the state emitted by meltano … any thoughts?

Hey, I’m seeing the same set of issues. My current workaround is to set retries (I’m orchestrating with airflow). @josh_lloyd did you ever get a resolution?

Never did find a way to run two or more pipelines against the same MySQL instance at the same time. In the end, I opted to run all pipelines against an instance serially to avoid this error. Not ideal, but none of the other solutions seemed plausible to me.

Did you ever try to play around with the session_sqls parameter?

Ok got it. One more thing - what version of aurora mysql are you using? I’m asking because if you have 5.6 I might have follow up questions about parameters, as could more easily compare my working parallelization with your non-working one in 5.6 (vs comparing my working 5.6 with my non-working 5.7).