Hi, have been using Meltano ( tap-mysql >> target-...
# troubleshootings
sumit_singh
11/19/2024, 9:25 AMHi, have been using Meltano ( tap-mysql >> target-snowflake). Recently around 2 months back shifted the Source Replication Database from MySQL to Aurora MySQL. It was working fine till last week. Facing issues such as High CPU usage when Meltano runs show binary logs command. The Job are scheduled for every 5 minutes. On turning off the dags, the database metrics goes back to normal.
Has anyone else faced this issue with Aurora MySQL and Meltano ?
👀 1
sumit_singh
11/19/2024, 9:26 AMPS: No Changes in Meltano project except Database Host Endpoint
e
Edgar Ramírez (Arch.dev)
11/19/2024, 7:18 PMHi @sumit_singh! Which variant of tap-mysql are you using?
s
sumit_singh
11/21/2024, 6:01 AMtransferwise
sumit_singh
11/21/2024, 6:02 AMAlso wanted to know, below setting of target-snowflake, does these have any effect on tap-mysql/ MySQL RDS ? CPU threads are of EC2 Instance where meltano runs or MySQL ?
Copy code
{
"name": "batch_size_rows",
"kind": "integer",
"value": 100000,
"label": "Batch Size Rows",
"description": "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Snowflake."
},
{
"name": "batch_wait_limit_seconds",
"kind": "integer",
"value": 120,
"label": "Batch Wait Limit Seconds",
"description": "Maximum time to wait for batch to reach batch_size_rows."
},
{
"name": "flush_all_streams",
"kind": "boolean",
"value": false,
"label": "Flush All Streams",
"description": "Flush and load every stream into Snowflake when one batch is full. Warning: This may trigger the COPY command to use files with low number of records, and may cause performance problems."
},
{
"name": "parallelism",
"kind": "integer",
"value": 0,
"label": "Parallelism",
"description": "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max."
},
{
"name": "parallelism_max",
"kind": "integer",
"value": 16,
"label": "Parallelism Max",
"description": "Max number of parallel threads to use when flushing tables."
},e
Edgar Ramírez (Arch.dev)
11/21/2024, 8:07 PMNo, those settings should have no impact on the MySQL RDS instance
👍 1
s
sumit_singh
11/22/2024, 5:56 AMSHOW BINARY LOGS11 Views