Hi Team, I am using tap-mysql to target-snowflake ...
# troubleshooting
s
Hi Team, I am using tap-mysql to target-snowflake to sync data using Binary Logging. Currently have purged data so there are n number of log files, making sync a little slow. It does sync in batches of 1,00,000 rows out of which 200 approx are insert, rest 99k rows are updates. Making an overall latest data at a lag Any way to make sync faster ??
Copy code
name=tap_mysql level=INFO message=BinLog reader (file: mysql-bin-changelog.004332, pos:117490273) has reached or exceeded end position, exiting!
Copy code
name=tap_mysql level=INFO message=Processed 8403663 rows[0m [36mcmd_type[0m=[35mextractor[0m [36mname[0m=[35mtap-mysql
So basically tap has read 84 lac rows, batching in 1 lac is taking a bit time and would make a batch 84 times. Can we make batching faster any how ?