https://linen.dev logo
#plugins-general
Title
# plugins-general
j

jean_sahlberg

06/01/2022, 2:02 PM
MySQL Hello! We are using MySql tap to move to a Snowflake target, it works, but I found something strange that I can’t fix, I played with the settings but none of them seems to fix the issue: We have ~500 tables to extract from MySql and load to Snowflake, it’s a long process (takes hours to complete) and it is running in 100k rows batches… the issue we have is that if a table has, for example, 630k rows, we will get 600k rows (6 100k batches) but the last 30k rows will be added to the table only when the sync finishes… Meltano will move to the next table and will start adding the rows and will stop in the same way (just “full batches”), only when the pipeline ends that it will add the missing rows, I don’t know why it is this way… if a table has less than 100k rows, it will be empty until the pipeline ends and adds all the rows.