Hi team! I am doing an initial load of some large tables (100-200 million rows) in MySQL to Snowflake , and it's taking a long time. Looking at the query from Snowflake's side, it's taking in a very small amount of data (2MB) every minute . Is there a way to shorten the pause between the mini-batch loader? Thanks! -David L. from Justworks Inc

Hi @david_lee. The MySQL tap is generally really slow about extracting data from the DB and formatting it to JSON (which is necessary to send it to Snowflake target). That's not a huge problem for incremental loads, usually, but it makes large initial loads like this take forever. 😞 Increasing the batch size on the loader side doesn't help, unfortunately, because that doesn't impact the behavior of the tap.

@thomas_briggs, do you know if Meltano already supports Fast Sync?

As far as I know that's a Pipelinewise-only feature. 😞 It would be nice to have though. 😉

set the state data for the table in the Meltano DB to make it look like Meltano loaded it

Would you mind pointing me to the right direction on how to achieve this?

Well... my process is a little out of date and involves manually manipulating the JSON stored in meltano.db (the SQLite database meltano uses for internal data). There is thus no documentation on that. 😉 I think you can achieve the same thing in a clean way using the state command though - see here. I haven't tried it myself but I expect the process is 1. Use

state get

to dump the current state data to a file 2. Modify the state data to include data for the newly-loaded table 3. Use

state set

to import the updated state data into Meltano That assumes you've already run the pipeline successfully at least once such that there's state data to export/change/re-import, BTW. So load a small table normally to establish some state and you can go from there.

@thomas_briggs is absolutely right. Unfortunately Fast-Sync is a feature of pipelinewise and not their individual connectors. That’s why we’re working on an extension to the singer spec for BATCH messages: https://github.com/meltano/sdk/discussions/859

I'll add to @edgar_ramirez_mondragon's comment above that we do plan to prioritize an SDK implementation of batch message support, hopefully starting development within the next 2-3 weeks.