Hey all, I've got a 67 GB table that I'd like to replicate incrementally. Some details • tap-mssql target-snowflake • incremental key: last_updated_date The initial load on this thing is killing me 😮 can I use another service for the initial load into Snowflake, and then have Meltano handle the incremental from there? Meltano won't go trying to replicate the whole table over again, will it?

If you're sure to update the state after the manual load you should be good!

What's the main pain @connor_lough ? Slow or just the finicky setup. Looking at you ❄️! The meltano variant is somewhat easier than the pipeline wise variant, with S3 in the middle…

The upload is just going slower than I expected. Think Airflow has run about ~22 hours so far and I've got 50% of the 25mil row x 300 col table in. Factors that could be affecting this: batch size: 100,000 s3 archive: true file type: parquet snowflake warehouse size: small parallelism: -1 I'll give the manual update of the meltano job a shot 👍🏻

Add network to your list 👊

Not having any experience with Snowflake, but the general high level view for investigating this is: • Is the source performing as expected? (What are the wait stats on your source looking like? This might already give you a good indicator of what's going on) • The tap-mssql uses the

Cursor.fetchone()

function from

pymssql

. I wonder if

Cursor.fetchmany(size=)

would be more efficient for doing extracts. • How does your meltano box look like from a resource perspective? The Meltano SDK doc mentions

viztracer

as a tool for performance tracing. https://sdk.meltano.com/en/latest/dev_guide.html#testing-performance • As mentioned already, what does your network topology look like • Is your destination (Snowflake) performing as expected. I have no experience with Snowflake, but I'm sure there are performance logs available.

I appreciate the notes @christoph, but before I investigate deeper, I should make sure I've setup incremental loads correctly 😅

Yeah. With ‘meltano elt’ you need to supply the state id, ‘meltano run’ does this for you