Hi y’all! I am looking to do a migration from Postgres to snowflake using log-based replication. I would like to know if there are any best practices/pro tips/well known techniques for a pipeline that uses log-based replication from the start (to do an initial full dump/sync and then orchestrate incremental uploads using log-based). Some side notes and observations… • in staging, the tables are small enough to manage. ◦ But in production, the tables are pretty huge. • When running the log-based pipeline in staging, it requires a few runs to get all the data migrated over. ◦ Also, within that same migration, one table is ~50k rows in size when it should be >200. Thank you so much for the help 😁 cc: @luke_winslow

My experience has been that it's necessary to do the initial load manually, then set the state data in meltano's internal DB to reflect the correct position in the log to capture data going forward. I'm working with a MySQL source though so I can't tell you have to make that work with a PG source, unfortunately. 😕

@alexander_rougellis Were you able to solve your issue? I have the same task to set up log-based replication for a very large table and would like to hear if you solved the issue. Thanks

Hey @slava_lenskyy what issue are you running into exactly?

I need to set up log-based a replication Postgresql to Snowflake for a very large tables in productions. I was curious if you did something like that. Is it possible to do, what @thomas_briggs described for mysql? I'm not sure if you can set up logical replication in postgresql if table already exists in Snowflake.