Is there a way to reset a table state ? suppose i am loading a table from mysql to redshift using log_based. A ned column "col_1" gets added to mysql table with historical data . but if i am doing log_based load it will just add the column and populate the data in column for new rows and will ignore old rows. If i do a full load and then log_based then the table will not be in sync. How to get this done ? Note: I dont have a primary key in the table

i'm not entirely following the setup, but you can run

meltano elt <tap> <target> --full-refresh

to run the whole pipeline without any state. this should do a backfill of your data as long as the tap is emitting it

@stephen_bailey but , next time if i run log_based from existing pipeline there will be a chance of data duplicacy /data loss right ?

i haven't used log-baed replication, but i would think that it would cover the case where: 1. Table is fully synced with historical job. 2. new column is added 3. historical data is INSERTed 4. new data is updated 5. Table is synced again, starting from last log position. Steps 2, 3, and 4 would be int he database logs and get replicated, right?

no , log_based replication skips the historical data