Hello , Is there a way I can refresh a meltano table starting from a particular timestamp. Like I specify a timestamp and meltano syncs a table starting only from that point onwards

You could achieve that by tweaking the state data in the internal meltano database. The new

state

command might make that easier to do but I can't remember whether that's been released or not.

Thank you Thomas. Would the key atttribute replication_key_value in payload column indicate time till when table was synced. Would updating attribute make the sync to start from this timestamp

replication_key_value is the last/highest value in the replication_key column that was replicated. So if your replication_key is a datetime then you can set replication_key_value to the time you want to re-start replicating from. If that column is just an int (e.g. a monotonically increasing ID) you can set it to the ID value you want to start from. etc. etc Just be sure to remove all the data from the destination first - re-retrieving all the rows from the source won't remove them from the destination so if you don't clean up your target you'll end up with extra/duplicate rows.

Oh ok. Why data needs to be removed from the destination? I was under impression Meltano would do a "MERGE" operation based on the primary key. Existing records will be updated while new records found would be inserted. Yes I get that deleted records from the source would not be replicated. Thank you for all the clarifications. This really helps. Thank you !

If your table has a PK then existing records will get updated, yes. Not all tables have a PK though. 🤪 And if records were deleted from the source then the target won't know they existed so they can't be removed. Cleaning up the data in the destination means you don't have to worry about any of that. If you haven't deleted any rows and have a PK you don't need to worry about it though. And you're welcome! 😉