Hello! Is there any paper to read about Meltano state practices to manage file deletions? My current problem is that Tap only handles new and modified files (using meltano state), but deletions are ignored. Since my target is vector db, I need to delete vectors from the db if they are no longer present in the original storage.

Some taps support ACTIVATE_VERSION, but I don't know of any file-source tap that supports it. It sounds possible, like you say, using the state to record existing files. How would you decide which rows/tables to delete from the target system based on the files that were deleted?

Thanks for answer In my personal view, deletion inside target storage could be performed in following way: 1. Tap provides data about deleted file, for example

{action: to_delete, name: new.pdf, id: 1}, {...}

2. Target reads this row and perform some search based on unique field, in our case it will be

id

3. Last step - delete found data objects from target. If we have structure 1 file = 1 data object inside target storage, then just delete single data object. In my case it is 1 file = many vectors, but every vector have assigned metadata with required

file_id

so this is not a big deal to find exact vectors to delete. On the other hand, the question is: How meltano tap will detect deleted files? We have saved storage state as simple JSON file. 1. During first iteration we can perform the same way we do previously (just detect new and modified files based on some field, for example

dateModified

) 2. Second iteration we'll go through state items and check if every item is still inside source storage. If we met state like this:

[
  {name:1, id:1, ...},
  {name:2, id:2, ...},
  {name:3, id:3, ...}
]

And file

name:2

and

id:2

is not longer represented inside source storage -> we push message with some key like

action: to_delete, id:1, name:1