This is exciting! https://github.com/datafold/data-diff

Oh that is a cool thing to open source!

cc @visch 👆

Thanks @aaronsteers I already commented on an issue, it's an interesting approach, simple!

Having built something like this for a previous company, this is super interesting to have in the open 👏 Would be amazing to release a Utility on the hub, and then raise a PR to their README under Use Cases to add us to the list:

• Verifying data pipelines. Moving data from a relational database to a warehouse/data lake with Fivetran, Airbyte, Debezium, or some other pipeline.

Also, detecting hard-deletes for pipelines running incremental replication (rather than log-based) is a low-key game changer.

• Detecting hard deletes for an

updated_at

-based pipeline. If you're copying data to your warehouse based on an

updated_at

-style column, then you'll miss hard-deletes that data-diff can find for you.

Incremental replication is convenient but lossy, but this plugs the gap! I wonder how feasible it would be for

data-diff

to output a Singer-compatible stream of deleted records (either for removal or delete-flagging) 🤔

Sounds like a great hack day project! 😄

Great minds 😅 https://github.com/meltano/internal-general/discussions/388?converting=1#discussioncomment-2956454