Hi folks,
I'm wondering if its possible to tell meltano to keep the change data as well. if not would there be appetite for developing such functionality. Please feel free to point me to a more appropriate channel to ask this question?
Maybe I'm about to get schooled, but to me, this feels like a pretty generic pattern, and would save myself and perhaps other teams to have to specify keeping snapshots in the transformation layer. Also, wondering how other teams go about this.
• for append only streams with incremental loads, this is not necessary, as each incremental load is the change
• for incremental loads where the target entity maybe inserted or updated, its should correspond to the incremental extract.
• for full refresh / initial loads, it would be the difference between the existing data and the new data
IIUC, meltano would put the data extracted into a staging table & then upsert into the final table. This functionality could leverage the staging data & apply a snapshot operation. The code already exists in the various dbt plugins, and would have to be implemented in each of the targets