Hey Meltano Team, Does Meltano offer anything to track Data Lineage? Any thoughts on how to build this?

@rahul_anand No, not currently. What kind of data lineage tracking would you be looking for?

I am thinking some ways to add additional metadata for each row - something like what time it was fetched, and the source details.

Ah, sure! The closest thing we have right now is the additional metadata columns that are added by PipelineWise targets like pipelinewise-target-snowflake, that are also populated when these targets are used with Meltano: https://transferwise.github.io/pipelinewise/user_guide/metadata_columns.html That would need to be implemented in each individual target, though, which is not ideal. Meltano, since it manages the entire pipeline, could potentially intercept

SCHEMA

and

RECORD

messages sent from the tap to the target, and insert some additional columns (if configured to do so), which would be pretty neat. Would you like to create an issue with some additional metadata columns you'd like to see, so we can discuss this some more and consider it for implementation (by me or anyone else who decides to pick it up)?

Sure @douwe_maan. Will do.

Thank you!