Anyone monitoring schemas with

--discovery

? What I'm thinking is diffing the output of

--discovery

with a tap, comparing that with the current catalog being used (filtered for only the streams I care about) and then notifying different systems alerting that they need to do something about the difference. Worried about *Source system having columns removed (would fail the tap run in most cases I believe) *Source system having columns added (need to update dbt models depending on what the change is)

What's the problem you're trying to solve? Is this something a specific target could detect when it gets the SCHEMA per stream and compares it against the actual data schema in the destination?

The target could detect it, but what would the target decide to do? Problems I'm trying to solve: *When the source system schema changes I want to alert a group of people to let them know. They can then make the choice on what action to take (or not take)

@flat-bear-81546 With Meltano, the catalog is re-discovered on every run and the selection and other metadata rules are applied on the catalog at runtime, so the tap would automatically be run with a new catalog appropriate for the schema at that time. The target would then receive the new

SCHEMA

message and update the destination schema accordingly. If updating the destination schema is not the desired resolution, I agree it would be useful if Meltano were to detect that the schema change and ask you to resolve it manually. Wanna file an issue? 🙂

Right, unless I provide an explicit Catalog. Which is what I assumed I'd do in this scenario to keep things from breaking. Meltano's argument is the opposite which is things will break more if you do provide an explicit Catalog. I hadn't considered this too much, I'll think about it then put an issue in