I'm trying to use a tap which doesn't actually have insight into the schema of the source data, so it emits a generic

"schema": {"type": "object"}

. I thought that by specifying the schema extractor extra in meltano.yml I could provide the actual schema of the data in question. However, even after specifying it the loader crashes when it sees an empty schema message.

meltano select --list --all

returns the table but doesn't list any columns. If my understanding is correct, the metadata+schema are simply passed back to the tap via catalog, so if the tap isn't using them when building the schema messages, that means the tap is at fault, right?

Meltano currently only overrides schema for streams/properties that already exist in the discovered catalog, it doesn't add properties to the schema that weren't actually discovered

the tap is dynamodb which has no documentation at all! 🙃 but based on some experimentation it seems it expects you to set the schema in the catalog definition. Putting it in the catalog doesn't change the tap's discover behavior but it does change it's run behavior. I'm not familiar enough with the spec to know if this is in line with the singer spec but given this is an official tap I think that maybe appending schema properties to the catalog makes sense. I'll open the issue

Thanks! The spec allows for taps that don't implement a (useful) discovery mode but still take a catalog, so implementing deep-merge behavior makes sense to me!