Hi All, I'm just upgrading from 3.1 to 3.7 including all the plugins to their latest versions and the pipelines stopped working. I have a custom REST tap with upgraded singer-sdk from 0.22 to 0.45, and target-postgres (meltano) and target-jsonl (andyh1203) on the other side. Both taps are now failing on missing required properties and it looks like they started validating the data against the JSON Schema provided by the REST tap, but meltano is configured to only pass a subset of properties (

select

key in extractor config in meltano.yaml). In short, meltano provides a keyword that makes data not follow its schema, and then plugins cant validate the data and crash. That doesn't sound right, what am I missing?

what's the command you ran that failed?

meltano run tap-gsmtasks target-jsonl

or

meltano run tap-gsmtasks target-postgres

When running with postgres, the wording was different, but in essence it complained about missing required field. tap-gsmtasks is my custom rest extractor

I've downgraded everything and I have the same issue

If your

pip_url

is not something like

-e <path to plugin directory>

then the catalog may be cached. Can you try

meltano run --refresh-catalog tap-gsmtasks target-jsonl

In short, meltano provides a keyword that makes data not follow its schema, and then plugins cant validate the data and crash.

Do you know what's the keyword in question?

I meant

select

keyword in extractor configuration

and can you share the schema validation that failed? Where's the missing required field coming from?

1. a rest extractor gets a json with a list of json objects and knows the json schema, against which the response validates 2. meltano configures

select

which omits one of the fields required by that json schema 3. jsonl and postgres targets fail complaining about missing value, in both cases it's one of the properties that's required by the json schema but not listed in

select

Gotcha. If you run

meltano select tap-gsmtasks --list

, do you see that field being selected? I guess I'm trying to pin down whether it was the Meltano or the SDK bump that surfaced the issue. It it was Meltano, it means it was previously selecting the field even if it was absent from

select

, but if it was the SDK, it means

required

fields started being enforced.

yeah, shouldn't have upgraded all at once, but I did look at changelogs... one moment

Both taps are now failing on missing required properties and it looks like they started validating the data against the JSON Schema provided by the REST tap, but meltano is configured to only pass a subset of properties

What's the exact validation error?

It was coming from json schema validator, schema was saying a property was required, but the property wasn't selected in meltano. Should schemas have always all properties non-required, or some component is responsible for applying selections to the schema?

I think that's a bug: https://github.com/meltano/sdk/issues/3062

that explains it, thank you