Having a strange issue with the

tap-postgres

or general Meltano configuration I've misinterpreted. I am passing a list of fields to select for a given stream via environment variables, and I see the fields that I want are flagged as

selected: true

in the catalog when dumped. However the sync fails due to a JSON Schema violation, which has flagged a group of other fields as

required

even though they were not explicitly selected. Is this expected behaviour and I need to explicitly deselect these fields, or could this be an issue with the JSON schema being generated including fields that were not selected?

It's ugly, but if you manually edit the catalog file to patch the contents of the

required

array, you can add

catalog

to your extractor pointing to the file:

plugins:
  extractors:
  - name: my-tap
    catalog: ./path/to/my/catalog.json

Oof, can't do this nicely with an environment variable? Right now pretty much all of our config is expressed via environment variables so I've been avoiding bespoke catalog edits

Unfortunately no, other than

MY_TAP__CATALOG=./path/to/my/catalog.json

🫤 https://github.com/meltano/sdk/issues/2224

Ok thanks for the detailed explanation, will see if I can switch to using catalogs, I already dump it to stdout as part of my job so I'll just need to introduce patching that file into the pipeline

I got a PR that seems to do the right thing: https://github.com/meltano/sdk/pull/2225. I got the following catalog changes for this sqlite table:

CREATE TABLE subscriptions (
	id CHAR(32) NOT NULL, 
	recipient VARCHAR(255) NOT NULL, 
	event_type VARCHAR(255) NOT NULL, 
	source_type VARCHAR(255), 
	source_id VARCHAR(255), 
	created_at DATETIME, 
	PRIMARY KEY (id), 
	UNIQUE (recipient, event_type, source_type, source_id)
);

Hi @Edgar Ramírez (Arch.dev) really appreciate you looking into this, looks like the exact change I'd be looking for!