Hello, I'm using

tap-smoke-test

but the select config I've introduced into

meltano.yml

isn't working

select:
 - animals.id
 - animals.created_at

meltano run tap-smoke-test target-jsonl

results in error

2024-10-27T15:36:03.302968Z [info     ] jsonschema.exceptions.ValidationError: 'description' is a required property cmd_type=elb consumer=True job_name=dev:tap-smoke-test-to-target-jsonl name=target-jsonl producer=False run_id=c9b2f522-094d-4099-9f02-95adae006c52 stdio=stderr string_id=target-jsonl
2024-10-27T15:36:03.303551Z [info     ]                
                cmd_type=elb consumer=True job_name=dev:tap-smoke-test-to-target-jsonl name=target-jsonl producer=False run_id=c9b2f522-094d-4099-9f02-95adae006c52 stdio=stderr string_id=target-jsonl
2024-10-27T15:36:03.304254Z [info     ] Failed validating 'required' in schema: cmd_type=elb consumer=True job_name=dev:tap-smoke-test-to-target-jsonl name=target-jsonl producer=False run_id=c9b2f522-094d-4099-9f02-95adae006c52 stdio=stderr string_id=target-jsonl

When I dump the catalog file I can clearly see that this and other columns are required... but I have no clue how to change that I found https://docs.meltano.com/concepts/plugins#schema-extra but it doesn't specify that specific part of the

schema

key.

I think

tap-smoke-test

just doesn't work with

select

- it infers the schema from the input files, so if all properties are present for all records, it is going to expect them to all be required. https://github.com/meltano/tap-smoke-test?tab=readme-ov-file#schema-inference

I see So let's say I rely on schema inference for a different tap, say

tap-mysql

, would the same thing happen to any given table that I am trying to extract data from? For context, for my POC with meltano I need to be able to simply drop certain columns that have PII data.

Probably not for SQL taps as these will (usually) construct a stream schema from a table definition (i.e. columns that are nullable should map to properties that are not required). I guess the issue you have discovered here would generally only be a problem for taps that sample a subset of records to determine the schema.

Ok that's reassuring, thanks for your attention 🙂