Is it possible to run a subset of an extractor's selected entities? I want to perform some one off tasks (i.e. run a full refresh for a single table).

you could pass the selection array as an env var for that one off run https://docs.meltano.com/concepts/plugins/#select-extra

does that env var work with the

run

command?

Yes, if you use it as

<TAP_NAME>__SELECT

ah this trick doesn't seem to work for inherited plugins

Might need to use the appropriate prefix, you can check the expected env var name with

meltano config <plugin name> list --extras

https://github.com/meltano/meltano/issues/9239 in case this is relevant here...

Thanks Edgar this has been helpful. To share a bit more, I'm running meltano 3.8.0. My extractor is inherits from the Postgres tap to only load from the public schema and is called

tap-postgres--public

. Running

meltano config tap-postgres--public list --extas

does show the

TAP_POSTGRES__PUBLIC__SELECT

env var does update the config. Running

meltano select tap-postgres--public --list

shows only the single table I included in the

__SELECT

env var. However

meltano run

still raises a

StreamMapConfigError

due to

Invalid key properties for 'public_my-other-table': [my_other_table_key,database_id]. Property 'my_other_table_key' was not detected in schema.

for a table that is not selected. I tried also setting the

TAP_POSTGRES__PUBLIC__METADATA

env var to just have the single table I selected, however the error persists. Next I thought I'd try setting

TAP_POSTGRES__PUBLIC__USE_CACHED_CATALOG=false

but that also did not help and the same error persists.

> However meltano run still raises a

StreamMapConfigError

due to

Invalid key properties for 'public_my-other-table': [my_other_table_key,database_id]. Property 'my_other_table_key' was not detected in schema.

for a table that is not selected. I wonder if this is a result of https://github.com/meltano/sdk/issues/1942. I did open a draft PR with the intention of fixing, but haven't had the time to get back to it. @Ellis Valentiner The original issue I posted is more to do with being able to configure an inherited plugin with the same environment variables you would for the base/parent plugin. So in your case, I guess the equivalent would be

TAP_POSTGRES__SELECT='[...]' meltano run tap-postgres--public ...

which I'm pretty sure you aren't doing.

meltano config <plugin> list

shows you where it is resolving values from: