Hey, I have a question regarding inheritance. Sometimes on inheritance meltano skips installing a plugin, but sometimes it installs the inherited plugin. How is that determined ? also, the output from the install command doesn't seem to align with what I have in the .meltano/extractors folder, see details bellow. This is my meltano.yaml

plugins:
  extractors:
  - name: tap-oracle
    variant: s7clarke10
    pip_url:
      git+<https://github.com/s7clarke10/pipelinewise-tap-oracle.git@53bb75ed27d7796d2f492e74cec87f10f1bce4d4>
    config:
      use_ora_rowscn: false
      ora_python_driver_type: thick
      use_singer_decimal: true

  - name: tap-oracle-demo_foo
    inherit_from: tap-oracle
    config:
      user: demo
      host: 123
      port: 420
      service_name: foo

  - name: tap-oracle-demo_foo-full_table
    inherit_from: tap-oracle-demo_foo
    config:
      default_replication_method: FULL_TABLE

  - name: tap-oracle-demo_foo-full_table-bar_baz
    inherit_from: tap-oracle-demo_foo-full_table
    select:
     - bar-baz.*

  - name: tap-oracle-demo_foo-full_table-asd_qwe
    inherit_from: tap-oracle-demo_foo-full_table
    select:
      - asd-qwe.*

when running

meltano install

I get the following output.

2024-09-16T09:45:39.392382Z [info     ] Installing 8 plugins          
2024-09-16T09:45:39.392942Z [info     ] Skipped installing extractor 'tap-oracle-demo_foo'
2024-09-16T09:45:39.393006Z [info     ] Skipped installing extractor 'tap-oracle-demo_foo-full_table-asd_qwe'
2024-09-16T09:45:39.393708Z [info     ] Installing extractor 'tap-testsource'
2024-09-16T09:45:39.408495Z [info     ] Installing extractor 'tap-oracle'
2024-09-16T09:45:39.454512Z [info     ] Installing extractor 'tap-oracle-demo_foo-full_table'
2024-09-16T09:45:39.587498Z [info     ] Installing extractor 'tap-oracle-demo_foo-full_table-bar_baz'
2024-09-16T09:45:39.763693Z [info     ] Installing loader 'target-jsonl'
2024-09-16T09:45:39.776647Z [info     ] Installing loader 'target-bigquery'
2024-09-16T09:45:43.563063Z [info     ] Installed loader 'target-jsonl'
2024-09-16T09:45:46.557149Z [info     ] Installed extractor 'tap-oracle'
2024-09-16T09:45:46.708771Z [info     ] Installed extractor 'tap-oracle-demo_foo-full_table'
2024-09-16T09:45:46.938197Z [info     ] Installed extractor 'tap-oracle-demo_foo-full_table-bar_baz'
2024-09-16T09:45:48.596501Z [info     ] Installed extractor 'tap-testsource'
2024-09-16T09:45:55.892412Z [info     ] Installed loader 'target-bigquery'
2024-09-16T09:45:55.892868Z [info     ] Installed 6/8 plugins         
2024-09-16T09:45:55.892947Z [info     ] Skipped installing 2/8 plugins

it skips the

asd_qwe

but installs the

bar_baz

why? I assume here due to

Skipped installing extractor 'tap-oracle-demo_foo'

the that no venv is created for the

tap-oracle-demo_foo

but this is the result I get in the .meltano folder ( see attached screenshot).

it skips the

asd_qwe

but installs the

bar_baz

why?

I think both are actually skipped, because I see

Skipped installing 2/8 plugins

in there. Though that means

Installing extractor 'tap-oracle-demo_foo-full_table-bar_baz'

is misleading, so we might need to fix that.

I assume here due to

Skipped installing extractor 'tap-oracle-demo_foo'

the that no venv is created for the

tap-oracle-demo_foo

but this is the result I get in the .meltano folder ( see attached screenshot).

I think those directories actually have empty

venv

subdirs, can you confirm? We might still wanna confirm that no unnecessary directories are created, similar to what we addressed in https://github.com/meltano/meltano/pull/8447.

I've been playing with this a bit recently as part of an infrastructure deployment. It seems that

install

installs an environment to the parent. I have a structure like

tap-mssql -> tap-mssql-mydb -> tap-mssql-mydb-installer

. I run

meltano install extractor tap-mssql-mydb-installer

which then installs a tap-mssql environment to the

tap-mssql-mydb

folder. No folder/files are created for

tap-mssql

nor

tap-mssql-mydb-installer

. For multiple databases, I ended up with a tap-mssql environment in each

tap-mssql-mydbXX

. If I were to just do

meltano install tap-mssql-mydb

then the tap-mssql environment is installed to the

tap-mssql

folder, and nothing to the

tap-mssql-mydb

folder. Unfortunately, if I do

meltano run --no-install tap-mssql-mydb-workerX

, it thinks

tap-mssql-mydb

is not installed then, hence the need for the "installer", so it seems multiple levels of inheritance are disregarded here. It's good that there is some attempt to avoid unnecessary copies of packages, but there's definitely still some duplication, questionable behaviour, and unintuitive design here.

@Edgar Ramírez (Arch.dev) The directories all seem to contain venvs ( see screenshot listing size or each one of them) @Hayden Ness I did some experimentation and I see something similar, it creates a venv for every tap that has something inhereting from it. There might be some good reason for it, but I can't see it.

I've opened https://github.com/meltano/meltano/issues/8783 to track this and similar problems. Do comment there with specifics and reproducible examples if you can, that'd make debugging and fixing much easier.

There may be a one-line fix... 🤔 https://github.com/meltano/meltano/pull/8785 I've tried to include some detail.

About to make a release with these fixes

That's working good for me, no more duplication and time spent checking if my workers have been installed. Thank you very much Reuben and Edgar. A nice bonus is that I can just install it in to the docker image, and not risk ending up with different versions of the tap over time or between workers.