Hi everyone 🙂 First of all, thanks for all the work on Meltano and Singer SDK. I really like the tools so far. I created a custom Extractor (Singer tap), which I would now like to use in a meltano EL pipeline. The custom extractor is very use-case specific, so it is not very suitable for being published to the meltano hub (Rest API extractor which supports rather exotic API with an unusual paging logic). I would like to install the custom extractor from within the same project structure, but I am not able to have the custom extractor settings populated automatically upon

meltano add --custom extractor xyz

(sourced using

pip -e ../xyz

). I would like to have both, custom extractor, and the meltano project using this extractor live in the same repository. Is there any way to add the custom extractor with all the settings options being populated automatically instead of specifying them all manually in the interactive prompt? (I tried consulting docs of both Meltano and SingerSDK, however, with no success 😞 I also tried searching in existing GitHub Issues for both project, but didn't manage to find anything relevant/helpful)

Not released yet, but the next minor version of Meltano v3 should support

--from-ref

(see this docs section and this PR). You could then maintain a plugin definition YAML file in your tap repository and

meltano add

from that. There was some discussion about the definition living in the repo or parsing it from the output of

--about

here and here.

Awesome. I noticed the mention of

--from-ref

in documentation (but I couldn't use it locally). Based on what is mentioned on the PR thread, it could alleviate some of the maintenance overhead for me. Whether by maintaining some yaml directly, or even better, parsing it out from

--about

as you mentioned. I will keep an eye on the release notes 🙂 For now, I will try some yaml templating or manual maintenance of EL project

meltano.yaml

file. Thank you :)