Challenge #1 - Detecting and Registering Tap/Target Setting Keys The hardest part to keep on top of in terms of initial onboarding, as well as ongoing maintenance, seems to be the identification of each plugin's settings. This is compounded by the fact that taps and targets continually add new setting options, and also that many taps (for instance, tap-sftp, tap-dynamodb, and tap-mssql) don't have any documentation at all about what settings are expected. Option 1A: The impactical-but-ideal solution would be that we push something machine readable (similar to discovery.yml but only one tap/target per doc) to each plugin's github repo. Then the metadata becomes scannable and it's supported in a decentralized manner by the owners of each repo. Option 1B: Given that 1A is not practical (at least not immediately so), I wonder if the Meltano platform would tolerate/support some ambiguity and flexibility into users specifying "other settings" - being not-yet-centrally-exposed-settings. Easy example: I'm using a tap and that tap adds "use_ssl" as an optional config. Can I specify that optional config even if it's not in

discovery.yml

? Option 1C: And this is in addition to Option 1B - Can Meltano, through platform usage reporting, allow users to opt-in to send back what setting combinations they are running with each plugin? Obviously, we would send back just the name of the settings used and not their values. But essentially, as users debug what combination of settings work for a given tap or target, they are helping improve the platform by providing stats back to us on what setting values are supported and usable for each tap and target.

Option 1A: The impactical-but-ideal solution would be that we push something machine readable (similar to discovery.yml but only one tap/target per doc) to each plugin's github repo. Then the metadata becomes scannable and it's supported in a decentralized manner by the owners of each repo.

This would be cool, but I think the first step would be to get to a place where all taps and targets actually clearly document their settings in the README. That's something that can be easily contributed to existing taps as well, doesn't require maintainer buy-in into this new (Meltano-specific?) metadata format, and benefits all users of these Singer taps, not just those using smart tools that can detect metadata. If taps document their settings, they'll be easy to copy over into Meltano's plugin definition, and as taps add new settings that Meltano users will want to use, these users can contribute these back to

discovery.yml

.

Option 1B: Given that 1A is not practical (at least not immediately so), I wonder if the Meltano platform would tolerate/support some ambiguity and flexibility into users specifying "other settings" - being not-yet-centrally-exposed-settings. Easy example: I'm using a tap and that tap adds "use_ssl" as an optional config. Can I specify that optional config even if it's not in 

discovery.yml

?

Meltano actually already has full support for custom settings, although I'm realizing now that this isn't documented explicitly. (Issue created: https://gitlab.com/meltano/meltano/-/issues/2201) You can try it for yourself, though: run

meltano config tap-example set use_ssl true

(which sets the setting in

meltano.yml

), and then notice it show up in

meltano config tap-example

and

meltano config tap-example list

.

Option 1C: And this is in addition to Option 1B - Can Meltano, through platform usage reporting, allow users to opt-in to send back what setting combinations they are running with each plugin? Obviously, we would send back just the name of the settings used and not their values. But essentially, as users debug what combination of settings work for a given tap or target, they are helping improve the platform by providing stats back to us on what setting values are supported and usable for each tap and target.

Presumably, if a user figures out what combinations of settings work for a given tap or target, they will have found out either through reading the readme or diving into the source code to figure out how it's using

config.json

and what keys it expects to find there. At that point, they can directly contribute those findings back to the tap README and Meltano's

discovery.yml

, along with the appropriate settings metadata. While that process could be easier (not everyone is comfortable contributing on GitLab, for example), I don't think usage reporting is the answer, since that moves the responsibility of gathering the data and figuring out the metadata to us Meltano maintainers, instead of the actual user who got it working and would know first-hand what that metadata and updated docs should look like.

On 1A, I would have originally agreed that yes, it's probably best to take the first step of just enhancing/creating better readmes across taps/targets. However, I've grown somewhat skeptical to the prospect of scaling this approach, in terms of it's cost in human-hours, but also, more to your point, the sole focus in the community on human-readable READMEs (ironically) becomes a blocker to those who would prefer to use user-friendly UIs and orchestrators - such as Meltano. In addition to the resource cost, I also think there's some DRY (don't repeat yourself) psychology/intuition preventing repo owners from creating machine-readable metadata when they're already maintaining the same thing in the READMEs. Which, again somewhat ironically, then forces orchestrator tools to violate DRY by duplicating the settings metadata in our own separate repos. 🤷‍♂️ 🤔

@aaronsteers I definitely like the idea of having each tap and target define its own settings and other metadata in its repo, but I think Meltano will need to be bit more mature and widely adopted among Singer users for that to actually be something we can get tap and target authors on board with. A generic, non-Meltano-specific Singer metadata format (whether based on YAML or Markdown tables) may be more appropriate at this time (and in general), but then Meltano's own plugin definition would still need to have additional metadata on top of that, so you still wouldn't get away from some kind of centralized database. As far as combining `discovery.yml`s is concerned, this is not currently possible (local just takes precedence over remote, without any merging), but I think a first step would be to allow individual plugins to be defined in individual YAML files (named

discovery/<type>/<name>.yml

, e.g.

discovery/extractors/tap-gitlab.yml

), that could then be compiled into the greater

discovery.yml

that's for download at https://meltano.com/discovery.yml. That would make it easier for users to contribute new known taps and targets as well, since they wouldn't need to modify the one massive file. Once that's supported, it wouldn't be much of a stretch to also compile remote plugin-specific definition YMLs into that file.

@douwe_maan - Yes, agreed 👍 💯 % I would be happy to move this conversation into one or more tickets. What do you think about one for something like "aggregate discovery.yml from individual plugin files" - and mentioning the spec and proposed folder structure you described above? And perhaps another to create a parser which could automatically generate/refresh the text for an individual plugin based on a README file as input?

Sounds good on both counts! I'm a little skeptical that such a README parser would actually work well enough for enough taps and targets to make it worth the effort, but I guess we'll find out 😄

Indeed! 🤣 And thank you!

If you pick up that "aggregate discovery.yml from individual plugin files" issue, I'll gladly help you figure out the details, of course

Logged: https://gitlab.com/meltano/meltano/issues/2206

Awesome, thanks!

Awesome - thanks. And for reference here is the one for auto-parsing; https://gitlab.com/meltano/meltano/issues/2207