what's the difference between plugin settings and ...
# announcements
t
what's the difference between plugin settings and config? https://meltano.com/tutorials/postgres-with-postgres.html#adding-a-custom-extractor has the following plugin config/settings:
Copy code
plugins:
  extractors:
  - capabilities:
    - discover
    - properties
    - state
    executable: tap-postgres
    name: tap-postgres
    namespace: tap_postgres
    pip_url: tap-postgres==0.0.61
    settings:
      - name: dbname
        env: TAP_PG_DATABASE
      - name: host
        env: TAP_PG_ADDRESS
      - name: password
        env: TAP_PG_PASSWORD
      - name: port
        env: TAP_PG_PORT
      - name: user
        env: TAP_PG_USERNAME
    config:
      default_replication_method: FULL_TABLE
      include_schemas_in_destination_stream_name: true
  loaders:
 ... ... ...
but I can't see why some things are in settings and some in config. Looking at the source (https://github.com/singer-io/tap-postgres/blob/master/tap_postgres/__init__.py#L677-L702) the tap seems to have no difference and pulls them all from config. The only thing I can think of is that
settings
is an awkward way to add things so that you can specify that it should be set from an environment variable because it would be tricky to do that with the
config
style key/value pairs but that is nicer if you want to just hardcode things in the meltano.yml file but not sure if I'm missing something there?
d
settings
defines metadata for each setting (config property) a plugin supports: https://meltano.com/docs/contributor-guide.html#setting-definitions For discoverable plugins (that are supported out of the box and can be added using
meltano add <type> <name>
), that metadata is collected in `discovery.yml`: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/bundle/discovery.yml#L180 On custom plugin definitions that Meltano didn't know yet, this plugin metadata can be found in your own `meltano.yml`: https://meltano.com/docs/project.html#custom-plugin-definitions The actual current configuration for a plugin in your project can always be found under
config
in
meltano.yml
: https://meltano.com/docs/project.html#plugin-configuration Technically,
settings
are not required on custom plugin definitions in
meltano.yml
because all values under
config
are passed to the plugin anyway as "custom settings": https://meltano.com/docs/configuration.html#custom-settings However, if you don't want to store a setting value in
meltano.yml
, because it may be sensitive or environment-specific (like a token or password), you'd want to use an env var in
.env
, which requires Meltano to know about the setting so that its env var shows up in
meltano config <plugin> list
.
Does that make sense? There are a few concepts at play here (discoverable plugins, custom plugins, setting definitions, custom config), so if you have a suggestion on how this could be documented more clearly, I'll gladly hear it 🙂 On a custom plugin,
settings
feels a little superfluous when you can just add keys and values to
config
, but like I mentioned, they do have a use in that you can provide additional metadata (which is used in Meltano UI) and use env vars
In the specific
tap-postgres
example, the reason for using
settings
vs
config
isn't quite obvious, but the idea is that the user would themselves provide values for the 5 settings using env vars (as seen further down in the doc), while the other config have hard-coded values
t
okay thanks, I think that's roughly what I'd figured out but wasn't 100% sure