tom_elliff
10/08/2020, 11:35 PMplugins:
extractors:
- capabilities:
- discover
- properties
- state
executable: tap-postgres
name: tap-postgres
namespace: tap_postgres
pip_url: tap-postgres==0.0.61
settings:
- name: dbname
env: TAP_PG_DATABASE
- name: host
env: TAP_PG_ADDRESS
- name: password
env: TAP_PG_PASSWORD
- name: port
env: TAP_PG_PORT
- name: user
env: TAP_PG_USERNAME
config:
default_replication_method: FULL_TABLE
include_schemas_in_destination_stream_name: true
loaders:
... ... ...
but I can't see why some things are in settings and some in config. Looking at the source (https://github.com/singer-io/tap-postgres/blob/master/tap_postgres/__init__.py#L677-L702) the tap seems to have no difference and pulls them all from config.
The only thing I can think of is that settings is an awkward way to add things so that you can specify that it should be set from an environment variable because it would be tricky to do that with the config style key/value pairs but that is nicer if you want to just hardcode things in the meltano.yml file but not sure if I'm missing something there?douwe_maan
10/08/2020, 11:47 PMsettings defines metadata for each setting (config property) a plugin supports: https://meltano.com/docs/contributor-guide.html#setting-definitions
For discoverable plugins (that are supported out of the box and can be added using meltano add <type> <name>), that metadata is collected in `discovery.yml`: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/bundle/discovery.yml#L180
On custom plugin definitions that Meltano didn't know yet, this plugin metadata can be found in your own `meltano.yml`: https://meltano.com/docs/project.html#custom-plugin-definitions
The actual current configuration for a plugin in your project can always be found under config in meltano.yml : https://meltano.com/docs/project.html#plugin-configuration
Technically, settings are not required on custom plugin definitions in meltano.yml because all values under config are passed to the plugin anyway as "custom settings": https://meltano.com/docs/configuration.html#custom-settings
However, if you don't want to store a setting value in meltano.yml , because it may be sensitive or environment-specific (like a token or password), you'd want to use an env var in .env, which requires Meltano to know about the setting so that its env var shows up in meltano config <plugin> list .douwe_maan
10/08/2020, 11:49 PMsettings feels a little superfluous when you can just add keys and values to config , but like I mentioned, they do have a use in that you can provide additional metadata (which is used in Meltano UI) and use env varsdouwe_maan
10/08/2020, 11:50 PMtap-postgres example, the reason for using settings vs config isn't quite obvious, but the idea is that the user would themselves provide values for the 5 settings using env vars (as seen further down in the doc), while the other config have hard-coded valuestom_elliff
10/09/2020, 12:50 AM