https://linen.dev logo
Join Slack
Powered by
<#9873 feature: Show a sensible subset of settings...
# meltano-repo-activity
g
#9873 feature: Show a sensible subset of settings by default in meltano config list Issue created by mahangu Feature scope CLI (options, error messages, logging, etc.) Description 
meltano config list
shows every setting for a plugin regardless of whether the user has configured it. For plugins with many settings, this buries useful information in noise. Most users checking their config only care about what they've set and what they still need to set. Here is real output from 
meltano config list tap-postgres
after configuring just 4 settings (
host
, 
user
, 
database
, 
filter_schemas
). The user still sees all 40 settings: 
$ meltano config list tap-postgres
— full output (40 settings)
Copy code
batch_config.encoding.compression [env: TAP_POSTGRES_BATCH_CONFIG_ENCODING_COMPRESSION] current value: None (default)
	Batch Compression Format: Compression format to use for batch files.
batch_config.encoding.format [env: TAP_POSTGRES_BATCH_CONFIG_ENCODING_FORMAT] current value: None (default)
	Batch Encoding Format: Format to use for batch files.
batch_config.storage.prefix [env: TAP_POSTGRES_BATCH_CONFIG_STORAGE_PREFIX] current value: None (default)
	Batch Storage Prefix: Prefix to use when writing batch files.
batch_config.storage.root [env: TAP_POSTGRES_BATCH_CONFIG_STORAGE_ROOT] current value: None (default)
	Batch Storage Root: Root path to use when writing batch files.
database [env: TAP_POSTGRES_DATABASE] current value: 'mydb' (from `meltano.yml`)
	Database: Database name. Note if sqlalchemy_url is set this will be ignored.
dates_as_string [env: TAP_POSTGRES_DATES_AS_STRING] current value: False (default)
	Dates As String: Defaults to false, if true, date, and timestamp fields will be Strings.
default_replication_method [env: TAP_POSTGRES_DEFAULT_REPLICATION_METHOD] current value: 'FULL_TABLE' (default)
	Default Replication Method: Replication method to use if there is not a catalog entry to override this choice.
faker_config.locale [env: TAP_POSTGRES_FAKER_CONFIG_LOCALE] current value: None (default)
faker_config.seed [env: TAP_POSTGRES_FAKER_CONFIG_SEED] current value: None (default)
filter_schemas [env: TAP_POSTGRES_FILTER_SCHEMAS] current value: ['public'] (from `meltano.yml`)
	Filter Schemas: If an array of schema names is provided, the tap will only process the specified Postgres schemas.
flattening_enabled [env: TAP_POSTGRES_FLATTENING_ENABLED] current value: None (default)
flattening_max_depth [env: TAP_POSTGRES_FLATTENING_MAX_DEPTH] current value: None (default)
flattening_max_key_length [env: TAP_POSTGRES_FLATTENING_MAX_KEY_LENGTH] current value: None (default)
host [env: TAP_POSTGRES_HOST] current value: 'localhost' (from `meltano.yml`)
	Host: Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored.
json_as_object [env: TAP_POSTGRES_JSON_AS_OBJECT] current value: False (default)
max_record_count [env: TAP_POSTGRES_MAX_RECORD_COUNT] current value: None (default)
password [env: TAP_POSTGRES_PASSWORD] current value: None (default)
	Password: Password used to authenticate. Note if sqlalchemy_url is set this will be ignored.
port [env: TAP_POSTGRES_PORT] current value: 5432 (default)
	Port: The port on which postgres is awaiting connection.
replication_slot_name [env: TAP_POSTGRES_REPLICATION_SLOT_NAME] current value: 'tappostgres' (default)
sqlalchemy_url [env: TAP_POSTGRES_SQLALCHEMY_URL] current value: None (default)
	SQLAlchemy URL: Example postgresql://[username]:[password]@localhost:5432/[db_name]
ssh_tunnel.enable [env: TAP_POSTGRES_SSH_TUNNEL_ENABLE] current value: False (default)
ssh_tunnel.host [env: TAP_POSTGRES_SSH_TUNNEL_HOST] current value: None (default)
ssh_tunnel.port [env: TAP_POSTGRES_SSH_TUNNEL_PORT] current value: 22 (default)
ssh_tunnel.username [env: TAP_POSTGRES_SSH_TUNNEL_USERNAME] current value: None (default)
ssh_tunnel.private_key [env: TAP_POSTGRES_SSH_TUNNEL_PRIVATE_KEY] current value: None (default)
ssh_tunnel.private_key_password [env: TAP_POSTGRES_SSH_TUNNEL_PRIVATE_KEY_PASSWORD] current value: None (default)
ssl_certificate_authority [env: TAP_POSTGRES_SSL_CERTIFICATE_AUTHORITY] current value: '~/.postgresql/root.crl' (default)
ssl_client_certificate [env: TAP_POSTGRES_SSL_CLIENT_CERTIFICATE] current value: '~/.postgresql/postgresql.crt' (default)
ssl_client_certificate_enable [env: TAP_POSTGRES_SSL_CLIENT_CERTIFICATE_ENABLE] current value: False (default)
ssl_client_private_key [env: TAP_POSTGRES_SSL_CLIENT_PRIVATE_KEY] current value: (redacted) (default)
ssl_enable [env: TAP_POSTGRES_SSL_ENABLE] current value: False (default)
ssl_mode [env: TAP_POSTGRES_SSL_MODE] current value: 'verify-full' (default)
ssl_storage_directory [env: TAP_POSTGRES_SSL_STORAGE_DIRECTORY] current value: '.secrets' (default)
stream_map_config [env: TAP_POSTGRES_STREAM_MAP_CONFIG] current value: None (default)
stream_maps [env: TAP_POSTGRES_STREAM_MAPS] current value: None (default)
stream_options [env: TAP_POSTGRES_STREAM_OPTIONS] current value: None (default)
use_singer_decimal [env: TAP_POSTGRES_USE_SINGER_DECIMAL] current value: None (default)
user [env: TAP_POSTGRES_USER] current value: 'postgres' (from `meltano.yml`)
	User: User name used to authenticate. Note if sqlalchemy_url is set this will be ignored.
By default, 
meltano config list
should only show required settings (from 
settings_group_validation
) and settings the user has explicitly configured (source is not 
DEFAULT
). An 
--all
flag would preserve the current behavior for users who need the full list. Example of proposed default output for the same tap-postgres configuration above:
Copy code
$ meltano config list tap-postgres

database [env: TAP_POSTGRES_DATABASE] current value: 'mydb' (from `meltano.yml`)
	Database: Database name. Note if sqlalchemy_url is set this will be ignored.
filter_schemas [env: TAP_POSTGRES_FILTER_SCHEMAS] current value: ['public'] (from `meltano.yml`)
	Filter Schemas: If an array of schema names is provided, the tap will only process the specified Postgres schemas.
host [env: TAP_POSTGRES_HOST] current value: 'localhost' (from `meltano.yml`)
	Host: Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored.
user [env: TAP_POSTGRES_USER] current value: 'postgres' (from `meltano.yml`)
	User: User name used to authenticate. Note if sqlalchemy_url is set this will be ignored.

36 optional settings at defaults are hidden. Use --all to show all settings.
And 
--all
would produce the current (full) output. Some considerations: • This changes default behavior. Scripts that parse 
meltano config list
output and tests that assert on specific output lines would need updates. • The 
--extras
flag should compose naturally with 
--all
— they filter on orthogonal axes (setting type vs. configuration status). • The 
--all
flag pattern is already used in other Meltano CLI commands (
meltano state
, 
meltano validate
, 
meltano select
). • Related: #6422 (visibility of setting values across all stores). Possible addition: 
--filter
option A 
--filter
option with case-insensitive substring matching on setting names could also help when debugging a specific group of related settings. For example, a user debugging SSL configuration currently has to scroll past 30+ unrelated settings to find the 7 
ssl_*
ones: ``` $ meltano config list tap-postgres --filter ssl ssl_certificate_authority [env: TAP_POSTGRES_SSL_CERTIFICATE_AUTHORITY] current value: '~/.postgresql/root.crl' (default) SSL Certificate Authority: The certificate authority that should be used to verify the server's identity. ssl_client_certificate [env: TAP_POSTGRES_SSL_CLIENT_CERTIFICATE] current value: '~/.postgresql/postgresql.crt' (default) SSL Client Certificate: The certificate that should be used to verify your identity to the server. ssl_client_certificate_enable [env: TAP_POSTGRES_SSL_CLIENT_CERTIFICATE_ENABLE] current value: False (default) SSL Client Certificate Enable: Whether or not to provide client-side certificates as a method of authentication to the ser… meltano/meltano
Open in Slack
PreviousNext
Powered by