The meltano logging documentation includes some examples of “custom” / non-standard loggers but they both just use standard formatters with no additional dependencies. Is it possible to provide meltano a custom log handler that uses its own additional Python dependency? (And if so, is this something that would be done by just adding an extra dependency to the virtual environment Meltano uses, or is this something that could be packaged as a Meltano “utility” plugin?). (Context: I am trying to update the dagster-meltano library to support Dagster 1.10, more details here.)

Hey @Matt Menzenski 👋 You can use any custom logging handlers and formatters as long as they implement the Python stdlib logging interfaces. For example, I've used the fluent-logger-python library like this:

version: 1
formatters:
  brief:
    format: "%(message)s"
  default:
    format: "%(asctime)s %(levelname)-8s %(name)-15s %(message)s"
    datefmt: "%Y-%m-%d %H:%M:%S"
  fluent_fmt:
    "()": fluent.handler.FluentRecordFormatter
handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: default
    stream: <ext://sys.stderr>
  fluent:
    class: fluent.handler.FluentHandler
    host: localhost
    port: 24224
    tag: test.logging
    buffer_overflow_handler: overflow_handler
    formatter: fluent_fmt
    level: DEBUG
  none:
    class: logging.NullHandler
root:
  handlers: [console, fluent]
  level: DEBUG
loggers:
  # Disable logging of tap and target stdout
  meltano.core.block.extract_load:
    # handlers: [file]
    level: INFO
  # Disable catalog processing debug logs
  meltano.core.plugin.singer.catalog:
    level: INFO

The third-party logging library does have to be installed in the same venv as Meltano, or you could do some PYTHONPATH dark magic.

awesome, thanks @Edgar Ramírez (Arch.dev)! that example seems very similar to what I am after