How do I get “fully JSON” Meltano logs? I have the following log config in

logging.yaml

(taken mostly from the docs). I also have

MELTANO_CLI_LOG_CONFIG=logging.yaml

set in my job’s environment variables.

version: 1
disable_existing_loggers: false
formatters:
  json:
    (): meltano.core.logging.json_formatter
handlers:
  console:
    class: logging.StreamHandler
    formatter: json
    stream: <ext://sys.stdout>
root:
  level: INFO
  propagate: 'yes'
  handlers:
    - console
loggers:
  tap_mongodb:
    level: DEBUG
    propagate: false
    handlers:
      - console
  snowflake.connector.cursor:
    level: WARNING
    propagate: false
    handlers:
      - console

I’m getting log messages that look like this (I have un-minified this):

{
  "consumer": true,
  "producer": false,
  "string_id": "target-snowflake-my-stream-name",
  "cmd_type": "elb",
  "stdio": "stderr",
  "name": "target-snowflake-my-stream-name",
  "event": "2024-03-21 12:37:12,535 | INFO     | snowflake.connector.cursor | query execution done",
  "level": "info",
  "timestamp": "2024-03-21T12:37:12.538133Z"
}

Note how

event

there is a formatted string. I want to have those fields as a structured object instead. Something like this would be ideal:

{
  "consumer": true,
  "producer": false,
  "string_id": "target-snowflake-my-stream-name",
  "cmd_type": "elb",
  "stdio": "stderr",
  "name": "target-snowflake-my-stream-name",
  "event": {
    "timestamp": "2024-03-21 12:37:12,535",
    "level": "INFO",
    "logger": "snowflake.connector.cursor",
    "message": "query execution done"
  },
  "level": "info",
  "timestamp": "2024-03-21T12:37:12.538133Z"
}

Is that possible to do via logging configuration?

You need to look at target-snowflake's implementation and have it also output structured logs if that's what you're after. Most folks do parsing in their log analyzer tool instead (for every tool I've seen) , but there's no reason taps and targets couldn't output structured logging

Also see: • https://sdk.meltano.com/en/latest/implementation/logging.html#custom-logging-configuration • https://github.com/edgarrmondragon/meltano-dogfood/blob/d69c4abcfdbb30ec11f174969912534656f6fc59/logging/singer_logging.yaml#L6-L9 • https://github.com/edgarrmondragon/meltano-dogfood/blob/d69c4abcfdbb30ec11f174969912534656f6fc59/meltano.yml#L25 (I really need to turn those examples into short guides in the docs 😅 )