Hey everyone, I was wondering if anyone had ever p...
# plugins-generals
Stéphane Burwash
07/19/2022, 6:55 PMHey everyone, I was wondering if anyone had ever parsed meltano/dbt/airflow logs using grafana and had a regex template for me? I'm trying to gain total observability on my stack
d
devon_seitz
07/19/2022, 7:47 PMthis is the yaml i set up
Copy code
version: 1
disable_existing_loggers: false
formatters:
default:
format: "[%(asctime)s] [%(process)d|%(threadName)10s|%(name)s] [%(levelname)s] %(message)s"
structured_plain:
(): meltano.core.logging.console_log_formatter
colors: False
structured_colored:
(): meltano.core.logging.console_log_formatter
colors: True
key_value:
(): meltano.core.logging.key_value_formatter
sort_keys: False
json:
(): meltano.core.logging.json_formatter
handlers:
console:
class: logging.StreamHandler
level: INFO
formatter: structured_plain
stream: "<ext://sys.stderr>"
root:
level: INFO
propagate: yes
handlers: [console]s
Stéphane Burwash
07/19/2022, 7:55 PMThanks! I'm a noob in log parsing and grafana; where would I apply this?
f
florian.hines
07/19/2022, 9:37 PMIf you're using Loki from the grafana folks - you could probably just configure meltano to log in json format and get pretty far using their json parser (https://grafana.com/blog/2020/10/28/loki-2.0-released-transform-logs-as-youre-querying-them-and-set-up-alerts-within-loki/#parse)
florian.hines
07/19/2022, 9:39 PMjust realized thats a possibly pretty ancient blog post (sorry, would mostly still apply though I'd hope)
s
Stéphane Burwash
07/19/2022, 9:47 PM@florian.hines do you know how to configure the log output?
f
florian.hines
07/19/2022, 9:49 PMThe yaml that @devon_seitz shared is actually a sample meltano logging.yaml that lets you control logging. Theres some more details at https://docs.meltano.com/reference/settings#how-to-use-10
florian.hines
07/19/2022, 9:50 PMIt some what depends on how your shipping log data to Loki (not sure if you're tailing a log file or the like). But you'll either need to switch the output format to json for the log file, or the console handler (if capturing console output).
s
Stéphane Burwash
07/19/2022, 9:51 PMAwesome, thank you so much!
f
florian.hines
07/19/2022, 9:51 PM👍 if you steal the example from the meltano docs. The "formatter" option in the "handlers" section is the important bit.