Just throwing this out here for discussion — I’m w...
# getting-startedd
drew_ipson
11/12/2021, 6:12 PMJust throwing this out here for discussion — I’m wondering if anyone has any pointers or recs on monitoring Meltano in production and getting alerts if an elt process fails. We currently run this in a docker image inside a k8s pod on our cluster via cronjob.
e
edgar_ramirez_mondragon
11/12/2021, 8:20 PMhey @drew_ipson! we (I mean @florian.hines 100% 😅) are working on https://gitlab.com/meltano/meltano/-/merge_requests/2395 and https://gitlab.com/meltano/meltano/-/issues/3008. The first one will help you for example with sending logs to datadog that it can parse nicely if you install the dd agent in your k8s cluster. The second one would make it easier to ship logs to that and other destinations. Alerts can then be set for error-level logs
s
stephen_bailey
11/13/2021, 2:00 AMWe use our orchestrator (Argo Workflows), which has Slack tie-ins, so we can alert on meltano failures just like any job on our platform
f
fred_reimer
11/16/2021, 1:51 AMI have it running with Airflow under k8s, and with Datadog integration. So multiple ways of alerting there...
d
drew_ipson
11/23/2021, 5:04 PM@edgar_ramirez_mondragon Awesome! Thank you for the link to the update. Is datadog the recommended logging tool? We’re currently using LogStash and OpenSearch on AWS and we’re having a hard time getting the logs back to OpenSearch.