On a related note but separate topic, what's best practice for handling logging and exceptions? I know that logs are written to a folder, but if the community has standardised on a way to flag errors and fire off an alert (even hitting a webhook to send an email), that would be ideal

@sam_woolerton If the tap or target exits with a nonzero exit code, so will

meltano elt

. The orchestrator you're using to run

meltano elt

(e.g. Airflow) will detect that failure, and will provide a mechanism for you to setup a webhook or other type of notification. Handling error notifications is out of scope for Meltano, which helps you setup and run an orchestrator (and we could do more to help set up error notifications), but isn't itself one, and doesn't replace it.

That's fair enough, thanks. I'd been using Windows Task Scheduler (running linux with

wsl

), so may need to swap that out with something more robust

@sam_woolerton I'd suggest looking at Airflow, which is very popular and is supported by Meltano out of the box: https://meltano.com/docs/orchestration.html, but there's also Prefect.io which has some fans in this Slack workspace 🙂