Hey everyone, I finally got Meltano up and running on AWS, yay! I was curious about logging, what do you guys usually use? I was thinking about ELK but I was wondering if there is an preferred logging system that the community uses.
a
aaron_phethean
05/26/2023, 1:45 PM
We use EL for client pipeline log collection in our platform (not K as our app displays the logs). Nothing challenging about the log collection - to do with Meltano anyway.
Here's an example: https://github.com/Matatika/matatika-ce/blob/main/docker-compose.yml Might be helpful because certificate creation and authentication to elasticsearch is a pain.
We use Datadog for our operational log collection. We chose to do this collection separately with EL so that our team can't see the client logs and it's portable.