Sharing here since there are more people in this channel 🙂

went down the rabbit hole on this, just peered back out and decided not to post this as my initial assumption was naive here. Much harder, simpler is better

I'll take a stab!

Seperate but related comment I can't help but think about the general architecture of logging with multiples procs, is there a writeup about this for Meltano and the general approach? I did something similar years back (not singer) with redirecting stdout / stderr and piping them around the issue is that technically your OS and Runtime (Python in this cases Java in my last case) can technically do whatever the hell it wants with stdout / stderr. So whatever guard rails are put up to make stdout and stderr behave "correctly" there's a long Tail of small "known unknowns" that we as developers try to say are "unknown unknowns" :D that pop up. <https://gitlab.com/meltano/meltano/-/issues/2881> is a good example of Click doing something special with `echo`. Dealing with this so that 100% of the time stdout / stderr get redirected to the right place is good to strive for, and there has to be some level of assumption that something you don't control is going to spew data to stdout / stderr (I vaguely remember something deep in a JRE doing this, I don't know about Python deeply enough, but you're most likely to hit this in System Panic state with your Runtime / OS and even more likely is one of the many libraries being used with have some edge cases that will throw data to stdout / stderr that you don't expect. ) 

tldr; What's the strtegy for stdout / stderr with the procs that get spun up by meltano and how does the main meltano process handle this 

To answer just the log format question, I'm partial to a lot more information. But you run into a bunch of issuses with the down stream taps/targets having their own logging methodolgy. Deep diving a bit you could probably 

Datetime - PID 

Stopped writing as I realized down stream taps/targets have their own logging information that meltano passes up here... :O

@visch what are your thoughts on https://gitlab.com/meltano/meltano/-/issues/2948? basically just dump everything to stdout.

Really depends on where Meltano UI is going. I'm not a good person to answer that portion. With where Meltano is today as a user I'd say ignore the UI entirely which makes the problem way easier to answer (as Kat is right that Meltanos built in orchestrator is less than what's desired), because then you tell them to run

meltano elt

with their orchestrator of choice which works great with stdout/stderr and all the docker stuff will work! Meltano pushes the logging stuff to the orchestrator so with airflow / dagster maybe a log provider like https://docs.dagster.io/_apidocs/libraries/dagster-papertrail would do what is wanted? I don't think airflow has an out of the box log prover for papertrail. All of that would probably get them to where they need to be, and would check the boxes of a 12 factor app?|