Meltano

Shower thought this morning. (feel free to say it's a terrible idea, and sorry if this is the wrong channel)

--debug should output multiple files in your working directory

tap.stdout
tap.stderr
target.stdout
target.stderr

Positives is that we could dump everything there, like the catalog, state, config, and more. Great for debugging.

Negatives is the size of these files could be too much sometimes (seems like a a no brainer for debugging faster)

---

Singer is great for tap and target development for the simplicity. Orchestrators are great as well as they combine the tap / target and add a bunch of stuff for you, but the ease of debugging starts to go away in the orchestrator. This would give you the best of both worlds while in the orchestrator.

I know that --debug puts everything in your console but sometimes that's too unwieldly.

--superdebug ? haha for the most part the current meltano elt process works but there's times having all the data in separate files is nice

Not a bad idea! Can you open an issue? :slightly_smiling_face:

Agreed. If we can figure out a good spec for it, I can definitely see the value to having those file artifacts for debugging purposes.