Hi all, We’ve built several custom taps using Singer SDK and we’ve got an issue we’re unable to solve. We’re emitting logs from our taps and they are being captured by Meltano like this:

[2022-02-01 20:47:38,510] [79340|MainThread|meltano.cli.elt] [INFO] {'run_id': 'e47bc532-998c-4f85-8d3a-273a0ce33bbd', 'job_id': 'tk-test', 'stdio': 'stderr', 'cmd_type': 'extractor', 'name': 'tap-bw-ga-experiment-data-fr', 'event': 'time=2022-02-01 20:47:38 name=tap-google-analytics level=WARNING message=some warning', 'level': 'info', 'timestamp': '2022-02-01T20:47:38.510142Z'}
[2022-02-01 20:47:38,510] [79340|MainThread|meltano.cli.elt] [INFO] {'run_id': 'e47bc532-998c-4f85-8d3a-273a0ce33bbd', 'job_id': 'tk-test', 'stdio': 'stderr', 'cmd_type': 'extractor', 'name': 'tap-bw-ga-experiment-data-fr', 'event': 'time=2022-02-01 20:47:38 name=tap-google-analytics level=ERROR message=some error', 'level': 'info', 'timestamp': '2022-02-01T20:47:38.510404Z'}

The problem is that Meltano emits all logs with

INFO

level, instead of mapping log levels from taps (

level=WARNING

and

level=ERROR

in the example above). Is it possible to map log levels or otherwise prevent Meltano from wrapping all stderr output in info logs?

To confirm, do you want to suppress the logs from printing or just fix the classification?

Is it possible to map log levels or otherwise prevent Meltano from wrapping all stderr output in info logs?

We've talked about trying to support some variation of this for taps built with the singer sdk, but theres no support for that today.

First problem is that getting any logs from our taps requires lowering Meltano log level to

info

or lower, which then generates more logs than we want, including some sensitive info that we don't want to be sending to DataDog. Another issue is the inability to easily control what gets emitted by each tap, without chaning the tap code. If there was an option to stop Meltano from capturing stderr at all, that would solve these issues as we could then configure taps individually.