#3213 Meltano configures consistent logging for tap/target Issue created by MeltyBot Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3288 Originally created by @paul.tiplady on 2022-02-25 211625 --- Problem to solve Meltano recently wired up Structlog to make it easy to do JSON logging, or otherwise customize logging: https://docs.meltano.com/reference/settings#clilog_config However tap/targets/transforms configure their own log format, which results in weirdly-formatted log messages from these components: From DBT:

[32mCompleted successfully[0m

From tap-mysql:.

time=2022-02-25 00:15:46 name=tap_mysql level=INFO message=Stream ... is using incremental replication

Target audience Pipeline Operator? Further details As noted by Edgar Ramirez in Slack, Pipelinewise libraries have some common logging infrastructure: https://github.com/transferwise/pipelinewise-singer-python/blob/master/singer/logger.py Proposal Since we're slurping up tap/target logs in Meltano, and re-emitting them, we probably want the tap/target to use a simple log format like just logging the message, or perhaps using JSON logging with Structlog, which Meltano could then parse and re-log with all the context. (e.g. passing through the log-level as well). Since Meltano is installing and invoking the tap, perhaps the

PluginInvoker

or the install machinery can inject a logging shim that overrides the tap/target's logger. (I'm not sure if there's a generic approach that would work here). Edgar notes that this could be included in the Meltano SDK -- but in that case it wouldn't help for non-Meltano taps. Failing those approaches, this is perhaps something that could be folded into the Singer spec & core library, so that all tap/targets have a configurable log format. What does success look like, and how can we measure that? Operators can clearly see logs from tap/target/transforms, so that success/failure is easy to determine, and standard text search can be applied in a log aggregator. Regression test (Ensure the feature doesn't cause any regressions) • Write adequate test cases and submit test results • Test results should be reviewed by a person from the team Links / references meltano/meltano