Hey folks 👋, I am wondering if there is a way to remove

RECORD

messages from debug logs, may be a flag or something ? I need this feature because the debug logs have lot of important details, but having the actual records in the logs causes 2 concerns 1) size when million of records 2) sensitive information in the data

by default they shouldn't be there. How are you running meltano, you could have a

logging.yaml

or maybe an

env

variable set that runs this in debug mode?

Running in debug mode is intentional, we have meltano pipelines orchestrated through A , for local test i run meltano elt using

--log-level=debug

for debugging, and at times i have to enable the

debug

logs in A

staging

as well, which means A logs would contain all the

RECORDS

in logs, forcing me to scroll down the logs to look for error or other information.

Got it, so you do mean running in debug mode. Hmm https://docs.meltano.com/guide/logging#logging I'm not certain you can but you should be able to do this via the formatters but I'm not exactly sure how to get the outcome you'd like!

There are a few similar discussions here and here, that talks about similar feature but doesn't conclude the discussion with needed outcome. I did explore the logging.yaml but not sure if i can modify the

message

to remove

records

, hence wanted to check if there was an option to do that.

Ok, that wasn't clear to me from your question, but it's clear now. I'm not sure what the answer is, sorry I can't help more!

Thanks Derek for quickly jumping on this one and providing your opinion. Cheers !

Cant you just redirect stderr to grep to apply live filtering?

meltano run tap-... target-... 2> >(stderr-filter >&2)