Meltano

image.png

Hi all; random Q. Is there a way to filter out certain logs from being emitted?

e.g. all these logs attached in image. we have a log resource in azure ingesting it so we can keep track of logs of any issues, and because we fan out like 90 + individual runs of EL it really pollutes our logs and brings up the $ cost. Logs are like 3x the cost of our container now :joy:

Are you already using a <https://docs.meltano.com/reference/settings/#clilog_config|log config>?

ooh, no not yet. i can look into this though!

i also found a more detailed page here ! <https://docs.meltano.com/guide/logging/>

Oh yup, that's better resource for getting started

Once you have that, you'll want something like <https://github.com/edgarrmondragon/meltano-dogfood/blob/17e48f6a3a357a9f0963fe292d8565e36c7b2fde/logging/dev.yaml#L37-L39> but set to `WARNING`.

Or <https://github.com/edgarrmondragon/meltano-dogfood/blob/17e48f6a3a357a9f0963fe292d8565e36c7b2fde/logging/singer_logging.yaml#L20-L22> also set to `WARNING`, but using <https://sdk.meltano.com/en/latest/implementation/logging.html#custom-logging-configuration>.