https://linen.dev logo
Join Slack
Powered by
There are a couple of new logging features I was t...
# contributing
d
There are a couple of new logging features I was thinking of that would be a big help to me. I brought up one of them and it was suggested that I take a crack at it. I took a quick poke around and I think it wouldn’t be too hard to add, but before I start I figured I check that it would be something you folks would actually want. The things I’m looking at are: 1. Logging in json. The logs get sent to datadog, and are much easier to work with in json format. 2. Filtering out 
RECORD
type messages when the log level is 
DEBUG
. I have an issue that’s only happening in prod, and I’m trying to avoid leaking sensitive data into the logs while investigating. I was thinking the easiest thing to do would be to add a new 
--log-config
option that would let you point to a yaml file, where you could configure logging however you like (e.g. https://docs.python.org/3/howto/logging.html#configuring-logging). What do you think?
e
I agree with thing 1 that JSON logs can be a quick win to start monitoring logs in datadog since error levels then get mapped. For thing 2 I'm not sure I follow. Do you need the tap to not output any record messages based on logging config?
a
Agreed with @edgar_ramirez_mondragon the logging config sounds like a nice quick win. I'm curious what you think about yaml vs the native python log-config format mentioned there in the link. I'm slightly more inclined to use the documented standard config format if it makes sense to do so. And regarding the suppression of record messages, would it be sufficient to sent them to something like a target-null which just disbands all stdin or do you need a different behavior?
d
For the 
RECORD
messages, they normally don’t appear in the logs (when the default 
INFO
level is being used). If I switch the log level to 
DEBUG
, I get a lot of useful extra info in the logs (which is helpful), but also those 
RECORD
messages (which isn’t desirable given their sensitive contents). The method of filtering them out doesn’t really matter to me, but I was looking at the filtering options in python logging as a possible way to do so 🤷. I’m not sure if that answers your question.
@aaronsteers An example yaml config is at the bottom of that section I linked. It looks like it’s the preferable way to config logging in >= 3.2
In Python 3.2, a new means of configuring logging has been introduced, using dictionaries to hold configuration information. This provides a superset of the functionality of the config-file-based approach outlined above, and is the recommended configuration method for new applications and deployments. Because a Python dictionary is used to hold configuration information, and since you can populate that dictionary using different means, you have more options for configuration. For example, you can use a configuration file in JSON format, or, if you have access to YAML processing functionality, a file in YAML format, to populate the configuration dictionary. Or, of course, you can construct the dictionary in Python code, receive it in pickled form over a socket, or use whatever approach makes sense for your application.
The nice thing is that you can just parse the yaml and pass it to 
logging.config.dictConfig()
. You can use json as well, but some (very brief) googling makes it seem like yaml is the more common format
e
In that case I prefer yaml since it's more common nowadays (and arguably more readable) than the .ini syntax of the old format
a
Nice! Yeah, thanks for the additional context. I agree, that does sound like the best approach. And yes, since we're getting into log-level configs, we should be able to configure different loggers (if they aren't indeed already created that way) so within the yaml log config you could say record-level logging is default (not emitted) but other log message are sent in JSON.
@dean_morin, from what I can see here, this is a valuable addition, for sure. Feel free to log and issue and start working on the MR if you feel you can take this on 👍
d
Sounds good!
s
@dean_morin, I'm curious where did this conversation get to? Is there are way to run with debug logging without emitting 
RECORD
 messages?
a
Not to disrupt the discussion flow here, but we have some progress to report on some points in this thread. I'm excited to share that "structured logs" is just about here and this includes some new built-in configurability 🎉 https://gitlab.com/meltano/meltano/-/merge_requests/2395 cc @florian.hines who's building this feature Thanks @steve_clarke for surfacing this thread again.
s
That's excellent news @aaronsteers, I'm hoping this might provide some additional information debugging the 100,000 issue with tap-salesforce to target-snowflake. Either way it will be a very handy feature. Thanks 👍🎉
a
@steve_clarke - Great! If you have the hour free, we have #C01QS0RV78D starting in just over an hour and we'll be presenting and discussing this upcoming feature. 🙂
d
Oh nice, glad this is almost done!
k
Hey folks, just came across this 🧵 , Could someone please help how the RECORD messages do not appear in debug logs ?
4 Views
Open in Slack
PreviousNext
Powered by