https://linen.dev logo
Join Slack
Powered by
Hey everyone. I’m interested in knowing if there’...
# troubleshooting
p
Hey everyone. I’m interested in knowing if there’s a way to have error/exception logs emitted from taps/targets show up in the console when calling 
meltano run tap-a target-b
with 
--log-level=warning
. It seems like Meltano has a wrapper around the logs emitted from the tap/target that converts them to log level 
info
, even though they’re logging an exception using the 
error
level. The main thing we’re trying to achieve here is the ability to troubleshoot tap/target errors with log level 
warning
to avoid the overload of information produced when running with log level 
info/debug
. I have gone through the documentation available here, but couldn’t really figure out how to achieve that. I’m happy to provide more context if needed. Any insights would be appreciated!
Here’s the console output using 
--log-level=info
.
log_level_info.log
Here’s the output using 
--log-level=warning
for the same tap running under the same condition — meaning it produced the same error as above.
Copy code
2024-05-22T19:37:36.216660Z [error    ] Extractor failed              
2024-05-22T19:37:36.217302Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Extractor failed') exit_codes={<PluginType.EXTRACTORS: 'extractors'>: 1} set_number=0 success=False
e
Hey @Pedro Ceriotti! This is something that's not currently possible but I've wanted to support for a while. The main blocker is the fact that we only capture a tap's stdout and stderr without parsing any log record details (e.g. level, logger name, etc.). The way I can imagine Meltano supporting this is by starting a socker server in a thread when the CLI starts, and then use Python's logging config in the tap side to configure a 
logging.handlers.SocketHandler
that points to Meltano's socket server. The biggest questions are probably: 1. How to safely run the socket server in a thread without causing too much overhead for Meltano 2. How to handle SerDe for log records, e.g. 
pickle
vs third-party like msgpack I'll try to put these thoughts in a feature request on the repo, but feel free to beat me to it and I can comment with details 😅
p
Got it @Edgar Ramírez (Arch.dev). That seems harder than I initially thought 😅 At this point, is there any workaround available, like using a logging.yml file or any other config I’m not aware of?
e
Yes! You could direct the tap logs to its own file: https://sdk.meltano.com/en/v0.37.0/implementation/logging.html#custom-logging-configuration You could also automatically apply it to all sdk-based taps and targets by setting the env var in `meltano.yml`: https://github.com/edgarrmondragon/meltano-dogfood/blob/a7e77c95f168e6ef1b1c7c538fc4d954b89b71cd/meltano.yml#L4-L5
p
You rock @Edgar Ramírez (Arch.dev)!
I’m afraid the workaround you suggested won’t work for our use case, as we have to parse the logs directly from the console output, but I’ll give it another try to see if I can figure something out.
e
I've got a very rough implementation of that in a branch based on Python's logging cookbook but it's gonna be hard to prioritize myself getting that to work cleanly 😅
and fwiw that does seem to work on simple scenarios:
Copy code
2024-05-23T01:29:14.747428Z [info     ] Environment 'dev' is active   
2024-05-23T01:29:14.778493Z [info     ] About to start TCP server...  
2024-05-23T01:29:15.339716Z [info     ] Beginning incremental sync of 'items'...
2024-05-23T01:29:15.339972Z [info     ] Tap has custom mapper. Using 1 provided map(s).
2024-05-23T01:29:15.835060Z [info     ] METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.490254, "tags": {"stream": "items", "endpoint": "/v3/get", "http_status_code": 200, "status": "succeeded"}}
2024-05-23T01:29:15.859086Z [info     ] JSONPath $.list.* match count: 1
2024-05-23T01:29:15.859605Z [warning  ] Stream is assumed to be unsorted, progress is not resumable if interrupted
2024-05-23T01:29:16.138136Z [info     ] METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.277045, "tags": {"stream": "items", "endpoint": "/v3/get", "http_status_code": 200, "status": "succeeded"}}
2024-05-23T01:29:16.138329Z [info     ] JSONPath $.list.* match count: 0
2024-05-23T01:29:16.138491Z [info     ] Pagination stopped after 1 pages because no records were found in the last response
2024-05-23T01:29:16.138615Z [info     ] METRIC: {"type": "counter", "metric": "http_request_count", "value": 2, "tags": {"stream": "items", "endpoint": "/v3/get"}}
2024-05-23T01:29:16.138676Z [info     ] METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.7984960079193115, "tags": {"stream": "items", "context": {}, "status": "succeeded"}}
2024-05-23T01:29:16.138810Z [info     ] METRIC: {"type": "counter", "metric": "record_count", "value": 1, "tags": {"stream": "items", "context": {}}}
2024-05-23T01:29:16.192627Z [info     ] Incremental state has been updated at 2024-05-23 01:29:16.192618.
2024-05-23T01:29:16.198145Z [info     ] Block run completed.           block_type=ExtractLoadBlocks err=None set_number=0 success=True
that warning log☝️ is coming from the tap
p
Niiice! That looks promising 🙏
4 Views
Open in Slack
PreviousNext
Powered by