How is everyone here instrumenting exception track...
# best-practicesd
david_wallace
06/30/2021, 3:41 PMHow is everyone here instrumenting exception tracking into their taps/targets?
v
visch
06/30/2021, 3:51 PMNot great 😕 I have been thinking about this a bunch though.
Right now if I get an exception it stops everything and gets thrown
d
david_wallace
06/30/2021, 4:51 PMsame for us. I’ve been considering forking the singer-sdk and wrapping the
cliv
visch
06/30/2021, 4:57 PMhttps://gitlab.com/meltano/sdk/-/issues/134 here's the issue I put together which has a bit more detail
visch
06/30/2021, 4:57 PMSounds kinda like what you're talking about
visch
06/30/2021, 4:57 PMIf you could add your thoughts there, Aj was asking me some questions that I haven't had time to dive into
d
david_wallace
06/30/2021, 5:00 PMthis is semi-related to what Im talking about. Im more just talking about instrumenting taps/targets such that exceptions/errors get forwarded to an error tracking service (e.g. Rollbar, Sentry, etc.)
david_wallace
06/30/2021, 5:00 PMless-so the underlying exception handling logic
v
visch
06/30/2021, 5:04 PMLog aggregation is more of what you're talking about?
d
david_wallace
06/30/2021, 5:04 PMsort of! Im thinking more about exceptions than logs
david_wallace
06/30/2021, 5:05 PMconcrete example - we had an exception get raised in a tap recently that never made it to datadog logs
david_wallace
06/30/2021, 5:06 PMeffectively a silent failure since we arent currently forwarding exceptions anywhere
v
visch
06/30/2021, 5:06 PMSeems like a major issue that exceptions are not being caught in logs!
d
david_wallace
06/30/2021, 5:08 PMyeah tbh it could also be a configuration issue. But the general idea of exception tracking being separate from log aggregation is a pretty common pattern.
v
visch
06/30/2021, 5:09 PMgot it! Sounds interesting I'd love to learn more
d
david_wallace
06/30/2021, 5:09 PMwas really just curious if anyone here is already doing this and if there have been any patterns established around it