Mar 12, 2020
Reporting Errors in Production: The Type/Stack Conflict
By Oren Rosenblum, Senior Developer at Minute Media
The following is an article that originally appeared on Minute Media’s Tech Blog.
Lately, my team and I have started reporting errors to Sentry, and I hate to admit it, but only then did I notice that my errors weren’t telling me much.. I would either get my custom error types, or a relevant stack trace, but not both.
In this post I’ll go over these concerns and suggest simple ways to get both information types. After all, the main reason we have errors is for us to have something to report on when our application fails, isn’t it?
What Reasonable Reporting Should Look Like
First, let’s review some of the main concerns when handling errors in production:
The Message should be composed of all the messages in the error chain.
The Stack Trace should be from where the error was created, not from where it was reported.
The Type should preferably be a name that means something to us. Take for example this mock Sentry interface:
In the above example, sql.updateArticleError is the only name which means something to us. All of the other types belong to the framework which created them.
To understand how we can fill these requirements, we first need to talk about a popular player in this field, pkg/errors.
pkg/errors
pkg/errors, created in 2016 by Dave Chaney, is a replacement for the standard error package. It provides great functionality for wrapping and unwrapping an error, which you can achieve by these two main functions:
Not only that, it also provides a stack trace for the errors it creates. To get the stack trace, just cast the err to stacker: Thanks to this package, the message we see in Sentry will be composed of all the messages during the chain. Unfortunately, if we want the relevant stack trace and our own custom type, we will encounter a small issue.
The StackTrace/Type conflict
Sentry unwraps the error before actually reporting it. You can see in the CaptureError implementation, that it uses Cause on the error reported.
The base of the conflict lies in the fact that the same error is used both for the type and for the stack trace. We can easily have a custom type or a relevant stack trace, but not both.
The next pseudo flow gives us a better idea of what’s happening:
Imagine that extsql.db returns an error. How should we treat it in our articlesRepository?
1) Create a new error
We can create a new error using pkg/errors:
In this case, we will have the stack trace from here, which is where it was created, but the type will be meaningless to us. This is because it’s the type of the inner struct of pkg/errors.
2) Wrap a custom error
Assuming we have this const error in our sql package:
We can wrap it like this:
As a result, we will see sql.articlesRepositoryError as the type,but unfortunately we will have an irrelevant stack trace. This is because our error doesn't implement the stacker interface.
3) Wrap the error from the db
What’s even worse, is when we wrap an external error, which may not follow pkg/errors practice:
If the external sql supports pkg/errors and we aren't interested in our own type, it might be ok and we'll want to do just that. But in some other cases, this error might just be errors.String. We won't have either a stack trace,or a meaningful type. So what's the solution if we don't want to just implement this stacker interface in all of our custom types? Compromise...
Compromise 1: Keep the Stack Simple, Complicate the Type
How will you make your custom error implement the stacker interface without actually implementing it? You can use embedded struct.
If our custom error is a struct embedding stacker, we can use it as the origin error without worries.
This way we will get our own type sql.articlesRepositoryError nd the stack trace. We won't need to do any extra work on the reporting side, thanks to Sentry's support of the stacker interface.
Compromise 2: Keep the Type Simple, Put Extra Work For the Stack Trace
Since not everyone likes to embed a struct on a daily basis, let’s stick with the type we defined earlier and just wrap it:
When this error is reported, we will see our type sql.articlesRepositoryError, but not the relevant stack trace, since updateArticleErr doesn't implement stacker. The compromise will be to report the stack as an extra parameter. Instead of the main view, we will see it in the 'additional data' section in Sentry. We will get the deepest stack trace we can find by casting the errors in the chain to stacker:
After you get the stack trace, you can pass it to Sentry as a slice of strings. Sentry can handle it pretty well.
A small improvement is to add this stack trace only in the cases where the original error isn’t a stacker:
When putting it all together, our Sentry abstraction will look like this:
It seems like manually adding the stack trace is a lot of work, but remember, it is only done once so it might be a better solution than doing extra work on every error type.
What’s next:
What we just saw, in my opinion, are very basic concerns when it comes to reporting errors. In the next post, we’ll see how we can wrap an error of a dependency, but still keep our custom error type. Not surprisingly, it can easily be done using a simple custom errors package.
Regarding the “deepest stack trace” it’s not a new idea. If it is implemented in Sentry, it will resolve the whole conflict issue.
Meanwhile, the Go 2.0 errors inspection proposal, which just came out a couple of months ago, shares some pkg/errors ideas. It is very interesting to read and I'm excited to see what will happen with errors in Go in the future.
Originally published at https://tech.minutemedia.com on September 4, 2019.
