Detecting blocking calls using async in C#

Way back in 2012 Stephen Cleary warned us Don’t Block on Async Code… and that’s still good advice today. The removal of AspNetSynchronizationContext means that ASP.NET Core developers have fewer issues with deadlocks… but ThreadPool Starvation is still a very real problem and so it’s critical that you avoid blocking calls in your codebase if you want to deliver a stable, high performing product to your users.

Table of contents

What are blocking calls?

Consider the following code:

var programStart = DateTime.UtcNow;
var task1 = Task.Delay(1000);
var task2 = Task.Delay(1000);
await task1;
await task2;
Console.WriteLine($"Total: {(DateTime.UtcNow - programStart).TotalMilliseconds} ms");

Running this will produce something like the following:

Total: 1046.045 ms

Walking through what the code is doing, we record the program start time, then we start a couple of tasks (these are started implicitly when they are created)… then we wait for the tasks to signal completion and finally we write out how much time the whole process took.

Note in particular that although each task takes at least 1000ms, the total time elapsed for the program was only about 1046 ms… which illustrates how .NET runs these tasks concurrently (possibly on separate threads).

So far so good. Now let’s change the code so that we block when running the first task:

var programStart = DateTime.UtcNow;
Task.Delay(1000).Wait();
var task2 = Task.Delay(1000);
await task2;
Console.WriteLine($"Total: {(DateTime.UtcNow - programStart).TotalMilliseconds} ms");

Now when we run the program, with the call to Wait() on line 2, we see something quite different:

Total: 2065.148 ms

Rather than running the tasks in parallel, .NET is waiting until the first task has been completed before it starts running task2. Blocking calls unnecessarily in your application leads to poor performance, which is really sad 🥺

There are maybe times when you actually want your program to behave like this. But even then, this is not how you should write the code as the execution time is not the only problem. In addition to running the tasks one after the other, the use of Wait() above also blocks a thread so that it cannot be used to execute other code until the task has completed. Blocking calls on hot paths in your application can lead to ThreadPool starvation and deadlocks, which would be really really bad 🦹🏻‍♂️

Instead, you can execute these things sequentially without hogging a thread by awaiting each task as soon as it’s created, like so:

var programStart = DateTime.UtcNow;
await Task.Delay(1000);
await Task.Delay(1000);
Console.WriteLine($"Total: {(DateTime.UtcNow - programStart).TotalMilliseconds} ms");

That version of the code will output something very similar to the blocking version, but it does so without blocking any threads.

If you take one thing away from this blog post then, as Stephen Cleary pointed out back in 2012, you really should avoid using blocking calls like Task.Result or Task.Wait() in your code.

How do I know if my application has blocking calls?

Now that you’ve read everything above (and once you’ve gone back over those awesome Stephen Cleary posts) you’re probably thinking, “Cool, I’m now shielded by my wisdom and don’t have to worry about blocking code anymore”. That’s kind of true.

However developers usually work in teams… so make sure you stay on your toes during any code reviews.

And then there’s all the code that you didn’t review, either because one of your other teammates reviewed it or because it was written before you joined the team. Maybe nobody in your team ever reviewed it. Maybe it was written during some dark days when code reviews weren’t a thing at your company…

Or maybe it’s in a third party NuGet package that you’re using and nobody at your company ever saw that code?!

How can you be sure none of that code has blocking calls? 🤔

Ben.BlockingDetector

In my day job, I’ve recently been playing around with integrating blocking detection capabilities from Ben Adams’ BlockingDetector into the .NET SDK for Sentry. Ben built his blocking detection library back in 2018 and, from the number of stars on the repository, you can tell that it’s pretty popular - with good reason!

However the docs for Ben’s BlockingDetector are pretty light. They explain how to use it, but there’s no detail on how it actually works. Since I had to figure some of that out in order to be able to extend it, I figured I’d share some of my learnings here.

Ben’s code includes two different mechanisms for detecting blocking calls:

1. DetectBlockingSynchronizationContext
2. TaskBlockingListener

DetectBlockingSynchronizationContext

If you’re writing a classic ASP.NET application or a WinForms application then there will be some kind of custom SynchronizationContext at play (e.g. AspNetSynchronizationContext in an ASP.NET application). In those instances, the DetectBlockingSynchronizationContext can be setup as a wrapper around the default SynchronizationContext, as a means to intercept and detect blocking calls.

Calls to Wait are intercepted by this custom SynchronizationContext and passed on to a privately held instance of a BlockingMonitor (which I’ll describe below).

Note: Since Ben.BlockingDetector is described as “Blocking Detection for ASP.NET Core”, I’m not sure why DetectBlockingSynchronizationContext is included… perhaps Ben was thinking of extending the blocking detector to support other application types at some point. In any event, it’s a cunning use of a custom SynchronizationContext and interesting code, if you’re curious.

ASP.NET Core - TaskBlockingListener

For ASP.NET Core applications (where there is no SynchronizationContext), that wrapper class won’t work. Instead Ben created TaskBlockingListener, which is a custom EventListener that listens to tracing events that get emitted from Tasks in .NET.

To be honest, the System.Diagnostics.Tracing library isn’t very nice to use and so this part of Ben’s code all looks quite obscure. It’s filled with lots of magic numbers that only make sense if you familiarize yourself with how the TPL was instrumented for tracing. Thankfully Ben did all the legwork of reading the source code to work out what those magic numbers are and, with luck, we’ll never have to look there ourselves.

The main thing to note in TaskBlockingListener.cs is that when blocking calls are intercepted, just as with the custom SynchronizationContext above, these are ultimately handled by a BlockingMonitor.

BlockingMonitor

The BlockingMonitor is what actually handles blocking calls and it’s doing a few things.

Firstly, it holds a t_recursionCount variable. This is used to track nesting. If you have nested or recursive blocking calls, only the outermost (first) blocking call triggers blocking detection.

Secondly, it trims off the stack trace to remove elements from the call stack that come after the blocking call (e.g. the code from Ben.BlockingDetector itself).

Thirdly, it “signals” that a blocking call was made… and in Ben.BlockingDetector this is done simply by logging a warning (using a little ILogger extension method).

Detecting blocking calls in production with Sentry

I would be remiss if I didn’t also mention how we extended Ben’s BlockingDetector and used this in the .NET SDK for Sentry.

Ben’s blocking detector is neat but it’s only really useful if you:

1. Have access to the diagnostic logs
2. Are monitoring these and notice the warnings or have some kind of alert mechanism setup

Detecting issues in production code running on your users’ devices is exactly the kind of thing that Sentry is really good at… so Sentry + Ben.BlockingDetector just seemed like Peanut Butter and Jam - why wouldn’t you?

We had to tweak Ben.BlockingDetector a bit so that it worked well with Sentry and we made a couple of changes that we thought/hoped users might appreciate.

Blocking calls as errors

The main thing we did was to send an “Error” to Sentry whenever a blocking call is detected, rather than simply logging a warning to the console or some file that nobody will notice until the roof is burning. This means you get real time notification of blocking calls detected in your application when it’s under real world workloads.

This is an example of what you see in Sentry:

You can find the code that generated that blocking call in the Sentry ASP.NET Core MVC sample.

Grouping and tidying the call stack

Something else we did is to modify the logic for which frames get included in the call stack that shows for blocking call events. Ben’s original code had some fairly simple logic to deal with this. We noticed that the call stacks we were seeing would vary, depending on how tasks had been scheduled to run by the TPL. So we put in place something a bit more robust that excludes any frames from the TPL itself (as well as any code from the Ben.BlockingDetector module).

Blocking suppression

Finally, in the Sentry implementation, we added the ability to suppress blocking detection for certain blocks of code by doing something like this:

using (new SuppressBlockingDetection())
{
    Task.Delay(10).Wait(); // This is blocking but won't trigger an event, due to suppression
}

Refactoring for testability

There were a few other things we did to refactor the original code, just to make it more testable but otherwise it’s largely the same bones as the one that Ben Adams wrote back in 2018 - only with a convenient dashboard and alerting mechanism, thanks to Sentry.

Wrapping up

Avoiding blocking calls in .NET is crucial for application responsiveness and scalability. You can use tools like Ben.BlockingDetector and log monitoring to discover potential issues in your codebase. Or you can use Sentry’s blocking detection that I helped build 😛. However you choose to do it though, make sure you’ve got some mechanism in place to detect blocking code early (before one of your customers tells you your application has frozen)!