How our new NFL stats can better rate pass-catchers

Have you ever wondered what sets George Kittle apart from Travis Kelce, or what makes A.J. Brown such a special talent? Well, now we have an answer to those questions and many more.

Since 2018, ESPN has introduced pass-rush, run-stop, pass-block and run-block player metrics. Those metrics have offered a new way to see a part of football that largely goes unnoticed — or at least unquantified. For the first time, these metrics provided a method to isolate and assess individual player performance in a consistent and objective way.

The aim now is to do the same thing for receiving. Our new Receiver Tracking Metrics (RTMs) use player tracking data from NFL Next Gen Stats to analyze every route run — including those that are untargeted — and assess receiver performance in three distinct phases: getting open, contesting and making the catch, and generating yards after the catch (YAC). These three components also are blended to create an overall receiving metric.

ESPN Analytics is launching RTMs leaderboards next month in conjunction with FiveThirtyEight — you’ll be able to view updated numbers every week — but we’re unveiling the metrics now to provide a reference to explain what they’re all about.

What exactly are Receiver Tracking Metrics?

RTMs are a set of four metrics we’ve named Open Score, Catch Score and YAC Score, plus the overall combination of those three. All four are a per-play rate metric, rather than a counting or cumulative stat.

Each score is on a 0-99 scale, where 50 is roughly league average. The purpose of the metrics is not solely to rank receivers from best to worst; the goal is to describe and explain how a receiver is — or isn’t — able to produce yards.

---

How do they work?

All three components generally work the same way. For each, a benchmark is set based on the context and dynamic inner workings of the play. The metrics measure the degree to which the receiver exceeds or falls short of that benchmark.

For example, YAC Score looks at the tracking data at the time of catch and makes a prediction of how many additional yards a receiver will typically make, based on the locations, directions and speeds of all 22 players.

The receiver is credited (or debited) for the yardage beyond (or below) that benchmark, rather than the raw yards after catch gained. Some plays and situations lend themselves to a lot or a little YAC, so YAC Score doesn’t measure mere yards but rather the yards the receiver was able to generate beyond the expected amount.

---

Open Score: How is it possible to assess untargeted routes?

For every route run, Open Score assesses the likelihood a receiver would be able to complete a catch, conditional on if he were targeted. The assessment takes place a moment before pass release (0.2 seconds prior), because defenders read the shoulders of the quarterback at release and break on the targeted receiver. Otherwise, actual targeted receivers would appear to be less likely to complete a catch. Unfortunately, our models can’t directly know the signal-callers pass progression (the sequence of reads he makes during each play), but they are aware of the route type, depth and time after snap of the pass release. That means our models do have some sense of timing.

In 2017, I developed a concept now commonly called Completion Percentage Over Expected (CPOE). The idea was that NFL Next Gen Stats tracking data could estimate the chance of a completion on a pass, given the locations, directions and speeds of relevant players. If a completion actually occurs, the quarterback would be credited with all the probability between that prediction and 1. And if it’s not completed, he would be debited accordingly. There are various versions of this metric — and it is quite useful in some applications — but there is a fatal flaw when applying the concept to pass-catchers, as tempting as it might be.

The problem is we would be measuring receiver success accounting for the dynamic context of what is happening on the field. From the perspective of the receiver, however, he is a primary and direct influence on that very context. Think of it like this: CPOE measures the catching ability of a receiver, accounting for his ability to get open. When you think about it, it makes no sense — the better a receiver is, the higher the benchmark he sets for himself in the metric.

This problem vexed me for months, but about a year ago I thought of a way to crack it. If we can establish the probability of a catch of a typical receiver, given all the contextual details of a pass route, including route type, depth, coverage and many other variables, we can set a benchmark of expected “openness” agnostic to the ability of the receiver to get open.

Then we could compare the typical, expected openness for an average receiver to the actual openness assessed by a model looking at tracking data. This is the key to solve the problem: a receiver’s openness is compared to the typical receiver’s openness given the route, coverage, and depth, rather than the raw assessment.

---

How does the Catch Score component work?

The assessment to catch and contest works in a similar way to openness. Given the array of all 22 players’ positions, directions and speeds, the model estimates the probability of a completion. If a completion occurs, the receiver is credited with the marginal difference.

For example, if the tracking data indicates a pass will be completed 75% of the time and the receiver actually catches the pass, he is credited with plus-0.25. If he does not catch the pass, he is debited at minus-0.75. There are important modifications to this calculation, which I’ll detail below.

---

How are the three components blended into the overall metric?

Each of the three components are weighted in a way to best match real-world production, specifically a blend of predominantly yards per route with a bit of yards per target added. The resulting weights tell us a lot about the importance of the three skills.

For wide receivers and tight ends, Open Score accounts for roughly half of the overall score, while Catch Score accounts for a little over a quarter and YAC Score accounts for the remainder. We think these weights make logical sense, in that a receiver has to get open to have the chance to make a catch. He then has to catch the ball to gain additional yards. Without success in the early part of the sequence, he wouldn’t have many opportunities through the remainder of the process.

For running backs, YAC Score accounts for about half of the overall score, with Catch Score the second largest component, followed by Open Score. We think this also makes sense. Backs typically run swing routes, check downs and screens, which don’t require excellent route-running skills but do rely on yards after catch for success.

---

How do RTMs account for the quarterback?

Quarterbacks are clearly an essential factor in whether a receiver makes catches and gains yards. RTMs account for who’s throwing the pass in two ways:

1. We adjust the Catch Score and the part of the Open Score that assesses openness at pass arrival based on the quarterback.

2. We use pass accuracy data from ESPN’s video analysis tracking to adjust both the Catch Score and YAC Score based on the accuracy (high, low, ahead, behind) and intent of the throw. (For example, pass-catchers often receive an official target stat when the pass was clearly a throwaway; RTMs exclude throwaways.)

Quarterbacks sometimes can make receivers look good, but sometimes it’s the other way around. How do we know which is which? Our QB adjustments borrow a concept from hockey and basketball called Adjusted Plus-Minus. This approach is able to estimate each individual’s contribution to overall effectiveness, accounting for the presence or absence of other players around them. In this case, the adjustment is a simple adjusted plus-minus among the QB and his receivers.

One interesting insight from the adjustments is that quarterbacks have a large effect on the openness of receivers at pass arrival. Making the right read and extending the play plausibly are two big reasons for this. Of note, this is one of Patrick Mahomes‘ superpowers. The Chief’s quarterback is not a particularly accurate thrower, but he helps his targets get open. Part of this effect might be due to scheme, but unfortunately scheme and signal-caller overlap too much to parse those effects apart.

---

How do we know RTMs are good at evaluating receivers?

For starters, we could look at the top 10 seasons since 2017 (when our data begins). It’s hard to argue these aren’t dominant seasons by elite receivers. We can immediately glean insights. For example, we can see that A.J. Brown‘s 2019 season was buoyed by his ability to generate yards after catch despite a mediocre Catch Score, and that Cooper Kupp’s ‘amazing 2021 season did not rely on any one specific ability but was consistently solid across all three components:

Here are the top-five seasons in YAC Score since 2017:

The top-five Catch Score seasons include who we might expect … and then Marvin Jones Jr. Jones had an amazing season for the Lions in 2017, with 1,101 yards and a league-leading 18.0 yards per catch. With so many yards and an average YAC score, those yards must have been from deep or contested low-probability routes:

How about Open Score? Here are the top-five seasons since 2017:

The top-rated players mostly match our intuitive sense of great receivers, but there are more concrete ways of determining the usefulness of metrics. One is to measure how consistent they are from year to year.

If what RTMs measure are truly intrinsic to each individual receiver, then receivers should carry these qualities from year to year. For qualifying receivers, Open Score has a correlation coefficient of 0.61, where 1.0 would be perfect consistency and 0.0 would be no consistency at all. That’s not bad for a strictly objective measure as something as mercurial as receiver performance. Catch Score correlates at 0.38, and YAC Score correlates at 0.35. The overall score correlates at 0.52.

RTMs also match up well with existing public benchmarks of receiver performance. Since 2017, the overall score correlates with Pro Football Reference‘s Approximate Value stat at 0.68, with EA Madden’s player rating at 0.59 and Pro Football Focus’ receiving grade at 0.76.

For qualifying wide receivers, the overall score correlates with yards per route 00 which I believe is the best conventional stat to measure receiver production — at 0.76. This shouldn’t be thought of as falling short of a 1.0 correlation, because RTMs are hopefully doing a good job of subtracting the influences of context, as in routes, depths, coverages, double teams, quarterback skill and so on. Still, it’s encouraging to have a strong correlation with real-world production.

What is also encouraging is the three components of RTM generally do not correlate with each other. This suggests our metrics are truly isolating three independent skills that comprise receiver ability.

---

How do RTMs account for double teams?

Some receivers attract more attention from defenses than others, which allows other pass-catchers to get less attention. To account for this effect, Open Score is adjusted for the number of defenders exclusively “assigned” to a receiver.

For example, if there is a cornerback covering a receiver and a safety deep above him who matches the receiver’s pattern much more than any other receiver, that receiver is credited with extra attention. This approach not only accounts for dedicated double teams, but for coverage methods such as bracketing.

---

What else should I know about these metrics?

One thing to know is we exclude assessments of any nontargeted routes on a screen pass, because receivers typically are blocking rather than trying to get open. Only the Catch and YAC Scores are counted for targeted screen routes, because openness on those routes is due to play design far more than receiver ability.

Also, there are several other factors considered in establishing the benchmark on each route. These include route type, depth of route, coverage type (Cover 3, Man 2 and so on), position at snap (wide, slot, tight, backfield), distance from sideline, time after snap, down/distance/yard line and whether or not the play featured play-action.

Ultimately, the hope is these metrics are used to understand and explain how pass-catchers perform, rather than simply ranking them from best to worst. Unlike our win-rate metrics for line play, there already are reliable statistics that do a good job of telling us who are the best receivers. Gaining insight into how they either excel or underperform could tell us which are ready to break out, if they were just targeted more often, and which receivers are making their quarterback look better than they actually are.