Analytics vs. Heuristics-Decision Making in the NFL

[ArtVandalay]

Analytics are good but often misunderstood and applied incorrectly. Analytics themselves don't tell you want to do, they give you additional detailed information that will need to be considered within context of the data and the application for which they are being used.  

 

Game situation, assessment of current player performance within the game and current series, weather, matchups, etc all still apply.  

 

Everyone wants to put a single number to everything that tells them what to do but that's not how analytics work, although that's how they are often used especially during a broadcast. 

 

Nothing drives me more nuts when they (the broadcast) flash a stupid single number on the screen and say you have a 58% chance here you should go for it, that generalized single figure isn't exactly applicable to the particular situation.  But most likely the teams are using it in the proper context, it's just the talking heads that aren't. 

[Old Coot]

I think the basis of decision-making must be heuristics; that is, relying on rules of thumb based on experience.  That said, analytics should inform heuristics.  Take for example, the 4th & short decision from your own end of the field.  Maybe the analytics say there's an 80% chance of success.

 

But you need to dig deeper.  What factors led to the 80% success rate & what factors account for the 20% failure rate.  Further, waht is your own team's experience with short yardage pickups.  I would not expect the Bills to have an 80% chance of success on 4th & short & I don't think they have.  Why? Their OL.

 

My 2 cents

 

 

[GoBills808]

22 minutes ago, KzooMike said:

In the example I stated, under a singular trial, I don't have enough trials to risk my life savings even with a 30+EV at the age of 60. That would be mathematically the correct decision, but in real life, I'm pretty certain even most statisticians would not take that proposal. In the situation Staley faced, I would rather extend the trials/plays I have with the Raiders because I feel I have the better team. Last thing I would want to do is introduce a huge amount of variance into the equation by doing things like going for it on my own 18 yard line even if it does come with a 3% EV.  

I think the argument is that at least the variance in taking the +EV play is a known element, whereas going 'by feel' is simply throwing darts blindfolded. Reducing variance is a function of going for it more often, not less.

[Mikie2times]

Just now, GoBills808 said:

I think the argument is that at least the variance in taking the +EV play is a known element, whereas going 'by feel' is simply throwing darts blindfolded. Reducing variance is a function of going for it more often, not less.

I never suggested you go by feel and that alone. I said it's somewhere in the middle of math and common sense. Hence my example of risking your life savings at 60 years old with a 30% EV. It's statistically the correct decision, but very few would ever consider such a thing. I disagree with your comments over variance in the course of one game or even one season or even many seasons for one team. Just not enough trials for the data to stabilize. A really easy way for a great team to get beat by a horrible one is do things like go for it on the 18 yard line. It generated a potential 17% swing in win probability on a singular play. If your team is better, you don't want to introduce those type of events voluntarily. As you stated yourself, you use the data as an outline and part of the decision making process. Even the correct decision statistically can equate to pure gambling depending on the context.   

[GoBills808]

21 minutes ago, KzooMike said:

I never suggested you go by feel and that alone. I said it's somewhere in the middle of math and common sense. Hence my example of risking your life savings at 60 years old with a 30% EV. It's statistically the correct decision, but very few would ever consider such a thing. I disagree with your comments over variance in the course of one game or even one season or even many seasons for one team. Just not enough trials for the data to stabilize. A really easy way for a great team to get beat by a horrible one is do things like go for it on the 18 yard line. It generated a potential 17% swing in win probability on a singular play. If your team is better, you don't want to introduce those type of events voluntarily. As you stated yourself, you use the data as an outline and part of the decision making process. Even the correct decision statistically can equate to pure gambling depending on the context.   

Of course context matters...that's why you shouldn't gamble with your life savings. But if you are (for example) playing cards with a responsible portion of your bankroll and NOT taking that +EV you are, over time, a losing player. I suspect over the next few seasons this will bear out in football as well.

[Mike in Horseheads]

1 minute ago, GoBills808 said:

Of course context matters...that's why you shouldn't gamble with your life savings. But if you are (for example) playing cards with a responsible portion of your bankroll and NOT taking that +EV you are, over time, a losing player. I suspect over the next few seasons this will bear out in football as well.

I just bet pineapples and coconuts

[Cash]

2 hours ago, GoBills808 said:

Calculating things that teams rely upon when making decisions in-game, like win% or EPA, shouldn't have a 'right' or a 'wrong' attached to it. It's just a formula that  informs your decision making. If you want to build a circular swimming pool and you calculate the diameter but then decide you'd prefer a rectangle, your initial measurement isn't wrong. You just didn't use it.

 

Can you clarify what you mean? I didn’t say anything about right or wrong in the portion you quoted. I specifically used the same language you did in the post I replied to. I doubt that you’re disagreeing with yourself, so I think I’m just not following you here. 
 

59 minutes ago, KzooMike said:

I never suggested you go by feel and that alone. I said it's somewhere in the middle of math and common sense. Hence my example of risking your life savings at 60 years old with a 30% EV. It's statistically the correct decision, but very few would ever consider such a thing. I disagree with your comments over variance in the course of one game or even one season or even many seasons for one team. Just not enough trials for the data to stabilize. A really easy way for a great team to get beat by a horrible one is do things like go for it on the 18 yard line. It generated a potential 17% swing in win probability on a singular play. If your team is better, you don't want to introduce those type of events voluntarily. As you stated yourself, you use the data as an outline and part of the decision making process. Even the correct decision statistically can equate to pure gambling depending on the context.   

And inversely, it might make a lot of sense for bad teams to try high variance strategies more. Who cares if you lose by 30 or lose by 17? (Note: this doesn’t apply if you can keep it close. NFL football has extremely tight margins, and even the worst team in the league can beat the best team without needing very much luck.) Jauron-ball can take even a pretty bad team to 7-9, but as a fan, that doesn’t do much for me. Especially since it’s boring to watch. 
 

Side note: some good discussion above around sample sizes. That’s the real crux for me. I can’t pretend I know what the true answer is, but I suspect that NFL games/seasons will never have enough decision points to justify blindly following a model for all decisions, even if that model was perfect. (And I can’t stress this enough: ALL MODELS ARE IMPERFECT.) I understand GoBills808’s point that going for it more increases your sample, and thus increases the chance that your +EV actually benefits you, but I’m skeptical that it’s enough of an effect to ultimately matter. Especially since many of the decisions we see are only very narrowly +EV according to the imperfect models we have. Many of those are probably actually -EV, and either the model is imperfect, or they’re just within the margin of error. 

[GoBills808]

40 minutes ago, Cash said:

 

Can you clarify what you mean? I didn’t say anything about right or wrong in the portion you quoted. I specifically used the same language you did in the post I replied to. I doubt that you’re disagreeing with yourself, so I think I’m just not following you here. 
 

And inversely, it might make a lot of sense for bad teams to try high variance strategies more. Who cares if you lose by 30 or lose by 17? (Note: this doesn’t apply if you can keep it close. NFL football has extremely tight margins, and even the worst team in the league can beat the best team without needing very much luck.) Jauron-ball can take even a pretty bad team to 7-9, but as a fan, that doesn’t do much for me. Especially since it’s boring to watch. 
 

Side note: some good discussion above around sample sizes. That’s the real crux for me. I can’t pretend I know what the true answer is, but I suspect that NFL games/seasons will never have enough decision points to justify blindly following a model for all decisions, even if that model was perfect. (And I can’t stress this enough: ALL MODELS ARE IMPERFECT.) I understand GoBills808’s point that going for it more increases your sample, and thus increases the chance that your +EV actually benefits you, but I’m skeptical that it’s enough of an effect to ultimately matter. Especially since many of the decisions we see are only very narrowly +EV according to the imperfect models we have. Many of those are probably actually -EV, and either the model is imperfect, or they’re just within the margin of error. 

I think right and wrong are largely defined by outcomes, for better or worse, so what I meant was simply that making +EV decisions as practice is at least a logical framework for in-game decisions.

 

And no, there probably aren’t enough data points to say conclusively that always talking the higher EV or W% 100% of the time is the correct approach…but you will be even harder pressed to argue the inverse. Similar to how there haven’t ever been two hands of poker played exactly the same way- we can still construct ranges, calculate EV at every spot, and make decisions that are as GTO as possible…and these decisions will invariably be better than someone who does not follow a similar strategy.

 

 

[hondo in seattle]

If I were a NFL HC, I'd read up on Game Theory, build up a good analytics department, and get smart on all this stuff.

 

And then follow my more informed gut when making big calls.

 

[colin]

15 hours ago, Nextmanup said:

Note that OP is using heuristics, effectively, in his arguments...and not math or logic.

 

Which is kind of funny.

 

It's this simple: as you approach a decision in a game, your choices have win probability associated with them.   We can use the benefit of thousands of games having been played before with similar situations to calculate what is the better approach, i.e., what has the highest probability of success. 

 

And better is better.  Period.

I think some fans confuse "better" with "guaranteed."

 

Analytics is NOT about calculating the winning play, it's about giving yourself the highest probability of success with a given play.  It can still go badly though.  

 

A coach can CORRECTLY choose to go for it on 4th down 10 times in a row, and not get the first down EVERY TIME and still be correct in his decision making (if the circumstances support that)....and he still had a better shot of winning the games by going for it all those times on 4th down than he would have by punting, even though they ALL went against him.  

 

Before arguing about that, think about it for a bit.

 

 

 

 

You just gave me a thought.  I wonder if, in 10 years, "punters" will no longer exist.

 

The idea being that punting is so rare, teams can't justify a roster spot for one.

Have the place kicker do it the best he can and move on.

 

Hell, that might be a good idea NOW.  Make better use of the roster spot.

 

 

 

 

The huge problem with the analytics set in the NFL is the data.  baseball and basket ball are great for it, because there are so many games and it is so repetitive so you get good data (where the data sets can be generally IID, interdependently and identically distributed). 

 

in football, the line ups, the combination of players (WRT the position they play rather than just independent talent levels) and the context of each play (how many yards needed for a first, did a corner just get dinged and you want to press the replacement, is your rb or wr winded from a big play just prior?) simply muddies the data way way too much.  as gunner pointed out before, the actual results of so many specific data sets are hugely colored by exogenous variables (i.e. better offense/worse defense vs an opponent slants people to go for it on 4th more, improving the results of that data which will not carry over to other specific match ups).

 

You also have data that is basically not quantified, how well the OL or DL plays, specific skills of individual players and how the impact a play (say a route runner vs a big fast guy at wr in a given down and distance).

 

if all of this wasn't enough, you have football played at the nfl in 72 degree shopping mall domes, as well as in various end of days weather events we've had in buffalo, and psychotic cold and snow in GB and other places.  one year a while back minni was re doing their stadium (or whatever) and had to play at the U of minni's field.  they had a playoff game vs the seahawks, who had a pretty nasty team.  that game was like 6-3 or 10-6 or some such, and was just a slug fest.  going for it on 4th down in that game would have been way less likely to succeed compared to a playoff game in a dome.

 

and last but also least, the actual rules and their selective enforcement have changed in the NFL much much more so than in the NBA and MLB.  things that were standard 10 to 15 years ago are now harsh penalties today, so if you take data over a long enough time, you have some real apples to oranges problems.

[Cash]

12 hours ago, GoBills808 said:

I think right and wrong are largely defined by outcomes, for better or worse, so what I meant was simply that making +EV decisions as practice is at least a logical framework for in-game decisions.

 

And no, there probably aren’t enough data points to say conclusively that always talking the higher EV or W% 100% of the time is the correct approach…but you will be even harder pressed to argue the inverse. Similar to how there haven’t ever been two hands of poker played exactly the same way- we can still construct ranges, calculate EV at every spot, and make decisions that are as GTO as possible…and these decisions will invariably be better than someone who does not follow a similar strategy.

 

 

Gotcha. Yeah I don’t disagree. Mostly I just think the “nerds” are way overconfident in both their models and their brains. 
 

EDIT: Also I like the characterization of right & wrong largely being outcome-based. When I play blackjack, I play by the book basically every hand no matter what. But I also love the mental exercise of seeing how things would’ve played out if I’d done the opposite action instead. It’s very common that “playing scared” (staying on 16 vs a 10) would lead to the dealer busting instead of me busting. I have no way of predicting when those come around, so I just stay with the book. But in something like football, there’s too many variables to have as simple a book as blackjack. One of the analytics-based tools I’ve seen phrases things a little differently than most: it basically says what the minimum probability of a conversion is, in order to make going for it a +EV decision. That, to me, is much more useful to a coach. I.e., if the model thinks you only need a 10% chance of picking up 4th and 3 to make it worthwhile, then it’s an easy decision. If it’s more like 60%, the decision’s much harder, but it’s still doable to think about how your offense is playing, what’s been working this game, and even how confident you are in a specific play call. 

Edited January 12, 2022 by Cash

[JohnNord]

I find that reporters who had advocated strongly for better use of analytics on 4th down are the ones firmly supporting Staley and making excuses for him.  
 

To me, he’s right on a lot of his decisions but he’s also cost his team a few games this season including Week 18.  To make matters worse, he’s been bailed out by Justin Herbert’s incredible play.  If he didn’t have that, it would be way worse but…perhaps that’s why he takes the risks he does