Analytics vs. Heuristics-Decision Making in the NFL

[stuvian]

owners are using analytics to justify firing coaches. The Eagles Jeff Lurid (no sarc) questioned Doug Pederson's play calling all during Carson Wentz' meltdown on the way out the door. I can see the day coming when these alpha male owners decide to use algorithms to call the plays themselves. If Jerry Jones can be a GM why not coach too? After all, rich people know everything, right?

 

The Browns have gone all in on Moneyball. Maybe they can win the Superbowl on Madden. 

 

Analytics cannot account for the human factor and hidden intel on injuries etc. It can tell us part of the story but not the whole story.

[Albany,n.y.]

5 minutes ago, aristocrat said:

Staley got reckless in that third quarter 4 down on their 20. That's a punt situation. 

I had futures on the Chargers this year (pushed the over 9, lost the over 10.5) and I watched more Chargers games than what I'd usually see.  Staley has made some pretty bizarre decisions this season, like going for 4th down deep in his own territory, going for it on 4th down instead of kicking field goals etc.  I understood early in the season when he had a kicker he didn't trust, but when they improved the position he still did the same things.  It was very frustrating watching him cost a game by being so unconventional.  By the time they played the Raiders I was pretty convinced I couldn't trust him to get me a 10th win & I almost expected him to blow it, which I believe he did with the crazy 4th from their own 20.  .  

[aristocrat]

3 minutes ago, Albany,n.y. said:

I had futures on the Chargers this year (pushed the over 9, lost the over 10.5) and I watched more Chargers games than what I'd usually see.  Staley has made some pretty bizarre decisions this season, like going for 4th down deep in his own territory, going for it on 4th down instead of kicking field goals etc.  I understood early in the season when he had a kicker he didn't trust, but when they improved the position he still did the same things.  It was very frustrating watching him cost a game by being so unconventional.  By the time they played the Raiders I was pretty convinced I couldn't trust him to get me a 10th win & I almost expected him to blow it, which I believe he did with the crazy 4th from their own 20.  .  

 

I could understand on your own 40 but they were on the 18.  That's a free 3 points and maybe a td.  Free points when you're already down.  

[hondo in seattle]

I think analytics are great.  But analytics themselves are a bit of a fancy heuristic.  Analytics are an attempt to take something infinitely complex and boil it down to simple numbers.

 

And you can't do that.  You can't quantify everything.  

 

For example, not all 4th and 1s are equal.  Maybe the right guard has a sore hammy.  Maybe your OL really needs a confidence builder.  Maybe that part of the field is slippery.  Show me a 4th and 1 chart that accounts for all the human, situational, and environmental variables?  There isn't one.  So the coach needs to make the final decision.   Analytics can help him make a better decision but shouldn't make the decision for him.

[Sweats]

Hey, if it were up to me, there'd be zero punts, no FG's and every team would go for it on 4th down.....regardless of score or field position.

It just makes the games more tolerable to watch sometimes.....sitting on the edge of your seat waiting to see if the team makes it on 4th and 22 is spectacular in my books, man.

[Mat68]

Anything that is absolute or not challengeable is a problem.  Staley going for it just because thats what they do is oppositely as stupid as always punting or kicking.  Nuance, is the key to coaching.  Every situation is different.  You cannot act like they are all the same in a vacuum.  I think Mcdermott isnt a perfect example but pretty close to what you would want.  In 17 with a pedestrian offense he was very conservative.  That resulted in a miracle playoff appearance.  Last season Buffalo was one of the more aggressive 4th down teams in football.  Different teams, different gambles.  For every try improving the odds of victory the failed attempt improves the odds of defeat.  Yes LAC were 4/5 on 4th down.  The 1 miss was when the outcome was much closer to even. The last 4 they made,  the team had no choice but to go for it because the game would be all but over.    

[GoBills808]

A good (mathematically sound) decision is a good decision regardless of outcome. 

[Rochesterfan]

2 hours ago, GunnerBill said:

My big issue with it is that the use of prior success rates creates an artificial slant towards aggression in the models. 

 

Teams with good offenses go for more 4th downs. And because they go for more 4th downs with their good offenses they convert them at a decent rate. That factors into the model going forward. 

 

If every team, including those with bad offenses, went for every 4th and let's say 3 or less, the conversion rate of 4th downs would fall and it would affect what the model says about whether to go for it. 

 

I would add things like situation sneed to be factored in many scenarios. 
 

For example - the 2 point conversion - many times the 2 point conversion is attempted by a team down multiple scores - late and the defending team don’t put up the same defense as what the Ravens saw when trying to win with the 2 point conversion and failing twice.

 

So overall from an analytical point of view - the 2 point conversion is >50%, but the numbers that occur late in specific situations is lower.

 

The same thing happens with 4th downs.  The numbers get impacted by situations like later game up by multiple scores and you are slowly letting a team move because time is more important than score/yards.

 

I think a conservative coach - like Fangio was - costs his team wins, but an overly aggressive coach can do the same thing.  Doug Pedersen used aggression to win a Super Bowl, but then that same approach cost him multiple games in the next couple of years when the offense was not as good.

 

 

Edited January 11, 2022 by Rochesterfan

[Rochesterfan]

1 hour ago, aristocrat said:

 

I could understand on your own 40 but they were on the 18.  That's a free 3 points and maybe a td.  Free points when you're already down.  

 

That is exactly the thing - even if the conversion rate is 80% - you are not in scoring range.  From the 20 - you still have a below average score rate.  
 

The risk of giving up immediate points - versus maybe getting 5 more first downs to score yourself has to be terrible.

 

The only way it makes sense is if you have a weather impact - going into the wind and want to maintain possession and not punt that direction.

 

 

[GunnerBill]

1 hour ago, hondo in seattle said:

I think analytics are great.  But analytics themselves are a bit of a fancy heuristic.  Analytics are an attempt to take something infinitely complex and boil it down to simple numbers.

 

And you can't do that.  You can't quantify everything.  

 

For example, not all 4th and 1s are equal.  Maybe the right guard has a sore hammy.  Maybe your OL really needs a confidence builder.  Maybe that part of the field is slippery.  Show me a 4th and 1 chart that accounts for all the human, situational, and environmental variables?  There isn't one.  So the coach needs to make the final decision.   Analytics can help him make a better decision but shouldn't make the decision for him.

 

And as of the present time it is impossible to build a model that factors in the human and environmental variables. You can account for situational variables, and indeed the models already do. They account for field position, they account for scoreboard, they account for time on the clock. The advanced analytical models really are amazing and you would be mad to discount them in the way the modern game works. However, the general models that networks etc use do have that inbuilt bias where they rely on past success rates that I outlined earlier. Indeed most NFL teams now have their own models for exactly that reason. Because when say Matt Nagy is deciding whether to go for a 4th a 3 with Andy Dalton the overall success rate of 4th and 3 plays from your own 48 when 7 points down with 8 minutes remaining that has been built in data from Tom Brady and Aaron Rodgers converting it a ton is of little use to him. Similarly if you have Justin Herbert converting 4th downs for fun your model probably doesn't care that Andy Dalton only gets 4th and 3 on 2 of 10 tries. So when you hear folks say things like "the number say you go for it there" those people have to be aware that teams are getting even more granular than a general model ever can. 

Edited January 11, 2022 by GunnerBill

[Cash]

3 hours ago, GoBills808 said:

A good (mathematically sound) decision is a good decision regardless of outcome. 

That’s technically true. But if your decisions keep leading to bad outcomes, how can you be sure they were good/sound decisions in the first place?

 

Not trying to accuse you of anything, but a lot of the analytics folks I’ve seen are deeply arrogant in a way that makes me question all of their conclusions. Their attitude seems to roughly be, “I’ve completely figured it out, therefore anyone who disagrees with my model is an idiot.” And every time they’re wrong, they come back with logic similar to the quote above. 
 

I have 2 main problems with these folks, plus a bonus third problem:

 

1.) How do they know their model is accurate? As GB & others pointed out above, the models are based on historical data. But as teams change their behavior, there’s no reason to expect that the models will continue to hold, or be predictive at a useful level. 
 

2.) Many of these folks seem to have no understanding of the concept of error. It’s maddening. I saw an article a few weeks ago, blasting a coach for a 4th down decision. Per the model quotes, the coach’s actual decision resulted in about 46.2% win probability, but the “correct” decision resulted in about 46.5% win probability. But there was no mention of the model’s margin of error. Off the top of my head, I can’t recall any NFL analytics people ever providing a margin of error. This is nuts. 
 

3.) Bonus: Arrogance. As I mentioned above, there’s a lot of “Everyone is dumber than me” energy coming from the current crop of analysts. This rankles me on a personal level, so I’ll admit some bias here. But I also think this is a legitimate issue. It ties in to both of the issues above: if you think you’re the smartest guy in the room, it’s often hard to self-evaluate. Much easier to make fun of “stupid” coaching decisions and tell your detractors that only process matters; never results. Plus, these guys are so high on their own farts that they can’t always interpret what their own models are saying. YardsPerPass pointed out an example on Twitter a while back. If you look at the Football Outsiders predraft QB write up in 2018, they lambaste Josh Allen and glow over Sam Darnold. (Nailed it.) But their own model only gave Allen about a 5% greater chance of busting than Darnold, and I think a pretty similar disparity in likelihood to break out. Based on the writeups, I would’ve expected Allen’s bust chance to be like 80-90%, compared to like 20% for Darnold. 

[Nextmanup]

Note that OP is using heuristics, effectively, in his arguments...and not math or logic.

 

Which is kind of funny.

 

It's this simple: as you approach a decision in a game, your choices have win probability associated with them.   We can use the benefit of thousands of games having been played before with similar situations to calculate what is the better approach, i.e., what has the highest probability of success. 

 

And better is better.  Period.

I think some fans confuse "better" with "guaranteed."

 

Analytics is NOT about calculating the winning play, it's about giving yourself the highest probability of success with a given play.  It can still go badly though.  

 

A coach can CORRECTLY choose to go for it on 4th down 10 times in a row, and not get the first down EVERY TIME and still be correct in his decision making (if the circumstances support that)....and he still had a better shot of winning the games by going for it all those times on 4th down than he would have by punting, even though they ALL went against him.  

 

Before arguing about that, think about it for a bit.

 

 

 

 

4 hours ago, Sweats said:

Hey, if it were up to me, there'd be zero punts, no FG's and every team would go for it on 4th down.....regardless of score or field position.

It just makes the games more tolerable to watch sometimes.....sitting on the edge of your seat waiting to see if the team makes it on 4th and 22 is spectacular in my books, man.

You just gave me a thought.  I wonder if, in 10 years, "punters" will no longer exist.

 

The idea being that punting is so rare, teams can't justify a roster spot for one.

Have the place kicker do it the best he can and move on.

 

Hell, that might be a good idea NOW.  Make better use of the roster spot.

 

 

[reddogblitz]

7 hours ago, Albany,n.y. said:

I had futures on the Chargers this year (pushed the over 9, lost the over 10.5) and I watched more Chargers games than what I'd usually see.  Staley has made some pretty bizarre decisions this season, like going for 4th down deep in his own territory, going for it on 4th down instead of kicking field goals etc.  I understood early in the season when he had a kicker he didn't trust, but when they improved the position he still did the same things.  It was very frustrating watching him cost a game by being so unconventional.  By the time they played the Raiders I was pretty convinced I couldn't trust him to get me a 10th win & I almost expected him to blow it, which I believe he did with the crazy 4th from their own 20.  .  

 

I was surprised he didn't go for it instead of kicking the FG to tie.

 

Its crazy. He's got a talented team but blows games with bizarre coaching decisions.

 

After they didn't get the 4th down from their own 18, the team fell off a cliff and muddled around for a quarter before getting too little too late at the end.

[GunnerBill]

2 hours ago, Nextmanup said:

Note that OP is using heuristics, effectively, in his arguments...and not math or logic.

 

Which is kind of funny.

 

It's this simple: as you approach a decision in a game, your choices have win probability associated with them.   We can use the benefit of thousands of games having been played before with similar situations to calculate what is the better approach, i.e., what has the highest probability of success. 

 

And better is better.  Period.

I think some fans confuse "better" with "guaranteed."

 

Analytics is NOT about calculating the winning play, it's about giving yourself the highest probability of success with a given play.  It can still go badly though.  

 

A coach can CORRECTLY choose to go for it on 4th down 10 times in a row, and not get the first down EVERY TIME and still be correct in his decision making (if the circumstances support that)....and he still had a better shot of winning the games by going for it all those times on 4th down than he would have by punting, even though they ALL went against him.  

 

Before arguing about that, think about it for a bit.

 

 

I agree with this but as I point out above teams are building their own models that balance the data over thousands of games with their players. Because the problem with the thousands of games argument is that there is an inbuilt bias. Until the last maybe 2 or 3 seasons the only teams who regularly went for 4th downs beyond 4th and inches were ones with good offenses and top quarterbacks and so they converted at a higher rate. That is the one thing clever teams are trying to balance against. If you have Justin Herbert or Josh Allen or Patrick Mahomes everything past your own 45 is potential 4 down territory (especially if it is 4th and less than 5). If you have Andy Dalton or Jared Goff that math is not the same. They will not convert at the same rate. 

[GoBills808]

3 hours ago, Cash said:

That’s technically true. But if your decisions keep leading to bad outcomes, how can you be sure they were good/sound decisions in the first place?

 

Not trying to accuse you of anything, but a lot of the analytics folks I’ve seen are deeply arrogant in a way that makes me question all of their conclusions. Their attitude seems to roughly be, “I’ve completely figured it out, therefore anyone who disagrees with my model is an idiot.” And every time they’re wrong, they come back with logic similar to the quote above. 
 

I have 2 main problems with these folks, plus a bonus third problem:

 

1.) How do they know their model is accurate? As GB & others pointed out above, the models are based on historical data. But as teams change their behavior, there’s no reason to expect that the models will continue to hold, or be predictive at a useful level. 
 

2.) Many of these folks seem to have no understanding of the concept of error. It’s maddening. I saw an article a few weeks ago, blasting a coach for a 4th down decision. Per the model quotes, the coach’s actual decision resulted in about 46.2% win probability, but the “correct” decision resulted in about 46.5% win probability. But there was no mention of the model’s margin of error. Off the top of my head, I can’t recall any NFL analytics people ever providing a margin of error. This is nuts. 
 

3.) Bonus: Arrogance. As I mentioned above, there’s a lot of “Everyone is dumber than me” energy coming from the current crop of analysts. This rankles me on a personal level, so I’ll admit some bias here. But I also think this is a legitimate issue. It ties in to both of the issues above: if you think you’re the smartest guy in the room, it’s often hard to self-evaluate. Much easier to make fun of “stupid” coaching decisions and tell your detractors that only process matters; never results. Plus, these guys are so high on their own farts that they can’t always interpret what their own models are saying. YardsPerPass pointed out an example on Twitter a while back. If you look at the Football Outsiders predraft QB write up in 2018, they lambaste Josh Allen and glow over Sam Darnold. (Nailed it.) But their own model only gave Allen about a 5% greater chance of busting than Darnold, and I think a pretty similar disparity in likelihood to break out. Based on the writeups, I would’ve expected Allen’s bust chance to be like 80-90%, compared to like 20% for Darnold. 

Calculating things that teams rely upon when making decisions in-game, like win% or EPA, shouldn't have a 'right' or a 'wrong' attached to it. It's just a formula that  informs your decision making. If you want to build a circular swimming pool and you calculate the diameter but then decide you'd prefer a rectangle, your initial measurement isn't wrong. You just didn't use it.

[LeGOATski]

"Kick it early, go for it late." Tried and true. 🙂

 

I think it certainly has gone too far with the amount of coaches going for 4th early in the game. It's really about whether or not you have the momentum. For instance, if your team just easily drove down the field in 10 plays and now it's 4th and goal, then go for it. You also have to consider how good your red zone offense is. However, if your team has struggled to get down the field and now it's 4th and short at the 35, take the field goal or try to pin them deep. If you wanna go for it, how good is your OL and RB? 

 

These decisions are pretty easy if you have your finger on the pulse of the game and your team, but too often we're seeing coaches go for it despite the factors I mentioned being unfavorable at the time. And I assume it's simply because "analytics said so."

[Mikie2times]

If I said you have a 65% chance of doubling your money and a 35% of losing it all, one outcome, like a coin flip, would you say yes? Maybe when your 20, probably not when your 60. That's the root of the problem with some of these guys. Staley looked at the decision as a +3% win probability in a binary way. Where another way to look at it, still from a statistical perspective, is you're electing to create a 17% swing in win% based on one play by going for it. Is that something worth doing midway through the 3rd quarter in a 3 point game? Further, is that worth doing vs a team that really wasn't doing much offensively without the refs help? Last part bringing more of the old school football mind into it. I really would like to think right answer exists in the middle of it all.    

[Generic_Bills_Fan]

Using analytics can’t really be evaluated on an individual play basis…it’s based on the probability of a play being successful but obviously the alternative could still happen.  Over the long term following the analytics should theoretically pick you up a win or two but obviously in a 60/40 should I go for this 4th down type situation you’re still gonna get it wrong a big chunk of the time and fans are gonna second guess you 😂

[GoBills808]

2 minutes ago, KzooMike said:

If I said you have a 65% chance of doubling your money and a 35% of losing it all, one outcome, like a coin flip, would you say yes? Maybe when your 20, probably not when your 60. That's the root of the problem with some of these guys. Staley looked at the decision as a +3% win probability in a binary way. Where another way to look at it, still from a statistical perspective, is you're electing to create a 17% swing in win% based on one play by going for it. Is that something worth doing midway through the 3rd quarter in a 3 point game? Further, is that worth doing vs a team that really wasn't doing much offensively without the refs help? Last part bringing more of the old school football mind into it. I really would like to think right answer exists in the middle of it all.    

So basically capturing 30% EV doesn''t convince you, but you're going to allow yourself to be informed by a difference of 17%? Where is the logic in that?

[Mikie2times]

4 minutes ago, GoBills808 said:

So basically capturing 30% EV doesn''t convince you, but you're going to allow yourself to be informed by a difference of 17%? Where is the logic in that?

In the example I stated, under a singular trial, I don't have enough trials to risk my life savings even with a 30+EV at the age of 60. That would be mathematically the correct decision, but in real life, I'm pretty certain even most statisticians would not take that proposal. In the situation Staley faced, I would rather extend the trials/plays I have with the Raiders because I feel I have the better team. Last thing I would want to do is introduce a huge amount of variance into the equation by doing things like going for it on my own 18 yard line even if it does come with a 3% EV.