Analytics vs. Heuristics-Decision Making in the NFL

[D. L. Hot-Flamethrower]

For years and years, NFL coaches made decisions based almost completely on heuristics. For those unsure of what a heuristic is https://en.wikipedia.org/wiki/Heuristic. Essentially, it is a rule of thumb, or trial and error aiding in decision making. For example, you don't go for it from your own 18 on 4th down, early, and in a close game.

 

Analytics came along and it including looking at your chances of winning a game if you went for it on 4th down much more. Initially, this challenge to the old line approach proved to be correct because coaches had evolved into punting way to often in opponents territory (Think Doug Marrone punting from the 32).

 

I would argue that analytics has been taken too far recently and is being misused by some coaches. I would also argue that some of the current decisions (think Brandon Staley) are just recklessly applied analytics. For one thing, the downside of this type of overly aggressive decision-making, the rewards (keeping the ball) vs. the downside risk (giving the opponents points) cannot possibly be worth it.

 

My current opinion on the situation, which I welcome all of your opinions, is that there is something to be learned from certain analytics. Yes coaches had become too conservative and needing to be pushed to go for it more. But, there is much more to it for consideration by the decision maker. I also do not believe some of these situations coaches go for it(Staley); are using data that accurately represents THIS situation. I question how many times in history some of these situations have occurred to have been studied properly. Apologize for the length. What say you?

[prissythecat]

Not sure how analytics played a role in Staley’s decision to call time out ? Seemed more like a gut feel decision. 

[BillnutinHouston]

I think the answer depends on what you are trying to decide.

 

Mike Schopp, an analytics devotee, would never have selected Josh Allen because the analytics argued (nay, screamed) against it.  The problem is, analytics can't measure or predict heart, commitment and the impact of vastly improved coaching.

 

The outcomes of in-game decisions, I think, are greatly improved by knowing the numbers/odds of doing certain things in certain situations.

 

In short, both analytics and gut have a place in practically every decision, but both have a place at the table.

Edited January 11, 2022 by BillnutinHouston

[GunnerBill]

My big issue with it is that the use of prior success rates creates an artificial slant towards aggression in the models. 

 

Teams with good offenses go for more 4th downs. And because they go for more 4th downs with their good offenses they convert them at a decent rate. That factors into the model going forward. 

 

If every team, including those with bad offenses, went for every 4th and let's say 3 or less, the conversion rate of 4th downs would fall and it would affect what the model says about whether to go for it. 

[D. L. Hot-Flamethrower]

Just now, prissythecat said:

Not sure how analytics played a role in Staley’s decision to call time out ? Seemed more like a gut feel decision. 

Yeah I wasn't thinking about that, more the fourth down from the 18. I believe he said he wanted to get his top run defenders on the field and that was why he called the ill-fated time out.

1 minute ago, GunnerBill said:

My big issue with it is that the use of prior success rates creates an artificial slant towards aggression in the models. 

 

Teams with good offenses go for more 4th downs. And because they go for more 4th downs with their good offenses they convert them at a decent rate. That factors into the model going forward. 

 

If every team, including those with bad offenses, went for every 4th and let's say 3 or less, the conversion rate of 4th downs would fall and it would affect what the model says about whether to go for it. 

I'd also add that if your defense is as soft as a team like San Diego that must be considered.

 

I also don't think that just looking at win probability of a given situation should be the default for a decision.

[Big Turk]

21 minutes ago, D. L. Hot-Flamethrower said:

For years and years, NFL coaches made decisions based almost completely on heuristics. For those unsure of what a heuristic is https://en.wikipedia.org/wiki/Heuristic. Essentially, it is a rule of thumb, or trial and error aiding in decision making. For example, you don't go for it from your own 18 on 4th down, early, and in a close game.

 

Analytics came along and it including looking at your chances of winning a game if you went for it on 4th down much more. Initially, this challenge to the old line approach proved to be correct because coaches had evolved into punting way to often in opponents territory (Think Doug Marrone punting from the 32).

 

I would argue that analytics has been taken too far recently and is being misused by some coaches. I would also argue that some of the current decisions (think Brandon Staley) are just recklessly applied analytics. For one thing, the downside of this type of overly aggressive decision-making, the rewards (keeping the ball) vs. the downside risk (giving the opponents points) cannot possibly be worth it.

 

My current opinion on the situation, which I welcome all of your opinions, is that there is something to be learned from certain analytics. Yes coaches had become too conservative and needing to be pushed to go for it more. But, there is much more to it for consideration by the decision maker. I also do not believe some of these situations coaches go for it(Staley); are using data that accurately represents THIS situation. I question how many times in history some of these situations have occurred to have been studied properly. Apologize for the length. What say you?

 

The problem with only using analytics is the sample size for a team in a given season.  For instance, it MAY statistically be the right move to go for it on 4th down many times, but that is over a huge sample size.  During an individual season, the sample size is extremely small meaning there is a chance it won't work out to those percentages, similar to if you flipped a coin 30 times.  Sometimes you might get 20 heads or 25 heads instead of 15/15.  Over a huge sample size of like 10K flips, it would be very close to 50/50, but over a smaller sample size there is a lot more variability. With teams only going for it on 4th down maybe 20-40 times a year, the variability is much more pronounced. So while it could be the right decision analytically to go for it, it could also cost a coach his job if he has a string of bad luck/variability due to small sample size and the gamble doesn't pay off.

Edited January 11, 2022 by Big Turk

[D. L. Hot-Flamethrower]

https://www.forbes.com/sites/liamfox/2021/08/12/how-the-nfl-uses-analytics-according-to-the-lead-analyst-of-a-super-bowl-champion/?sh=759e4b5424e6

 

To be fair, analytics means much more than just going for it on fourth down of course. This article talks about Doug Pedersen and how his decisions to go for it on 4th down and 2 pointers likely helped their SB run. I think being receptive to new ways of thinking (innovation) can give you a short -term advantage. I also think not giving adequate weight to experience can be deadly.

2 minutes ago, Big Turk said:

 

The problem with only using analytics is the sample size for a team in a given season.  For instance, it MAY statistically be the right move to go for it on 4th down many times, but that is over a huge sample size.  During an individual season, the sample size is extremely small meaning there is a chance it won't work out to those percentages, similar to if you flopped a coin 30 times.  Sometimes you might get 20 heads or 25 heads instead of 15/15.  Over a huge sample size of like 10K flips, it would be very close to 50/50, but over a smaller sample size there is a lot more variability. With teams only going for it on 4th down maybe 20-40 times a year, the variability is much more pronounced. So while it could be the right decision analytically to go for it, it could also cost a coach his job if he has a string of bad luck/variability due to small sample size and the gamble doesn't pay off.

Excellent post! To have a sample size large enough for some of these situations they'd have to include ALL games and all teams otherwise as you say they'd be too small. And, as Gunner pointed out, the teams in question and they're abilities must be considered as well which reduces sample sizes.

[Evian]

The great coaches use instinct. The situations in the NFL are too different in each game to be tied to a certain way of decision making. 

[dave mcbride]

26 minutes ago, D. L. Hot-Flamethrower said:

For years and years, NFL coaches made decisions based almost completely on heuristics. For those unsure of what a heuristic is https://en.wikipedia.org/wiki/Heuristic. Essentially, it is a rule of thumb, or trial and error aiding in decision making. For example, you don't go for it from your own 18 on 4th down, early, and in a close game.

 

Analytics came along and it including looking at your chances of winning a game if you went for it on 4th down much more. Initially, this challenge to the old line approach proved to be correct because coaches had evolved into punting way to often in opponents territory (Think Doug Marrone punting from the 32).

 

I would argue that analytics has been taken too far recently and is being misused by some coaches. I would also argue that some of the current decisions (think Brandon Staley) are just recklessly applied analytics. For one thing, the downside of this type of overly aggressive decision-making, the rewards (keeping the ball) vs. the downside risk (giving the opponents points) cannot possibly be worth it.

 

My current opinion on the situation, which I welcome all of your opinions, is that there is something to be learned from certain analytics. Yes coaches had become too conservative and needing to be pushed to go for it more. But, there is much more to it for consideration by the decision maker. I also do not believe some of these situations coaches go for it(Staley); are using data that accurately represents THIS situation. I question how many times in history some of these situations have occurred to have been studied properly. Apologize for the length. What say you?

It was Gregg Williams who punted from the 32, not Marrone. Just an FYI. Atlanta basically punted from the 32 a couple of weeks ago too, although they took a five-yard penalty to move it back five yards. And of course, they kicked it into the end zone, netting 12 yards.

Edited January 11, 2022 by dave mcbride

[D. L. Hot-Flamethrower]

https://theathletic.com/3030201/2021/12/23/hate-nfl-analytics-based-decisions-the-math-adds-up-more-often-than-you-might-think/

 

Some examples where decisions that were analytics-based may have helped this season.

 

I'm not a pro-analytics or pro-old school. But, I do get a kick out of guys like Bradshaw and Long, everything is black and white with these tools.

3 minutes ago, dave mcbride said:

It was Gregg Williams who punted from the 32, not Marrone. Just an FYI.

Doug must have been close at least, especially in volume.

6 minutes ago, Evian said:

The great coaches use instinct. The situations in the NFL are too different in each game to be tied to a certain way of decision making. 

Yes I agree. I remember Belichick going  for it against Peyton Manning and the Colts years ago. If I remember correctly the score was tied and it was fourth down from about his own 25 and he went and got stopped. His reasoning? We weren't going to stop Peyton tonight!

[BuffaloRebound]

You still gotta go with your gut, but the analytics should still be used in coming to every decision.  The analytics tell you the baseline case, and then the coach has to factor in what’s happening on the field right now.  It’s like tweaking a model.  You need both.  

[dave mcbride]

10 minutes ago, D. L. Hot-Flamethrower said:

https://theathletic.com/3030201/2021/12/23/hate-nfl-analytics-based-decisions-the-math-adds-up-more-often-than-you-might-think/

 

Some examples where decisions that were analytics-based may have helped this season.

 

I'm not a pro-analytics or pro-old school. But, I do get a kick out of guys like Bradshaw and Long, everything is black and white with these tools.

Doug must have been close at least, especially in volume.

Yes I agree. I remember Belichick going  for it against Peyton Manning and the Colts years ago. If I remember correctly the score was tied and it was fourth down from about his own 25 and he went and got stopped. His reasoning? We weren't going to stop Peyton tonight!

The Pats player actually got the first (it was clear on replay) but got a terrible spot from the ref. Belichick couldn't challenge because they were out of timeouts. That was what was famous about that one.

[Matt_In_NH]

34 minutes ago, prissythecat said:

Not sure how analytics played a role in Staley’s decision to call time out ? Seemed more like a gut feel decision. 

4 seconds on the play clock is what I heard...that was a tough situation where both sides were wondering what the other was thinking.   I have not watched it in the real time flow but not sure he deserves all the criticism.

[mannc]

28 minutes ago, Big Turk said:

 

The problem with only using analytics is the sample size for a team in a given season.  For instance, it MAY statistically be the right move to go for it on 4th down many times, but that is over a huge sample size.  During an individual season, the sample size is extremely small meaning there is a chance it won't work out to those percentages, similar to if you flopped a coin 30 times.  Sometimes you might get 20 heads or 25 heads instead of 15/15.  Over a huge sample size of like 10K flips, it would be very close to 50/50, but over a smaller sample size there is a lot more variability. With teams only going for it on 4th down maybe 20-40 times a year, the variability is much more pronounced. So while it could be the right decision analytically to go for it, it could also cost a coach his job if he has a string of bad luck/variability due to small sample size and the gamble doesn't pay off.

This is the real drawback with football analytics: How reliable are numbers upon which you are basing your decisions? And as many have pointed out, every situation is different.  If your defense has been unable to stop a Pat Mahomes, then you’re probably going to be more inclined to go for it on 4th down to keep the ball away from him late in the game.
 

In my view, the best thing about analytics is that it has caused coaches to think about every decision in a fresh light, instead of automatically doing what every other coach has done since time immemorial.  

Here’s one I’d love to see more often: You score a touchdown and on the scoring play or the extra point, the other side gets a 15-yard personal foul that is enforced on the kickoff.  Why don’t you go for a surprise onside kick?  You’re kicking from midfield so the worst that happens is the other team gets it on their own 35…and the chances of recovering a surprise onside kick are very good.  But no one ever does it.

[GoBills808]

I think what you’re calling ‘analytics’ is just the process by which NFL coaches are attempting to capture as much EV (or EPA, W% etc) in a given situation.

 

It’s a well defined mathematical set by which one informs his or her decision making. It really can’t be taken ‘too far’.

[aristocrat]

Staley got reckless in that third quarter 4 down on their 20. That's a punt situation. 

[Mojo44]

There is definitely an over reliance on analytics in football. Several posters of already noted the small sample size in most of the statistics which significantly limits any predictive value. I would also like to see standard deviation measures in football analytics. I’m guessing the standard deviation will be quite high making it even more difficult to predict in a specific situation. A nomothetic approach doesn’t work with a small sample size. In this case an idiographic or situational approach is better. 
 

The only place in sports where analytics seem to work well is in baseball regarding the infield shifts. And this is only because most hitters are unable or unwilling to change how they hit. 
 

also, keep in mind that when a coach makes a decision based on the situation and not on analytics this is not just using his “gut”.  These decisions are usually based on accumulated empirical information gained from experience.
 

 

[Albany,n.y.]

52 minutes ago, BillnutinHouston said:

I think the answer depends on what you are trying to decide.

 

Mike Schopp, an analytics devotee, would never have selected Josh Allen because the analytics argued (nay, screamed) against it.  The problem is, analytics can't measure or predict heart, commitment and the impact of vastly improved coaching.

 

The outcomes of in-game decisions, I think, are greatly improved by knowing the numbers/odds of doing certain things in certain situations.

 

In short, both analytics and gut have a place in practically every decision, but both have a place at the table.

I thought of the draft too when I saw the OP.  The problem with people using analytics in player evaluation is they're applying statistics to performances that are much better understood by looking at film.  The analytics people in the media predicted Josh's future with a bunch of numbers & maybe watching a few clips or remember watching some games.  The Bills watched every play on film, were at at least 3 workouts (Combine, in Wyoming, in Orchard Park), had multiple meetings with him and talked to his coaches.  By watching film or going to a game, they were able to put his numbers in the proper context.  One of the things Beane mentioned after drafting Josh was a game played in horrible weather where Josh's statistics looked bad on paper, but on the field he willed the team to victory.  Analytics without a visual review of the games is not a viable way to scout players.  Unfortunately, the so called draft experts in the media who criticize a high pick without doing the due diligence that the drafting team does will never really get it.  It took Schopp years to figure Josh was the real deal.   

[mannc]

3 minutes ago, aristocrat said:

Staley got reckless in that third quarter 4 down on their 20. That's a punt situation. 

True, but I’m not sure that was an analytics-based decision…

[aristocrat]

4 minutes ago, mannc said:

True, but I’m not sure that was an analytics-based decision…

 

You have an 80 percent chance of converting the fourth down I believe is what I heard from the nerds.