Organizational Success Since 2017

[JGMcD2]

I started to discuss some of this in another thread about what folks consider a successful draft. Nobody really had a great way to measure success in the draft outside of their perception of a player.. I wanted to make an attempt at examining this objectively. Pro Football Reference has their Weighted Approximate Value which assigns a value to a player based on their performance. It's not an all encompassing stat like WAR is in baseball, it definitely has its flaws, but PFR said it's steady to use to measure draft success. I'm going to dive into the results below on the draft, as well as some analysis I have done of the FA signings. Nothing is really over the top, I'm going to take some feedback and try to refine this. I had to do a lot of it by hand in excel and couldn't just scrape everything because different pieces were all over.

 

I chose 2017 specifically because that is when Sean McDermott and Brandon Beane arrived. Some folks want to argue that Beane shouldn't get credit for that draft and FA, which is perfectly fine. I am in the camp that believes McDermott and Beane are in lockstep, they make up the upper management of the football department, therefore the decisions made since 2017 have been made with certain goals in mind. I don't feel the need to omit 2017 because it was scrambled, it's very clear these two are close and the decisions made were made by McBeane in some way, shape or form starting in 2017. 

 

"Sometimes, for example if you want to assess a trade or determine the top draft classes of all time, you need a metric that is capable of comparing players across positions and eras. In baseball and basketball, lots of stats have been cooked up to do this, and they can do so with a reasonable degree of precision. In football, no such stat exists. In most cases, people use "starter" or "number of years as a starter" or "number of pro bowls" as the metric when they have to compare across positions.

AV is intended to be an improvement over those metrics, and nothing more. It is not Not NOT an ubermetric whose purpose is to decide once and for all who the best players in NFL history were." - Doug from PFR

 

The career AV is computed by summing 
100 percent of the AV of his best season,
95 percent of the AV of his next-best season,
90 percent of the AV of his third-best season,
and so on 

 

You will also see DrAV, which is just the Weighted Career Approximate Value but solely for the team that drafted them, so we won't get credit for Wyatt Teller performing well in CLE because it's not helping the Bills. Using 2019 may be a little premature but that's the case across the board for every team. I omitted 2020 because they don't have those draft values... this will be a fun exercise to conduct even further down the road when we have some more clarity on how McBeane's picks pan out. But to give us an idea of what things look like right now in comparison to the rest of the NFL... Here we go..

 

 

If you look at the McBeane drafts (2017-2019) sorted by DrAV you will see that the Bills have been a top 5 drafting team in the NFL since McBeane arrived. The Colts, Ravens, and 49ers are all ranked above us, with New Orleans right below. This lines up fairly well with what a lot of posters have said with those teams being very good drafters, but the Bills are right up there with them. This will obviously fluctuate the longer players play in the NFL, but that also includes them being retained by their drafting team in order to continue to provide value. Based on what I saw we could actually rise because of the great season Josh is having - we're actually top 5 while including Josh's below league average performance in his first two seasons. This could also drop with the inclusion of the 2020 Draft, but as of right now the first 3 McBeane drafts look like they went pretty well when compared to their peers.

 

I should note that 5 teams should probably be removed from the discussion because their GMs came in after 2017. I'm not including BAL and GB in that number because DeCosta and Gutenkunst were the top lieutenants in each city and are an extension of Newsome and Thompson who are both still heavily involved with their organizations. The teams that should probably be removed to fairly recognize the new GMs that came after 2017 are the NYJ, NYG, WSH, OAK and CLE. Regardless, McBeane was been at the top of the league as of the end of the 2019 season. 

 

I also dove into free agency, this took a lot longer to put together and could probably use some feedback. My methodology was taking all free agents signed between 2017 and 2019 and applying each player's Weighted Approximate Value to the team once they were signed. I will walk through the chart below to give you a better understanding of what I found.

 

 

The far left column shows the team. The next column is the total Weighted Approximate Value that each organization has acquired via free agency since 2017. The Bills are highlighted there and they actually have the most AV acquired via free agency in the entire NFL since 2017, they're the only team over 200 AV. If I were to just add up the AV that has been brought to Buffalo since 2017 the Bills would be far and away the best with 380 with only NO (335) and IND (311) clearing the 300 AV mark. 

 

I don't necessarily thing that's the best way to break things down though. Each team spent a different amount of money to acquire those players and brought in a different number of players as well. I like to use $/WAR in baseball to see the best player value, so I tried to use something similar here with with $/AV. The Bills are still above average here but they're around 12th in the NFL. The Rams, Seahawks and Patriots pace the NFL in this category, which makes a lot of sense the way the teams have been constructed in recent years. 

 

The only team that ranks higher than the Bills in both DrAV and $/AV in FA is Indianapolis. New Orleans is very close as well, as they're one spot behind us in DrAV. Overall it looks like some teams are good at one or the other, or just plain bad at both. Nobody outside of Indianapolis, New Orleans and Buffalo has had substantial success in both the draft and FA since 2017. Teams like SEA, LAC and PIT seem to be in the next tier. It's a little bit of validation for some of the top organizations in the NFL.

 

I think most can agree IND, NO, PIT and SEA are well run. Buffalo is right up there with them, if not better based on this analysis. There are undoubtedly flaws, but this is a little bit closer to being objective then some people just FEELING that certain teams are so much better run than the Bills.

 

I'm open to thoughts and feedback, feel free to rip me apart... as long as you bring some facts and a thoughtful argument 😀

[Rew]

Great work pulling this together. I'm not sure that AV is the perfect metric here, but I can't think of anything better (which is sort of the problem).  

 

A missing piece to this, and much more challenging to track, would be the inclusion of trades and some way of valuing based on draft location.  If I understand your ranking correctly, any team that trades away picks will show up lower in the ranking and any team that habitually amasses draft picks should do better.  One suggestion would be to average a teams AV per round and then rank based on round performance.  This would also be controversial though, because it would rank 2 average players the same as drafting an all pro and bust.  In that case, what is best?  Someone who is risk averse would prefer the 2 decent players to just 1 blue chip player.  I think the blue chip hit rate is likely going to be more correlated with team success.

[JohnRVA]

Thanks for pulling this together. This passes the eye test IMHO. We had so little talent when they arrived that we might have kept more draft pics than usual and maybe that helps our average?

[Don Otreply]

Nice work! Thank you, 

 

This info does fly in the face of the fire Beane / McDermott sucks posters here, but that should come as no surprise, they are the hot take drama queens of our corner of the football world, again thanks for your work to present this, It does gives a reasonable perspective on our decision makers. 👍

 

Go Bills!!!

[JGMcD2]

55 minutes ago, Rew said:

Great work pulling this together. I'm not sure that AV is the perfect metric here, but I can't think of anything better (which is sort of the problem).  

 

A missing piece to this, and much more challenging to track, would be the inclusion of trades and some way of valuing based on draft location.  If I understand your ranking correctly, any team that trades away picks will show up lower in the ranking and any team that habitually amasses draft picks should do better.  One suggestion would be to average a teams AV per round and then rank based on round performance.  This would also be controversial though, because it would rank 2 average players the same as drafting an all pro and bust.  In that case, what is best?  Someone who is risk averse would prefer the 2 decent players to just 1 blue chip player.  I think the blue chip hit rate is likely going to be more correlated with team success.

By no means do I think it’s perfect, I wish there was something better than AV, but it’s hard or else I start comparing apples to oranges... this is at least apples to apples. I was glad some of the findings at least passed the eye test with teams like NE doing pretty well in FA but not the draft recently... or IND doing really well in both, they’re just missing a QB to make them truly elite IMO. 
 

Yeah, you’re 100% right. Draft pick trades/trades in general is probably the next step along with smoothing this all around the edges as well. 
 

The FA stuff was time consuming and pick trades are pretty intertwined, so it would take a bit... but I’ll probably take a crack at it. I’m also not sure the best way to value that, I’d have to look into it and think on it. 
 

I could probably do what you’re suggesting and just find the AV over ~20+ years for rounds 1-7 and calculate the NET for each team from 2017-2019. Maybe? Idk I have to think deeper. 
 

Your point you bring up at the end is a really good one... I’m used to discussing it through the lens of baseball. My personal feeling in baseball is in Rounds 1-2 you should be drafting a low variance college player and then supplementing the low variance with high risk in future rounds. 
 

Similar concept here, do I take a Josh Allen RD1 and then to safe in later rounds? Do I take a Saquon Barkley (pretty low variance guy) and then go someone high risk in RD2-3? 
 

A lot to chew on... thanks for the response! 

53 minutes ago, JohnRVA said:

Thanks for pulling this together. This passes the eye test IMHO. We had so little talent when they arrived that we might have kept more draft pics than usual and maybe that helps our average?

I see what you’re saying, yeah that very well could be the case. Probably best to relate it all to the # of picks like suggested above. Although just off the top of my head I feel like we haven’t made more than 7-8 picks in a given draft, and trades a bunch away to move up for guys like Josh, Tremaine, Ford, Dawkins, etc. 

Edited December 2, 2020 by JGMcD2

[JGMcD2]

29 minutes ago, Don Otreply said:

Nice work! Thank you, 

 

This info does fly in the face of the fire Beane / McDermott sucks posters here, but that should come as no surprise, they are the hot take drama queens of our corner of the football world, again thanks for your work to present this, It does gives a reasonable perspective on our decision makers. 👍

 

Go Bills!!!

You got it... just trying to do something a little different. Opinions are great but are often skewed... it’s hard for any one to see all 32 teams and evaluate every player outside of a glance here and there. 

This isn’t perfect by any means but at least it kind of casts a wide net, and relates it to the entire league. 

[Don Otreply]

30 minutes ago, JGMcD2 said:

You got it... just trying to do something a little different. Opinions are great but are often skewed... it’s hard for any one to see all 32 teams and evaluate every player outside of a glance here and there. 

This isn’t perfect by any means but at least it kind of casts a wide net, and relates it to the entire league. 

Being that all the teams are evaluated the same way it makes comparisons league wide equal, and as such valuable. 👍

[ToGoGo]

Great stuff. This sort of data is tough to nail down, so I appreciate you finding something within the vicinity of the bullseye. 

 

The analytics back up what those with a "good eye" can see, which is that McBeane always seem to make the correct, savvy move. The perfect mix of patience, long-term strategy, foundation first, faith, and shrewdness (a good shrewdness). 

 

That kind of decision-making is not based in analytics, believe it or not. It is based in intuitive-thinking, with analytics simply being a supplementary source. A nice combination of right brain/left brain thinking. Rare, but wherever it is found, the people who have it are at the top of their professions. The most competent bunch. 

 

I'd like to play Beane in a chess match one day. I have a feeling he would be good at it. 

[elijah]

Not much to comment on here, but as a data freak I wanted to give props for this and really enjoyed the work 🍻

[JGMcD2]

17 minutes ago, elijah said:

Not much to comment on here, but as a data freak I wanted to give props for this and really enjoyed the work 🍻

I appreciate the kind words! 

[H2o]

So are we firing Beane or not? It's hard to figure it out reading the posts around here. 

Edited December 2, 2020 by H2o

[DrDawkinstein]

 

 

But for real, thanks for putting all this data together. I'll have to re-re-read it again tonight to digest it.

[GunnerBill]

They have completely ripped down a .500 football team and built one on course to win double digit games for the second consecutive season by year 4. I'd be amazed if the data didn't support the idea that they'd done a good job. The Bills are one of the best run football operations in the league at the moment. I know that is hard for people to get used to after years of dysfunction and ineptitude but they are. All I hear from my main source into organisations in the league is that whoever he speaks to in other organisations there is huge respect for what McDermott, Beane and their staffs have done. When these 4 or 5 complete rebuilds happen this offseason there will be teams saying "the Bills are a model to look at." 

[Big Turk]

Very nice....and this won't even include the Diggs trade since he wasn't an FA but has added a huge amount of AV for this year...

 

I think sometimes it helps to see things in perspective compared to other teams when we might be upset that player X from round Z hasn't produced more, but then you start looking  around the  NFL and realize that we have done really well overall...

[Big Turk]

Also, in their entire draft class Edmunds is 6th in Career AV and Allen is 10th...pretty impressive. And that is before Allen's year this year which might catapult him to #1.

[Big Turk]

A lot better than the Bills 2010 class where the highest Career AV is Arthur Moats taken in the 6th round with 21...most of that coming with Pittsburgh after leaving the Bills. His AV with the Bills was 9.  What a terrible class that was...actually shocked to see that it was worse in terms of career AV than the amazingly bad 2000 draft class which was highlighted by Sammy Morris' 28, again, most of which after leaving Buffalo.

 

However, came across this, listing the worst draft classes of all time using DrAV from 2017...

 

Bills 2000 class ranks as 15th worst in NFL history(the only Bills draft class on the list) with a Total DrAV of 13...the 2010 class was good enough not to make the list. Actually...now that I look at it more closely, they are only taking the worst for each team and how they compare to each other....

 

https://www.espn.com/nfl/story/_/id/19092546/stacking-worst-classes-nfl-draft-history

Edited December 2, 2020 by matter2003

[foreboding]

I like it, also it is not subjective which is cool. It looks at data and not surprisingly the best teams are all in there.
Thanks for the effort.

[JGMcD2]

2 hours ago, matter2003 said:

Also, in their entire draft class Edmunds is 6th in Career AV and Allen is 10th...pretty impressive. And that is before Allen's year this year which might catapult him to #1.

Yeah I thought it was pretty encouraging to see that the drafts over that 3 year time period looks solid without Josh really being valued as the star he has started to become. 

[ComradeKayAdams]

On 12/1/2020 at 7:52 PM, JGMcD2 said:

I'm open to thoughts and feedback, feel free to rip me apart... as long as you bring some facts and a thoughtful argument 😀

 

Excellent work, JGMcD2!!! I am a stats nerd as well, so we are kindred spirits. PFR is such an amazing website and one of my top three football sources, right up there with the NFL’s home page and this place, of course. If you’re looking for a single metric for football player evaluation, PFR’s Career AV is as good as any that I’ve seen out there so far.

 

We agree that McDermott and Beane are one entity (McDeane!) united in football philosophy, so I’m comfortable considering Buffalo’s 2017 draft to be Beane’s even though it was technically Whaley’s. Supposedly McDermott was consulting with Beane and with some of the Carolina scouts leading up to the 2017 draft weekend? I have no idea if that’s true or just a bad internet rumor. That seems somewhat unethical and treasonous, actually…

 

I’m sorry that I don’t have any novel quantitative analysis insight for you at the moment. When it comes to evaluating front office abilities using the Career AV metric, I believe that draft picks should be normalized by selection order and free agents should be normalized by cap space used. Trade evaluation should maybe incorporate the NFL’s historical Career AV average at the particular draft number of any draft pick that is involved in the trade.

 

As everyone knows, NFL general managers are ultimately judged by regular season records and by playoff achievements. So now take a look at the current top 8 NFL teams by record: Pittsburgh, KC, New Orleans, Green Bay, Seattle, Tennessee, Cleveland, and Buffalo. The first 5 are led by QB’s that are likely Hall of Famers and who have already won Super Bowls. The Titans are getting high-quality QB play and are led on offense by a future Hall of Fame RB. Cleveland is somewhat of an anomaly because they are 1-3 against teams with a record above 4-7. Allen is playing at an elite level. The Bucs and Colts are not too far behind in the standings, either, with their Hall of Fame-caliber QB’s. So the QB position appears to be unusually important (duh…). That is why I’d consider placing an extra numerical weighting factor on the QB position when doing any sort of quantitative front office evaluation. Maybe reward those who drafted and groomed their own elite QB, as opposed to the front office regimes that inherited their guy?

 

My final comment is for any avid Beane skeptics. Remember the common refrain that it takes three full seasons to make a fair evaluation of an NFL draft. Mr. Beane has placed a special emphasis on the defensive front 7 since 2018, using 4 of his 9 top-100 picks on them (Edmunds, Phillips, Oliver, Epenesa). Injuries, many new faces on Frazier’s D, and COVID-19 disturbances may have slowed their development trajectories quite a bit this year, unfortunately, but look for their Career AV stock to begin rising quickly! My expectation is that McDeane will be judged very favorably in time, using whichever metric you prefer: PFR AV, wins, championships, etc.

[JGMcD2]

I did some further digging and made some adjustments based on some feedback. I wanted to find a way to compare things based on round, but I think using historical pick values is tricky because each draft is so much different in terms of positional value and player talent. I decided it was best to compare each draft to itself, because each GM was selecting from the same pool of talent.

 

What I did was found the average value for a player drafted in each round (1-7) in each year (2017-2019) and the calculated what I am calling the Net Drafted Accumulated Value (NETDrAV) for each pick in each round. I only compared each draft to itself. I then found the Total Net Drafted Accumulated Value (TOT_NETDrAV) for each team in each draft and ranked them against each other. Rather than just looking at how much raw value the Bills brought in as compared to the 31 other teams, this gives an idea of how much extra value they extracted in each round as compared to the 31 other teams in the league. 

 

The results are attached here below. 

 

 

I feel most comfortable with the 2017 draft class because we've had 3 years to see every player perform. The Bills are highlighted in yellow in each graph. As you can see in 2017 the Saints absolutely crushed it. This is the Lattimore, Ramczyk, Williams, Kamara, Anazalone, Hendrickson draft that many pundits have been touting as one of the best draft classes of the modern era. Obviously that shows up in the results here, as they nearly double the next best draft in terms of AV. Highlighted in yellow there are the Buffalo Bills. They're currently #3 by a slim margin, but can be expected to continue to remain towards the top of the draft class. Overall, it tells us exactly what we knew already, it's a strong draft class. Now we know exactly how it compares to the rest of the league in terms of the value they extracted from the draft. I will go into further detail later in this post as I've broken down the value by each round for each team as well. 

 

Moving over to 2018, which is lead by the Indianapolis Colts who brought in guys like Quentin Nelson, Darius Leonard, Braden Smith and Nyheim Hines. The Ravens also did very well for themselves in this draft with Lamar, Orlando Brown and Mark Andrews amongst others. You can see the Bills highlighted in yellow again. They rank just outside the top 5 at 6 this time. This draft for Buffalo is boosted by getting both Josh Allen and Tremaine Edmunds in RD1, but they also get some value from Taron Johnson in the 5th round as well. Again, compared to their peers the Bills extracted some of the best value out of this draft. 

 

In 2019 the Bills move back into the top 5. This is the hardest year to judge and will fluctuate the most in the coming years, but after 1 season the Bills seems to have done a solid job here when compared to the rest of the league. Oliver was positive in NETDrAV as was Ford, Singletary, Knox and Daryl Johnson. Again, this is all just in comparison to the average player in the round that year. The Bills are extracting more value in each round than 27 other teams in the league, which is pretty impressive. 

 

This image below shows the NETDrAV ranking for each team in each draft. Buffalo is highlighted in yellow there for you to see. They are the ONLY team to finish in the top 10 in NETDrAV between 2017-2019, and very nearly finish in the top 5 in all three drafts, just missing in 2018 at #6. This obviously WILL change over time as seasons progress. 2017 will be the most stable, the Bills may jump or fall a place or two but will likely remain in the top 5. 2018 has the potential for major growth because of Josh, they could leap into the top 3 there if he continue the path he is on. 2019 is going to fluctuate the most because some players may pop more than other moving forward, rookie seasons come with a learning curve, but early returns look good in comparison to the rest of the league.