Jump to content

Proposed Draft Evaluation Methodology...


cage

Recommended Posts

In this day of measuring everything about every sport, I'm kind of surprised that there isn't a definitive benchmark of team by team data analysis? When we talk about how well the Steelers, Ravens, or Patriots drafts it all seems subjective. There doesn't seem to be a way to scale what the difference is between the best teams and the worst teams... is it 10% or 200%??

 

So here's a method I came up with. Please feel free to suggest modifications/refinements/critiques. Once we innovate this a bit, I'll look back at several drafts and see if we can grade everyone by this methodology

 

Methodology starts in year #3 and each year can be modified based on ongoing results of the players drafted.

 

Players are divided into 4 categories with Base points for each type of player

-- PRO/Pro Bowler 5pts (selected for at least 3 Pro Bowls)

-- STR/Starter 3pts (at least 32 starts, 2 full years)

-- BUP/Back-up 1pt (on NFL roster for at least 48 games, 3 full years)

-- WSH/Washout 0pts (out of the NFL by 3rd season)

 

In addition each round of the draft is scaled per expectations

-- Rd #1, Starters and Pro Bowlers expected, otherwise negative penalty, including -3 for a washout

-- Rd #2, Starters still expected but scaling back penalty for misses

-- Rd #3, Starters requiring greater skill and scaling back penalty for misses

-- Rd #4, Solid players expected, with extra points for starters/pro bowlers

-- Rd #5, Solid players rewarded

-- Rd #6, Solid players expected, no extra points for Pro Bowlers (ie Tom Brady type picks)

-- Rd #7, No penalties, just solid players get a 1 pt bonus

 

Round Drafted

1 2 3 4 5 6 7

PRO 0 1 2 3 1 0 0

STR 0 1 2 3 2 1 0

BUP -1 -1 -1 0 1 1 1

WSH -3 -2 -1 0 0 0 0

 

Have at it...

Link to comment
Share on other sites

I don't get it. If a team drafts a Pro Bowl player in the 7th round, they don't get any extra credit for that?

 

No, by then you're not expecting to draft Pro Bowlers. You pretty much got lucky. Tom Brady is the classic example. Pro Bowler w/ a 6th round pick. If the Patriots had any idea that he could even be a legitimate starting quality QB in this league (much less what he's become) they would have grabbed him by the 3rd or 4th round. I don't want to reward them extra (beyond the 5pts for a Pro Bowler) because a Pro Bowler in the late rounds is luck, not skill

 

So explaining PRO... in the first round that's what's expected so no bonus... rd 2-4, there's increasing skill in being able to nail a pro-bowler in the top half of the draft. After that you're starting to get more lucky than skillful, so I don't want to reward for that.

 

I'm trying to match up results with reasonable expectations for a given round

Edited by cage
Link to comment
Share on other sites

I would say its luck, but I would also say its skill. Reason being is that even if they feel he is their next starting QB and has great potential, they arent going to draft him in the 3rd round just because they like him. They have to judge it on his draft value. That may have been the QB they were eyeing the whole time. If they know that no one else is watching him then they certainly arent going to waste a 3rd rounder on him. Whether he is worth it or not

Link to comment
Share on other sites

It's not bad, it fails to reflect the dynamic whereby you wait until a later round to take a guy you want simply b/c you anticipate nobody else will take him though...the scaling down (or up depending on the way you look at it) skews that but it's not bad the way you have it.

Link to comment
Share on other sites

Not sure I understand "you got lucky" in the grading scheme. Day 2 (now 3) and the UDFA market are where the scouts have to do their hardest work. That's where you're finding small college guys who are undervalued and taking good risks on somebody with a proclaimed "issue" (injury/attitude/size etc). Teams should in fact be lauded for unearthing good players in the later rounds.

 

I also disagree that it's the expectation that a 6th rounder turns out to be a solid pro. I expect a guy who is about replacement level, maybe a little worse (think Keith Ellison) - a guy who plays hard for you but is not going to be expected to do anything more than contribute on STs.

 

Where luck comes in for me is when a guy like Brady turns out to be more than a benchwarmer by proving himself due to what seems at the time like an unfortunate injury. When the sixth rounder just makes the team and starts to excel, I think that's the mark of good drafting.

Link to comment
Share on other sites

For the bills the strategy the best approach might be this...

 

Are we a playoff team....no

Where are we bad...defense mostly, average offense.

What positions are good kickers, Kyle Williams, some D-backs,

What positions don't actually suck? Running back, D-backs, receivers, qb

What sucks and dooms our team on the field on sunday? Inability to stop the run, no pass rush and poor pass blocking.

 

How can we get better in the areas where we suck through the draft?

 

Draft a killer pass rusher. Draft a run stopper Draft a great and aggressive Outside linebacker This should help with the pass rush and stopping the run. Getting the d-backs into position to make interceptions and get the offense on the field.

 

Will AJ Green do any of these things? NO

 

Offense...Get two tackles through free agency or draft. Get a good tight end who can block and catch passes (Luke Stocker?).

 

Can AJ Green pass block? NO Can AJ Green play tight end? NO.

 

hmmmm....under this strategy AJ is pretty much a pass or a possible trade down to get three or four of the players we really need. St. Louis would DIE for AJ Green.

Evebn at a minimum of a trade with St. Louis at 14 for a first, second and a second next year would be kind of interesting.

 

That would give us possibly Costanza or Von Miller as a first choice, Luke Stocker as our secone pick, and a very good Defensive End, OLB or OT prospect with St. Louis second pick.

Link to comment
Share on other sites

In this day of measuring everything about every sport, I'm kind of surprised that there isn't a definitive benchmark of team by team data analysis? When we talk about how well the Steelers, Ravens, or Patriots drafts it all seems subjective. There doesn't seem to be a way to scale what the difference is between the best teams and the worst teams... is it 10% or 200%??

 

So here's a method I came up with. Please feel free to suggest modifications/refinements/critiques. Once we innovate this a bit, I'll look back at several drafts and see if we can grade everyone by this methodology

 

Methodology starts in year #3 and each year can be modified based on ongoing results of the players drafted.

 

Players are divided into 4 categories with Base points for each type of player

-- PRO/Pro Bowler 5pts (selected for at least 3 Pro Bowls)

-- STR/Starter 3pts (at least 32 starts, 2 full years)

-- BUP/Back-up 1pt (on NFL roster for at least 48 games, 3 full years)

-- WSH/Washout 0pts (out of the NFL by 3rd season)

 

In addition each round of the draft is scaled per expectations

-- Rd #1, Starters and Pro Bowlers expected, otherwise negative penalty, including -3 for a washout

-- Rd #2, Starters still expected but scaling back penalty for misses

-- Rd #3, Starters requiring greater skill and scaling back penalty for misses

-- Rd #4, Solid players expected, with extra points for starters/pro bowlers

-- Rd #5, Solid players rewarded

-- Rd #6, Solid players expected, no extra points for Pro Bowlers (ie Tom Brady type picks)

-- Rd #7, No penalties, just solid players get a 1 pt bonus

 

Round Drafted

1 2 3 4 5 6 7

PRO 0 1 2 3 1 0 0

STR 0 1 2 3 2 1 0

BUP -1 -1 -1 0 1 1 1

WSH -3 -2 -1 0 0 0 0

 

Have at it...

 

I like the idea you have here, but I don't think the grading system is that great of a representation of what you are trying to say. Part of the problem you are going to have is how to weight the production of the players by round. Does a starter in the first round equal a primary backup in the second? The third? Is there a difference between a backup who never sees the field and a player who plays in sub packages only? How do role players fit?

 

I suspect that you will see a slightly skewed bell curve with a higher number of teams consistantly drafting poorly and a very small number (if any) who consistantly draft well. I think the first thing taht needs to be accepted for this is that there are probably no teams, over a 10 year period for instance, that have had 10 good drafts. Therein lies the problem witht his exercise: it's really easy to pick out the teams that draft poorly, but hard to decide who drafts well and to then quantify it.

 

One team might draft a bunch of really good, near elite players, but can't find a role player or second or third tier player to save their life. Other teams might only be able to find the second and third tier players, but not elite players to save their lives (ehem, cough *Buffalo Bills* cough). How many third tier players equal an elite player? All these questions are important, but unfortunately don't go anywhere near the real question of how good a team is at drafting because none of these things deal with wins and losses. If a team hits on a few (5-7) elite players and then a whole load of third tier players and wins the SB with a predominantly self-drafted team, I would say they are much better than a team that found 10-15 elite players, but can't get out of the first round of the playoffs.

 

I think in the end, the majority of teams are too close to explicitly quantify and rank their drafting against other teams because of all these differences and how they relate to winning. I'd bet that you'd find that the good drafting teams tend to get players who fit into their system (or a system is designed around them) effectively.

Link to comment
Share on other sites

I really like the idea but I think this math will work a little better...

 

1,2,3,4,4,4,4

-1,0,1,2,3,3,3

-2,-1,0,1,2,2,2

-3,-2,-1,0,0,0,0

 

-Gleeb

 

I'm struggling with this one... I'll address each of the four lines. Remember, you're already getting base points for each level PRO=5 STR=3 BUP=1 WS=0

 

With the table I want to make adjustments based on how good the draft team was against "expectations" (in quotes). My expectation is that:

Round 1 pick should be a pro-bowler so why give bonus points

Round 2/3/4, several players have already been picked so it takes increasing skill to select a PRO, so I'm giving bonus pts

Round 5/6/7, by now you're "expectation" is that you're drafting for depth/special teams, so getting a PRO here is really unexpected and so I'm taking away the bonus points, they still get the base points. You got luckier than you were skillful.

 

I'll go back to the Tom Brady example. I have trouble calling the Brady pick a selection of supreme skill. Which is what you're system would classify it as. I'm happy to say the Pats knew what they were doing when they picked Seymour, Mayo, Wilfork, Warren, Gronosky,... but it seems that if they really knew that Brady would be so good, they would have grabbed him earlier with the fear that someone else would pick him. You can't really leave a starting QB sitting around till the 6th round and expect he'll be there

 

My main issue with your math is that teams are getting double-rewarded for getting hits in the late rounds. They get the points for the postion and the bonus. Again I'm trying to evaluate against expectations at that point in the draft.

 

I like the idea you have here, but I don't think the grading system is that great of a representation of what you are trying to say. Part of the problem you are going to have is how to weight the production of the players by round. Does a starter in the first round equal a primary backup in the second? The third? Is there a difference between a backup who never sees the field and a player who plays in sub packages only? How do role players fit?

 

I suspect that you will see a slightly skewed bell curve with a higher number of teams consistantly drafting poorly and a very small number (if any) who consistantly draft well. I think the first thing taht needs to be accepted for this is that there are probably no teams, over a 10 year period for instance, that have had 10 good drafts. Therein lies the problem witht his exercise: it's really easy to pick out the teams that draft poorly, but hard to decide who drafts well and to then quantify it.

 

One team might draft a bunch of really good, near elite players, but can't find a role player or second or third tier player to save their life. Other teams might only be able to find the second and third tier players, but not elite players to save their lives (ehem, cough *Buffalo Bills* cough). How many third tier players equal an elite player? All these questions are important, but unfortunately don't go anywhere near the real question of how good a team is at drafting because none of these things deal with wins and losses. If a team hits on a few (5-7) elite players and then a whole load of third tier players and wins the SB with a predominantly self-drafted team, I would say they are much better than a team that found 10-15 elite players, but can't get out of the first round of the playoffs.

 

I think in the end, the majority of teams are too close to explicitly quantify and rank their drafting against other teams because of all these differences and how they relate to winning. I'd bet that you'd find that the good drafting teams tend to get players who fit into their system (or a system is designed around them) effectively.

 

Paragraph 1... I think I'm accounting for differences in rounds through the bonus/demerit system. You make a good point that it doesn't account for role players. A great example of that is Kevin Faulk, who's been invaluable to the Pats for a decade, but might not be rated high enough on this

 

Paragraph 2... I don't know how this would look yet, I'm trying to refine the method and then see how it applies. Ultimately my thought is to measure 10 years worth of drafts and then see how it moves over time, can do moving averages to show trends and even measure it by GM/Coach regimes. How good was Donahoe compared to Marv compared eventually to Buddy Nix

 

I've made some additional modifications to the method to account for Trading UP/DOWN... for example the Bills should be penalized extra for having traded up for John McCargo and JP Losman. Those busts are more expensive because they gave up additional choices to get them.

 

I've also added a category called HOF (potential Hall of Famer... Brady, Manning, Ray Lewis) who there should be higher points for overall.

 

We are doing some evaluation at DraftTek. We're up to Round 4 for each team. See what you think. We had each Analyst submit commentary, a grade, and points based on round, then had the head of DraftTek pasteurize the scores a little so that one wasn't too severe a grader.

 

Astro

 

Thx,.. I'll look at this more thoroughly, but my first impression is that you need more explicitly quantitative measure of the categories. Here's where I am now:

 

HOF = 8pt = Potential Hall of Famer

PRO = 5pt = Selected to a minimum of 3 Pro Bowls

STR = 3pt = At least 32 career starts (2 full years)

BUP = 1pt = On NFL roster for at least 64 games (4 full years)

WSH = 0pt = Less than 64 games on NFL roster

INJ = 0pt = Less than 48 games on NFL roster due to injury

 

Had to add the potential Hall of Famer category. There's a difference between Manning, Brady, Ray Lewis, Troy Palamalu and someone like Matt Hasselbeck or Carson Palmer who may also be 3 time pro bowlers

 

Expanded BUP and WSH to 4 years as guys like John McCargo,... could be rated too high at 3 years. Most first round picks will last 3 years unless they're the most colossal of busts. Ryan Leaf may even have passed the 3 year level

 

Created INJ category which basically removes the penalties in the per round table if a player washes out due to injury. For example, the Ravens selected Sergio Kindle last year, who may never be able to play because of a fall the left him w/ a cracked skull. I don't want to over-penalize the Ravens for that, because it is beyond their control.

Link to comment
Share on other sites

I'm struggling with this one... I'll address each of the four lines. Remember, you're already getting base points for each level PRO=5 STR=3 BUP=1 WS=0

 

With the table I want to make adjustments based on how good the draft team was against "expectations" (in quotes). My expectation is that:

Round 1 pick should be a pro-bowler so why give bonus points

Round 2/3/4, several players have already been picked so it takes increasing skill to select a PRO, so I'm giving bonus pts

Round 5/6/7, by now you're "expectation" is that you're drafting for depth/special teams, so getting a PRO here is really unexpected and so I'm taking away the bonus points, they still get the base points. You got luckier than you were skillful.

 

I'll go back to the Tom Brady example. I have trouble calling the Brady pick a selection of supreme skill. Which is what you're system would classify it as. I'm happy to say the Pats knew what they were doing when they picked Seymour, Mayo, Wilfork, Warren, Gronosky,... but it seems that if they really knew that Brady would be so good, they would have grabbed him earlier with the fear that someone else would pick him. You can't really leave a starting QB sitting around till the 6th round and expect he'll be there

 

My main issue with your math is that teams are getting double-rewarded for getting hits in the late rounds. They get the points for the postion and the bonus. Again I'm trying to evaluate against expectations at that point in the draft.

 

 

 

Paragraph 1... I think I'm accounting for differences in rounds through the bonus/demerit system. You make a good point that it doesn't account for role players. A great example of that is Kevin Faulk, who's been invaluable to the Pats for a decade, but might not be rated high enough on this

 

Paragraph 2... I don't know how this would look yet, I'm trying to refine the method and then see how it applies. Ultimately my thought is to measure 10 years worth of drafts and then see how it moves over time, can do moving averages to show trends and even measure it by GM/Coach regimes. How good was Donahoe compared to Marv compared eventually to Buddy Nix

 

I've made some additional modifications to the method to account for Trading UP/DOWN... for example the Bills should be penalized extra for having traded up for John McCargo and JP Losman. Those busts are more expensive because they gave up additional choices to get them.

 

I've also added a category called HOF (potential Hall of Famer... Brady, Manning, Ray Lewis) who there should be higher points for overall.

 

 

 

Thx,.. I'll look at this more thoroughly, but my first impression is that you need more explicitly quantitative measure of the categories. Here's where I am now:

 

HOF = 8pt = Potential Hall of Famer

PRO = 5pt = Selected to a minimum of 3 Pro Bowls

STR = 3pt = At least 32 career starts (2 full years)

BUP = 1pt = On NFL roster for at least 64 games (4 full years)

WSH = 0pt = Less than 64 games on NFL roster

INJ = 0pt = Less than 48 games on NFL roster due to injury

 

Had to add the potential Hall of Famer category. There's a difference between Manning, Brady, Ray Lewis, Troy Palamalu and someone like Matt Hasselbeck or Carson Palmer who may also be 3 time pro bowlers

 

Expanded BUP and WSH to 4 years as guys like John McCargo,... could be rated too high at 3 years. Most first round picks will last 3 years unless they're the most colossal of busts. Ryan Leaf may even have passed the 3 year level

 

Created INJ category which basically removes the penalties in the per round table if a player washes out due to injury. For example, the Ravens selected Sergio Kindle last year, who may never be able to play because of a fall the left him w/ a cracked skull. I don't want to over-penalize the Ravens for that, because it is beyond their control.

 

I guess I didn't quite comprehend the bonus points part. My table was assuming there were no bonus points, just points given from the table.

 

Also, I hear what you're saying about being lucky in the later rounds, but similar to a pass that bounced off a receiver's hands, intercepted and returned for a TD, luck should be rewarded. It won't skew the numbers too much because there maybe one or two 3x probowlers drafted in the later rounds and maybe a few dozen 2 of 3 year starters out there. Also, if I'm not mistaken, Brady doesn't even meet your criteria of making the probowl his first 3 years in the league.

Link to comment
Share on other sites

I guess I didn't quite comprehend the bonus points part. My table was assuming there were no bonus points, just points given from the table.

 

Also, I hear what you're saying about being lucky in the later rounds, but similar to a pass that bounced off a receiver's hands, intercepted and returned for a TD, luck should be rewarded. It won't skew the numbers too much because there maybe one or two 3x probowlers drafted in the later rounds and maybe a few dozen 2 of 3 year starters out there. Also, if I'm not mistaken, Brady doesn't even meet your criteria of making the probowl his first 3 years in the league.

 

Its gotten a bit fragmented as we've been discussing it. To freshen up the methodology would work like this:

 

For each pick the team gets a base #pts based on the result of that pick as follows:

 

HOF = 8pt = Potential Hall of Famer (ex. Tom Brady, Ray Lewis, Troy Palamalu)

PRO = 5pt = Selected to a minimum of 3 Pro Bowls (ex. Reggie Wayne, Matt Hasselbeck)

STR = 3pt = At least 32 career starts (2 full years) (ex. Kyle Williams, Terrence McGee)

BUP = 1pt = On NFL roster for at least 64 games (4 full years) (ex. George Wilson, Keith Ellison)

WSH = 0pt = Less than 64 games on NFL roster (ex. Aaron Maybin before long)

INJ = 0pt = Less than 48 games on NFL roster due to injury (ex. Kevin Everett)

 

From there I created an adjustment table based on where they were picked. Which looks like this:

 

1 2 3 4 5 6 7 UN

HOF 1 2 1 0 0 0 0 0

PRO 0 1 2 3 1 0 0 0

STR 0 1 2 3 2 1 0 0

BUP -2 -1 -1 0 1 1 1 1

WSH -3 -2 -1 0 0 0 0 0

INJ 0 0 0 0 0 0 0 0

 

Its scaled to what I'm calling expectations:

Round 1 expecting elite players, no bonus for doing well, but penalty for doing poorly

Round 2/3/4, expecting starters, with bonus for elite players

Round 5/6/7, expecting depth players with some bonus for hitting on that, but don't want to over-reward for unexpected hits in late rounds. Intended to measure skill. Hits in late rounds are like winning the lottery, not skill.

 

I realize that luck factors into how good a team becomes, but I'm trying to strip that out in order to measure drafting capability. I'm trying to remove luck form the metric, if possible.

 

So you get both the base points and respective adjustments.

 

The draft can be measure each year and the numbers update each year, so a teams draft might move around between year 3-6 until it stabilizes. There may be late bloomers like Eric Moulds was who was thought to be a bust until he electrified in year 3. He didn't reach a 3rd Pro Bowl until year 6, I believe, so the Bill's score for that draft year would have moved around all those years.

 

The idea is not to measure a single year, but to look at how the scores move around over time. If several years of these scores were available probably 3 or 5 year moving averages would be very predictive of success and really quantify the difference between the capabilities of Front Offices

Edited by cage
Link to comment
Share on other sites

 

Also, I hear what you're saying about being lucky in the later rounds, but similar to a pass that bounced off a receiver's hands, intercepted and returned for a TD, luck should be rewarded. It won't skew the numbers too much because there maybe one or two 3x probowlers drafted in the later rounds and maybe a few dozen 2 of 3 year starters out there. Also, if I'm not mistaken, Brady doesn't even meet your criteria of making the probowl his first 3 years in the league.

 

One more example on the issue of "luck". A QBs passer rating is what it is based on the results in each game. However, when the coaches review film, the re-adjust a real rating to better evaluate the QB based on what should have happened based on his actions. Fitz should get credit for a game winning TD pass in OT against the Steelers despite the fact that its not in his recorded rating as Stevie Johnson dropped the ball. Conversely, I don't know how many passes he threw right into a defenders hands that were dropped INTs. While is official rating stays the same, the coaches use his adjusted rating to evaluate what they really have in him... same thing here.

Link to comment
Share on other sites

An interesting attempt, but I have a BIG problem with awarding points for a starter. This methodology fails as a fair relative measure among all teams.

 

To put my objection as clearly as possible: if the Packers draft a player that becomes a starter on their team, do they deserve (only) the same amount of points as the Bills, if the Bills draft a starter?

 

Weaker teams with weaker pre-draft rosters have an easier time earning those points, while teams with strong rosters have an inherently harder time earning those same points. Example - the Bills could draft Colin Kaepernick and he could plausibly start right away for the Bills, but there's no chance he starts over Aaron Rodgers on the Packers roster. Thus, two teams could select the EXACT SAME PLAYER but the stronger team earns no points while the weaker team earns 3 points.

 

I've always hated the "but, he's a starter so he must have been a good draft pick" analysis. At the end of the day, all that a player's starting status measures is the quality of the roster the guy landed on.

Link to comment
Share on other sites

An interesting attempt, but I have a BIG problem with awarding points for a starter. This methodology fails as a fair relative measure among all teams.

 

To put my objection as clearly as possible: if the Packers draft a player that becomes a starter on their team, do they deserve (only) the same amount of points as the Bills, if the Bills draft a starter?

 

Weaker teams with weaker pre-draft rosters have an easier time earning those points, while teams with strong rosters have an inherently harder time earning those same points. Example - the Bills could draft Colin Kaepernick and he could plausibly start right away for the Bills, but there's no chance he starts over Aaron Rodgers on the Packers roster. Thus, two teams could select the EXACT SAME PLAYER but the stronger team earns no points while the weaker team earns 3 points.

 

I've always hated the "but, he's a starter so he must have been a good draft pick" analysis. At the end of the day, all that a player's starting status measures is the quality of the roster the guy landed on.

 

hmmm,... I have to think about that one a bit as you make a good point. Here's my first reaction:

 

Ultimately this methodology is intended to evaluate a team's draft, not individual players. So over time the cumulation of players that Packers draft over the Bills will not be differentiated by Starters, but by Pro Bowlers and Potential HOFers. Those get the bigger points. Given that the Packers have a young Aaron Rodgers, they won't be drafting QBs in the first 3-4 rounds, but developmental projects in the later rounds. Given that the Bills have QB issues, they would be drafting QBs higher. So this should work itself out when looking at the whole draft (especially over time) as in the first 3-4 rounds where the adjustment table gives bonus or demerit points focuses on teams drafting for their needs. Those players have a higher probability to start.

 

Plus the STR, BUP and WSH don't generally come into use until year's 3-4 after the draft, so by then if a player's good enough they're not riding the bench anymore.

 

How's that??

Link to comment
Share on other sites

We are doing some evaluation at DraftTek. We're up to Round 4 for each team. See what you think. We had each Analyst submit commentary, a grade, and points based on round, then had the head of DraftTek pasteurize the scores a little so that one wasn't too severe a grader.

 

Astro

No offense Astro, but the grading system you are working on his super subjective and all over the place.

 

 

You guys might what to go with a more empirical system like the one that cage has laid out.

 

I like cage's "starts" and maybe for certain skill players you set up statistical goals.

Like 150 carries and 500 yards for a 1st or second round RB

and 50 catches 400 yards for a 1st or second round rookie WR

 

I think Cage is on to something.

Link to comment
Share on other sites

×
×
  • Create New...