Nerd Level : 1

Before the 2013 Football season I wrote a couple pieces detailing a quantitative fantasy draft methodology. You can find the details here. In short, I used player stat projections aggregated by fantasypros.com with some adjustments to quantify each player's value. With this season past and in the morgue, I'm going to perform a postmortem: slice up 2013's fantasy football results and find out whether building a drafting system was worth the trouble.

I'm feeling pretty good about the model since all three of my fantasy teams were first or second this year. But three teams is a very small sample size and a ton of other factors that could effect that success (like luck or obsessive waiver wire watching). This article takes a more robust look at whether it was a waste of time.

The punch line is that a draft methodology based on aggregating expert ranks added between 4 and 7 points per game over just using ESPN's Average Draft Position to determine picks. That is statistically significant.

I broke this article in to Nerd Levels: increasing degrees of method detail and quantitative complexity. So you can take my word that this method adds value, stop reading, and check in before next football season or move on to Level 2.

Nerd Level : 2

To quantify the points added by my method, I ran a mock draft for a fictional 12 team leage where 6 teams pick according to ESPN's pre-season average draft position (ADP) and 6 teams pick the the player with the best expert consensus value according to my methodology. I'll refer to the two types as ADP and Value Based going forward. Once the rosters were set, I calculated how many points each team would have scored over the sytem

If you're unusually interested, intrepid, or bored you can find more detail on the draft methodology here. You can also find the R code that gathers and analyzes the data on my GitHub repo.

Broad Results

To evaluate the whether Value Based was better than ADP, I want to know how many more points per week a Value Based team would have score than an ADP team, on average. I used two methods to translate the fantasy points each draft strategy's players scored in to a weekly point difference.

  1. Average the weekly points of the players at each position for each strategy. Multiply by the number of starters at each position. With this calculation, the Value Based teams averaged 4.4 points per week more than the ADP teams.
  2. Same as method one, but instead of averaging the points of all players, only average the points of players that would have started. This way, the Value Based teams averaged 6.2 points per week more than the ADP teams.

The second method throws out the worst outliers from each strategy's draft, so flukes like a player getting injured for the entire season in week 1 will not overly effect the results. This method is also more foregiving of high risk / high reward picks that don't pan out.

With 6.2 points per week instead of 4.4, method two is more flattering of the Value Based strategy. The main reason for the difference is Josh Freeman. The Value Based system drafted him (unfortunately). His almost non-existant season is included in method one but thrown out of method two, because there's no way he would have started. More on that later.

It is possible that the success of Value Based approach was luck, but it seems unlikely. This table summarizes the stastical significance of the results for each calculation method.

Weekly Points of Value Based over ADP
Method 95% Confidence Interval Probability
Points > 0
Low Mean High
(1) All Players -0.8 4.4 9.6 95.6%
(2) Starters Only 1.6 6.2 10.8 99.7%

The probability that the Value Based approach added positive points over ADP is greater than 95% for both calculation methods. 95% is usually good enough for scholarly articles, so it's good enough for me.

Nerd Level : 3

There are alot of additional questions to answer. How are the extra points being generated? What did the Value Based method do right? Where did it screw up? Breaking down the points added by position adds some insights.

The weak spot in the Value Based drafting is highlighted by the red in the table above: QBs. With only starters included, I added positive value at QB, but it's still the smallest of any position. What happened?

The visualization below helps answer that question by providing more granularity. Each bar is a draft pick, arranged by draft order from left to right. The size of each bar indicates how well each pick turned out, measured by points scored above replacement. I define points above replacement as the number of fantasy points a player scored over the worst player that would have started at that position in a 12 team league. The color indicates whether they were drafted by a Value Based team (my system) or an ADP based team.

The chart is interactive. Hover over any bar and to see more information about the player it represents. Use the buttons at the top to highlight only certain positions. Zoom in and out by clicking and dragging on the small chart at the bottom.

If you filter on QB's, you can see that the two biggest bars are black, indicating that the ADP teams drafted them (damn!). Hover over to see that those bars are Peyton Manning and Drew Brees.

The Value Based method missed the MVFP (most valuable fantasy player) and MVRP (most valuable reality player) Peyton Manning. I also missed the 3MVFP (3rd most valuable fantasy player) Drew Brees (my finance background makes awkward acronyms second nature). Those misses hurt my QB cred.

But the next 10 or so QBs drafted all produced about the same value. I waited until late to draft those QBs, while ADP took them early. That earned back a bit of my QB cred.

The most negative QB bar, Josh Freeman, was drafted by my Value Based method. That pick accounts for the QB improvement in the "Starters Only" method. Providing almost no points at QB, the highest point scoring position, he was by far the worst pick of the draft. Still given he was a late round pick drafted as a back up QB, I think excluding him gives a better assesment of the system's overall value.

The Value Based picking made up for the QB deficiencies at the other positions. It picked the most top 3 RB's, Jamal Charles, LeSean McCoy, and Matt Forte. It picked six of the top eight WR's, in particular picking the most valueable WR, Josh Gordon, in the 9th round.

Nerd Level : 4

The number of points garnered at each position can be broken down in to two sources: Position Preference and Player Choice.

(1) Position Preference

Position Preference is the value derived from drafting a position early or late. For example, if I draft running backs early I should get more points from running backs than my opposition, regardless of how good I am at choosing between running backs. But, I give up points at other positions by going after RBs aggressively.

However, I can add value throughout my roster by drafting a position early if the skill generally drops off fastest at that position. In other words, if the marginal value of drafting early at that position is greater than the other positions. For example, if the points scored by the top QB's drop off faster than the points scored by the top WR's, I can add value by drafting QB's earlier in general.

To calcualte the difference in points between the two draft strategies attributable to position preference, you need to know (A) where each strategy drafted a particular position on average (Mean Draft Position) and (B) how many points it is worth to draft a player one pick earlier (Marginal Points).

Calculating the Mean Draft Position for each strategy and position is straightforward, just take the mean of all the draft picks. It tells you how early a strategy type picked that position on average.

I used a linear regression of each player's Points Scored versus their Draft Position to estimate the Marginal Points for each position. High Marginal Points indicates there is more value in drafting early at that position. To be really robust, I probably should have used data from multiple seasons to get more data points from the regression. But since I work full time, have a baby, and am not getting paid to do this, I'm going to favor expediency over statistical integrity this once.

This table summarizes the results. It was most important to pick QB's early (they had the highest Marginal Points). Unfortunately, the Value Based system did not do that. That results in the big red bar you see in the table indicating that I gave up a lot of value by not drafting QB's early enough.

Generally, this big negative should be offset by a positive somewhere. If I wasn't picking QB's I was picking something else. But because I was picking RBs, which had less marginal value than QB's, the Value Based strategy gave up some points on average by not picking QBs as early as the ADP strategy.

In total the Value Based system lost 0.58 points by not going after QB's aggressively enough early.

(2) Player Choice

Player Choice is the value from drafting better players with the same pick as your opponents. For example, if I use the 13th pick on a better RB than the RB my opponent got with the 12th pick, I am adding value via Player Choice. In total, this is the number of points that cannot be explained by Position Preference.

The points from Player Choice is the residual of the linear regression used to calculate the Marginal Value for the Position Preference. Drafting a player that ends up scoring more than expected of that draft pick adds positive Player Choice points.

This table summarizes the Player Choice points for each position. The Value Based approach beat the ADP approach at each position.

QB was again the weakest for missing Peyton Manning and Drew Brees, but was still positive. The value was added by picking QB's in the 5th, 6th, and 7th round that actually did better than several QB's taken before.The Value Based approach chose better RB's in the early rounds (got the top 3), chose better WR's in the late rounds (particularly Josh Gordon), and chose better TE's in the late rounds (Jordan Cameron and Martellus Bennett).

Over all players, the Value Based approach added 4.99 points of value. 4.99 points of Player Choice minus 0.58 from Position Preference equals the 4.42 points total value added number (with some rounding error).

Best Picks

The best picks of any draft are the players that outperform the usual expectations of a pick from the same area of the draft. In the real world, Tom Brady, drafted 199th, was a better draft pick than Peyton Manning, drafted 1st, regardless of any arguments of which player is better. In the fantasy world, drafting Andy Dalton with the 122nd pick was a better pick than Drew Brees with 13th, even though he scored 80 fewer points on the season. The 13th pick is on average worth over 80 points more than the 122nd pick.

This table summarizes the best picks of the draft. I use the same calculation to evaluate the "best picks" as I do to calculate Player Choice above. This table demonstrates that great picks can come from anywhere in the draft, so it is just as important to pay attention to the bottom of the draft as the top. In fact the highest scoring WR was Josh Gordon who went with the 98th pick in this mock draft. Looking at this table, it's pretty clear why my fantasy football teams succeeded this year. All of my teams had Matt Forte and Josh Gordon, two of the top six most successful picks.


Lessons

A systematic Value Based method seems to consistently beat just drafting according to the average draft position. It doesn't always make the best choice, missing out on gigantic years by Peyton Manning and Drew Brees are obvious exceptions, but it consistently adds value across positions and throughout the draft.

This postmortem encourages me to try this methodology again next year, maybe with some added wrinkles. This year I only used the average projection in ranking players. Information about the uncertainty of projections might add value. For example, if the distribution of expert projections are all over the place, that might suggest the player is a risky pick and I can downgrade the rank accordingly.

Better benchmarking might be another area for improvement. Comparing players within positions is pretty easy. If Adrian Peterson is projected to have more points than Arian Foster, take Adrian Peterson. But choosing BETWEEN positions is difficult.

To decide whether a player at one position is better than another, the projected points scored must be adjusted to make player values comparable across positions. This year I used the worst player I might end up with at a position as the benchmark, but that can be improved.

A more dynamic benchmark might use the best player at this position I could get next round if I refrain from drafting this player this round. The intuition is, if the first 10 wide receivers are all projected about the same, I don't want to draft the first wide receiver. It makes sense to take another position and wait until next round or later to take a wide receiver.

This dynamic benchmark would have to live-update throughout the draft, though you could get an idea of what might happen ahead of time using pre-draft ADP numbers.

---

I'm always happy to get any feedback, positive or negative... either way it means someone's reading this. You can reach me at ross@vividnumeral.com. Again, you can find the source code on GitHub. The R code for the number crunching is in the rross repo. The html, javascript, and css for this page is in the vividnumeral repo. I used d3 and underscore to make the visualizations in this article.