Pointshare ratings

Subtitle

How well did the rating systems on the Massey comparison site predict the 2011 NCAA basketball tournament?

Using the last ranking (Sunday 13th March 2011) before the tournament from the Massey basketball ranking comparison site, http://www.masseyratings.com/cb/arch/compare2011-18.htm I did a simple analysis of the results.

I deleted the three systems (De Simone, USA today and the AP) which only ranked the top 25 as they didn’t rank most of the teams in the tournament.

 This left 49 systems which I analysed by assuming that the higher ranked team would be predicted to win for each game.

 Of the 67 games in the tournament only 23 games were predicted correctly by all systems while 11 games were not picked by any systems.

Number of systems out of 49 that predicted the winner

Number of games

49

23

48

5

47

4

46

1

45

1

44

1

43

2

41

1

41

1

33

1

32

1

27

1

22

2

19

1

16

1

15

1

13

1

11

2

10

2

7

1

3

2

1

1

0

11

 

The second last row reflects the fact that LMC was the only system that had  Butler ranked above Florida  and so that was the single game which only one system predicted correctly.   The final was picked correctly by all 49 systems. The biggest upset was clearly Morehead vs Louisville which not only nobody selected but Morehead were ranked on average 80.9 places worse than Louisville. The next three biggest were Virginia Commonwealth vs Purdue, Kansas and Georgetown.

In terms of picking winners the best was Ken Pomeroy’s system which predicted 46 although all the systems were closely bunched together with the worst system picking 39.

Wins

System

 

 

46

POM

45

DC

DCI

DOK

LMC

44

AUS

BIH

CNG

DC2

MOR

PEQ

PTS

REW

43

COL

DUN

KMV

MAS

MB

PIG

RTH

RTP

SE

SPW

STH

TMR

WIL

WLK

WOL

42

BOB

BPI

CPA

JEN

KRA

NOL

OMY

PGH

RPI

RTR

SAG

TW

WOB

41

DOL

KOS

KPK

ROG

RT

RTB

SEL

39

CPR

 In contrast the NCAA selection committee went 43-20 excluding the 4 play-in games which were between the same seed. If they had picked the four play-in games they would have had on average two correct and so would have picked 45 correct.

But just picking the winners isn’t the only measure of a ranking system.

To look in more detail at the results for each system I ranked the 68 teams from 1 to 68 according to their rankings in each system. For each game a system gained points if its higher ranked team won and was deducted points if the higher ranked team lost. The number of points added or deducted was simply the difference in ranks between the two teams. I call this Compressed Ranking Weighted Wins.

To put these numbers into context I used the same method with the NCAA selection committee’s seeds and created a pseudo-ranking using the seeds by assuming the number 1 seeds would be ranked 1,2,3 and 4 so I gave all the number one seeds a joint ranking of 2.5. The number twos are jointly ranked 6.5 etc and accounting for the 6 No.16 seeds, the 5 No.11s and the 5 No.12 seeds.

Adding all the points for each system for the 67 games produces the following results.

 

System

Compressed Ranking Weighted Wins

DCI

881

NCAA seed-based pseudo-ranking

859.5

DC

788

WLK

788

PIG

779

PTS

766

PEQ

763

JEN

758

MAS

756

LMC

753

KMV

748

RPI

743

MOR

739

RTR

739

STH

738

CNG

737

TMR

732

AUS

731

CPA

730

DC2

730

RTH

726

DOK

724

REW

723

COL

722

KPK

722

BOB

714

SAG

713

POM

711

SEL

710

WOB

706

BIH

702

PGH

700

WOL

700

BPI

696

WIL

693

DOL

689

SPW

689

MB

686

RTP

685

KRA

683

ROG

682

DUN

679

TW

679

RT

675

RTB

670

KOS

664

SE

662

CPR

651

OMY

650

NOL

649

 

Only one of the 49 systems was better than the NCAA seed based ranking and most were well behind.

So what does this mean?  Well I have taken a lot of liberties, many of the systems are not designed to be predictive which is the test I have applied to them. While  67 games is a tiny sample. In addition given the unique nature of the upsets this year, particularly Virginia commonwealth’s run these may distort the picture and a more “conventional” tournament might discriminate between the systems better. But these restrictions apply equally to the NCAA committee.

I would have preferred if more of the systems had beaten the NCAA and perhaps a more sophisticated analysis might be more insightful but by this analysis the NCAA committee did much better than the almost all of the ranking systems.

 

 

 

 

College football Evaluation

In college football I entered the 10-11 ESPN bowl mania competition where a winner has to be picked for each game and confidence value assigned.  My entry based on Pointshare finished in the 94th percentile. Auburn's win cost the system dearly as Oregon were one of my highest picks.

Members Area

Recent Blog Entries

Recent Podcasts

Recent Forum Posts

No recent posts