NASHVILLE, June 19, 2008 –
Adding a new twist to the sometimes raucous debate over the best way to
“rank” business schools, an economist from the Vanderbilt Owen Graduate
School of Management has come up with a novel approach to measure the
“inherent quality of an institution.” The result of this experiment is
part new ranking and part commentary on the shortcomings of all
rankings…including his own admittedly “tongue-in-cheek” version.
Owen School professor Mike Shor – a noted expert in game theory – has
developed a rogue methodology that tries to go beyond ranking a
school’s performance based primarily on the quality of the students who
choose to attend it. Instead, he has attempted to quantify the “market
value” that each school contributes to its student body overall.
Using the widely accepted “predictive criteria” of starting salary and
average GPA/GMAT scores from the top 50 schools in the 2009 US News & World Report ranking,
Shor tried to eliminate the statistical problem known as the
“self-selection bias,” or the reality that admitted students will
almost always choose to go to a higher ranked school and possibly
distorting the rankings outcome. Calculating expected data points for
the “average” B-school within the US News Top
50, Shor re-ranked schools based on their performance above or below
the average to identify which schools actually provide the greatest
relative increase in expected earnings after graduation.
Shor’s unorthodox analysis was conducted blindly (and in the middle of
the night), with the best and worst performers including (with
Owen/Vanderbilt coming in at #6):
Top 5
Focusing on outputs versus inputs
According to Shor, focusing on the concept of market value is akin
to the distinction between corporations who recruit the best talent
versus those who actually do a better job of guiding that talent toward
its potential. “My admittedly half-baked methodology
tries to measure the efficacy of a school with its existing talent
pool, so students have a clearer indication of which schools are more
likely to increase their market value down the road,” he says.
The genesis of Shor’s new methodology idea came from another
research project he was contemplating: how best to determine and
measure “outcomes” from hospitals conducting bone marrow transplants.
“The fact that some hospitals only accept cases with a high chance of
success created the same sort of self-selection bias that afflict the
B-school rankings and make apples-to-apples assessment challenging,”
adds Shor.
He notes that existing rankings inherently reward schools that admit
students with high credentials, but believes his approach can make such
admissions behavior a negative. “If two schools have
the same market outcome in terms of starting salary, the school that
started with a lower-credentialed class should actually be rewarded for
making the most increase in the market value of the student,” Shor
posits. “It’s a direct indicator of the educational value that the
school is providing to the students.”
In the end, Shor acknowledges that the methodology resulting from
his academic tinkering is less than perfect, but does serve to
highlight the shortcomings of B-school rankings overall.
“There really is no magic formula for determining which school is
better than another, and starting salary is only one possible
statistical measure of outcomes,” he says. “The rankings are certainly
here to stay, but my aim is to further the discussion over what
constitutes an appropriate analysis when it comes to judging schools.
”Shor’s complete “re-ranking” of the Top 50 schools from US News using the publicly available data, along with his witty methodology, disclaimers and rankings commentary, is available on his blog.
