The United States usually scores
below average on PISA tests. U.S. politicians and media often uncritically accept
the tests as valid and point to U.S.
schools as being at fault. Such conclusions are wrong on a number of counts.
In the first place, we should question
the good sense of comparing a diverse,
300-million-person nation like the
United States with tiny homogeneous
city-states like Hong Kong and Singapore. In addition to size, other factors
complicate the issues. In Hong Kong,
schools concentrate on English,
Chinese, and mathematics. Proposals to
introduce “liberal studies,” which looks
like critical thinking to me, have stirred
great controversy. In Singapore, schools
serve a relatively small proportion of
low-income students because many low-paying jobs are done by thousands of
Malaysians who enter the country each
day and return home in the evening or
by “guest workers,” mostly from
Indonesia and the Philippines, who
cannot bring their spouses or families.
Those who cite PISA results to criticize the U.S. education system also
ignore a number of characteristics that
keep PISA from being useful for
comparing the quality of schools in
different nations. One problem is the
fact that PISA is administered only to
15-year-olds. Because different nations
start formal schooling at different ages
and have different policies about
students repeating a grade, such a
limited snapshot can hardly tell us
much about a nation’s overall success in
educating students.
Another problem is the design of the
test items. As PISA officials write, “The
assessment focuses on young people’s
ability to use their knowledge and skills
to meet real-life challenges, rather than
merely on the extent to which they have
mastered a specific school curriculum”
(OECD, 2005, p. 12). Because the test
purportedly measures students’ ability
to incorporate information that they
might not have learned in school, PISA’s
design would seem to bias it toward
affluent students whose homes and
families have more resources.
The University of Oslo’s Svein Sjøberg
(2007) points out that PISA’s “
requirement that the text should be more or
less identical [in different countries]
results in rather strange prose in many
languages” (p. 14), and that the translations of at least one PISA item word-for-word from English to Norwegian
rendered it nonsensical. He is quite
skeptical, as am I, that questions can be
rendered free of cultural bias and trans-
Trends in International
Mathematics and Science
Study (TIMSS)
TIMSS comes to us from the International Association for the Evaluation
of Educational Achievement in The
Netherlands, but most of the technical
work is conducted at Boston College. It
measures selected math and science
skills in grades 4 and 8 using short, fact-oriented stems and mostly multiple-choice questions.
We have been through four rounds of
TIMSS: 1995, 1999, 2003, and 2007.
As with PISA, politicians and the public
are quick to use TIMSS results to criti-
Schools under the gun to raise test scores
increasingly rely on strategies that
get immediate, but short-lived, results.
lated into the many languages of PISA
countries and still be the “same” questions. And some of the passages for
science and math questions are so long
and discursive that they obviously
measure reading skills as well.
PISA reports contain the nations’
average score, rank, and proportion of
students reaching various levels of
achievement. Virtually all the media and
political attention goes to the average
scores and ranks. But as Hal Salzman of
the Urban Institute and Lindsay Lowell
of Georgetown University observe
(2008), the students scoring average are
not likely to become national leaders in
their chosen fields. Future innovators
and leaders are more likely to come from
high scorers—and the United States
produces more than twice as many of
these as any other OECD nation does.
The bad news is that the United States
also produces more low scorers than any
other nation except Mexico.
cize the quality of U.S. schools. In his
March 2009 speech to the Hispanic
Chamber of Commerce, President
Obama observed, “In 8th grade math,
we’ve fallen to 9th place.” Ninth place
was indeed the U.S. rank (among 46
nations) for the 2007 TIMSS administration, but in 1995, the United States
ranked 28th out of 41 countries. U.S.
scores as well as ranks have actually
risen for 8th graders, and they have
been stable for 4th graders.
The TIMSS developers explicitly
make a causal connection between high
scores and a country’s economic health
and claim that “there is almost universal
recognition that the effectiveness of a
country’s educational system is a key
element in establishing competitive
advantage in what is an increasingly
global economy” (Mullis, Martin, & Foy,
2008). Even if this were true, the question would be, Does TIMSS measure that
effectiveness? The answer is no. No test
