FIGURE 2. CALIPERS Problem: Rescuing Injured Skiers
fundamental ways. One of the largest
efforts to pilot new forms of technology-based assessment is the Problem Solving
in Technology-Rich Environments (TRE)
project. It was launched in spring 2003,
when a nationally representative sample
of 2,000 students participated in a study
to explore how information technology
could be incorporated into NAEP. The
goal was to create scenarios that would
simulate real-world problem solving.
The TRE scenarios test scientific
inquiry skills such as the ability to find
information about a given topic, judge
what information is relevant, plan and
conduct experiments, monitor one’s
efforts, organize and interpret results,
and communicate a coherent interpretation. In one component, 8th graders
used a simulated helium balloon to solve
problems of increasing complexity about
relationships among buoyancy, mass,
and volume. For example, the students
were asked to determine the relationship
between payload mass and balloon altitude. To solve the problem, students
gathered evidence by running simulated
experiments using a variety of different
payload masses. Once they had enough
evidence, they submitted their conclusions using both open-ended and
multiple-choice responses.
The TRE approach demonstrates
Source: From Calipers: Simulation-based assessments, by SRI International, 2006-2007, Menlo Park, CA:
Author. Copyright © 2006 by SRI International. Reprinted with permission. Available: http://calipers.sri.com
/ assessments.html.
Test takers get a chance to choose from multiple options and explain their choice.
several unique capabilities of technology-enabled assessments. They can
offer more complex, multistep problems
for students to solve. In addition,
multiple forms of media, such as the
animated helium balloon and instrument panels in the TRE simulation, can
present information in more useful and
compelling ways than text alone.
Finally, technology-enabled assessments can present tasks based on
complex data sets in ways that even
elementary school students can use. In
the TRE problem-solving exercise, for
example, students see both visual and
graphical representations showing what
happens to the balloon during each
experiment. Figure 1 (p. 49) shows a
graph generated from the results of an
8th grader’s simulated experiments.
Tracking the Learning Process
The problems in Technology-Rich Environments can be dynamic, presenting
new information and challenges on the
basis of a student’s actions. This enables
students to take different approaches
and even test multiple solutions. Moreover, databases can record descriptive
data about the strategies students used
and the actions they took.
In the simulation exercise, for
instance, every student action—which
experiments students ran, which buttons
they pushed, and what values they chose
and in what order—is logged into a
database. The database also records
when the student took each of these
actions.
The quality of students’ experimental
design choice is evaluated using a set of
rules and then scored using statistical
frameworks. These algorithms are linked
across multiple skills, enabling instructors to evaluate students on the basis of
multiple points of evidence. And
because each of the component skills
can be traced back to observable student
actions, instructors can gather detailed
evidence to determine why a student
