Is traditional testing failing?

The Economist recently published a brief and pessimistic summary of the IDB Peru study. Their writer looked only at standardized math and reading tests as measures of success, ignoring large improvements in cognitive skills. Or maybe, as Richard Jennings suggests, they didn’t read to the sixth sentence of the study:

Over the weekend, The Economist picked up on a 40-day-old report from the Inter-American Development Bank (IDB), and interpreted it as saying the program was failing in Peru. The anonymous Economist writer summarizes IDB’s findings thus:

GIVING a child a computer [does not] accomplish anything in particular… Peruvians’ test scores remain dismal. Only 13% of seven-year-olds were at the required level in maths and only 30% in reading… the children receiving the computers did not show improvement in maths… reading… motivation, or time devoted to homework or reading.

That’s depressing… I suppose there might be some intangible benefit, but it rather looks like there’s no quantitative gain.

But hang on a minute. It appears that The Economist has missed a key point from the report. IDB’s working paper, entitled Technology and Child Development: Evidence from the One Laptop per Child Program, also says:

The results indicate that the program… translated into substantial increases in use both at school and at home… Some positive effects are found… in general cognitive skills as measured by Raven’s Progressive Matrices, a verbal fluency test and a Coding test.

So, instead of a “disappointing return,” or “not accomplish[ing] anything in particular,” IDB did actually find a measurable benefit.

Could it be that the disparity between test scores and actual measured achievement means that it’s the tests that are lacking, rather than the laptops? It certainly wouldn’t be the first time that academic testing was shown to be seriously wanting.

And is it too much to ask for The Economist’s journalists and fact-checkers to actually get as far as the sixth sentence in the report’s abstract, before writing the story? I know that many of today’s workers exhibit short attention-spans, but really!

Too right. Unlike traditional standardized tests, cognitive studies tests are more likely to demonstrate the sort of growth envisioned by constructionist approaches to learning: exploration, empowered learning, and creativity.

There were three separate cognitive skills tests measured by the IDB. All four showed large improvements – equivalent to 5-6 months of cognitive development. As the report notes, “these are sizable effects under this metric considering that the treatment group had an average exposure of 15 months” — that’s like getting an extra 3 years of cognitive development by the start of highschool; something worth further study.

IADB studies OLPC in Peru

The Inter American Development Bank recently published the results of a study of the Peruvian schools that received OLPCs in rural primary schools in Peru, over the first 15 months of the program.

The methodology of the study was quite good, with a randomized study of over 300 schools.  But the measurements and focus were not aligned with the goals of Peru’s program, and there is no clear way to compare these results with the other detailed results available from Plan Ceibal’s program in Uruguay.  The after-analysis of their work has tended to focus on short-term math and reading results, whereas the goals of the program were access to knowledge, improvements in pedagogy, and access to computing – which might be expected to show up in the short term only in the abstract cognitive results.

The measured improvement in abstract thinking – roughly 5 additional months of cognitive development, over a 15-month period – is tremendous. It is interesting to note how this result is downplayed in parts of the world where schools live by less abstract standardized testing.

Some recent comments from OLPC staff and implementers, paraphrased for brevity:

Claudia Urrea:

‘The OLPC program in Peru, or any other place, has to be evaluated according to its initial goal. “math, language, and cognitive test results” showed outputs, but have no clear connection to Peru’s 2007 stated objectives, which targeted pedagogical training and application.’

Oscar Becerra, who oversaw OLPC in Peru’s government:

‘We succeeded in giving access to technology to 100% (220,000) of children and teachers at one-teacher schools, who otherwise would have had no opportunity to use ICT.  Most had the option to take laptops home with them.’

Oscar has published other comments that are a good representation of the OLPC perspective.