These pieces originally appeared as a weekly column entitled “Lessons” in The New York Times between 1999 and 2003.
[ THIS ARTICLE FIRST APPEARED IN THE NEW YORK TIMES ON MAY 9, 2001 ]
Flaws in voucher findings
By Richard Rothstein
With education high on the national agenda, some scholars have been tempted to publish sensational but premature conclusions. These can exploit policy makers’ inability to distinguish simplistic analyses from those that stand up to closer examination.
This was apparently the case with an analysis of Florida’s voucher program, published in February by the conservative Manhattan Institute. The author, Jay P. Greene, a former University of Texas professor and now a Harvard researcher, concluded that Florida scores showed schools “rising to the challenge of vouchers.” On the eve of Congressional debate over President Bush’s plan to give students at low-performing schools federal money for private school tuition vouchers, Dr. Greene announced that Mr. Bush’s proposal would work as well.
“That’s not a theory,” Dr. Greene stated, “but proven fact.”
In Florida, schools are rated A, B, C, D or F. If a school gets a second F in four years, its students can get state vouchers for private school tuition. Dr. Greene concluded that this makes public schools improve.
His method was creative. He reasoned that schools in the bottom of the D category were very similar to those in the top of the F category. He showed that after failing one time, higher-scoring F schools posted greater gains than lower-scoring D schools. Because these schools were otherwise alike, Dr. Greene stated that a threat of vouchers must have made F schools improve more rapidly.
His claims garnered extensive press coverage, but his report made no mention of having been submitted, before publication, to experts expected to express caution or even doubt about the results. Such review is normal for academic studies where complex statistics or extraordinary claims are involved.
Had Dr. Greene’s report undergone this process, nonpartisan scholars might well have suggested an obvious natural experiment to test his conclusions. Other states also rate schools, but threaten nonvoucher interventions for failure. If in these states, schools labeled failures improved rapidly when vouchers were not threatened, it would suggest that vouchers did not cause Florida’s more recent gains.
Indeed, before 1999, Florida itself already grouped schools by test scores, but students in the lowest category did not get vouchers. Douglas Harris, an economist at the liberal Economic Policy Institute, analyzed the earlier Florida data. Dr. Harris found that before 1999, higher-scoring schools in the failing group also gained more than lower-scoring schools in the next group. The subsequent voucher policy apparently had no added effect.
Others have tested this elsewhere. At Duke University, Prof. Helen F. Ladd and Elizabeth J. Glennie, a researcher, examined North Carolina’s accountability plan. There, schools are ranked “exemplary,” “meets expectations,” “no recognition” or “low performing.” Low-performing schools are not given vouchers; instead, state experts intervene to improve the schools.
Dr. Ladd and Dr. Glennie found that as in Florida, higher-scoring schools in the bottom group gained more than lower-scoring schools in the next group. The researchers concluded that Dr. Greene’s Florida results “probably have little or nothing to do with vouchers.” If vouchers explained Florida gains, “it is unlikely we would have found comparable patterns” in North Carolina.
Texas has a similar system. There, too, students in failing schools do not get vouchers, but state experts intervene.
Scholars at the Charles A. Dana Center at the University of Texas have frequently defended the state’s system against partisan attacks by critics of President Bush when he was governor. So it is noteworthy that a Dana Center analyst, Amanda Brownson, challenged Dr. Greene’s conclusions by applying his method to Texas scores.
As in Florida and North Carolina, Ms. Brownson found that higher performing schools in the failing category posted greater gains than lower-performing schools in the next-to-bottom group.
These studies of North Carolina, Texas and earlier Florida scores are being published by the Economic Policy Institute, together with a summary by Martin Carnoy, a Stanford University professor. Dr. Carnoy speculates that these studies, along with Dr. Greene’s, may only show a “scarlet letter” effect. Being termed failing is embarrassing and motivates schools to improve, whether vouchers or other interventions follow.
As education debates get more politicized, special burdens are placed upon scholars and the public. Academics ought to resist the temptation to publish hurried findings, even though exciting results assure easy publicity. And nonscholarly audiences should be more wary of researchers who fail to discipline themselves in this way.
