One obvious question charter schools face is whether they actually improve educational outcomes, which is their stated purpose. In the interest of testing this assertion, a number of researchers and organizations have examined educational outcomes for students who attend charter schools.
Center for Research on Education Outcomes
In 2009, the most authoritative study of charter schools was conducted by the Center for Research on Education Outcomes (CREDO) at Stanford University. The report is the first detailed national assessment of charter schools. It analyzed 70% of the nation's students attending charter schools and compared the academic progress of those students with that of demographically matched students in nearby public schools. The report found that 17% of charter schools reported academic gains that were significantly better than traditional public schools; 46% showed no difference from public schools; and 37% were significantly worse than their traditional public school counterparts. The authors of the report considering this a "sobering" finding about the quality of charter schools in the U.S. Charter schools showed a significantly greater variation in quality as compared with the more standardized public schools with many falling below public school performances and a few exceeding them significantly. Results vary for various demographics with Black and Hispanic children not doing as well as they would in public schools, but with children from poverty backgrounds, students learning English, and brighter students doing better; average students do poorer. While the obvious solution to the widely varying quality of charter schools would be to close those who perform below the level of public schools, this is hard to accomplish in practice as even a poor school has its supporters.
Criticism and debate
Stanford economist Caroline Hoxby criticized the study, resulting in a written debate with the authors. She originally argued the study "contains a serious statistical mistake that causes a negative bias in its estimate of how charter schools affect achievement," but after CREDO countered the remarks, saying Hoxby's "memo is riddled with serious errors" Hoxby revised her original criticism. The debate ended with a written "Finale" by CREDO that rebuts both Hoxby's original and revised criticism.
National Bureau of Economic Research study
In 2004, the National Bureau of Economic Research found data that suggested Charter Schools increase competition in a given jurisdiction, thus improving the quality of traditional public schools (noncharters) in the area. Using end-of-year test scores for grades three through eight from North Carolina's state testing program, researchers found that charter school competition raised the composite test scores in district schools, even though the students leaving district schools for the charters tended to have above average test scores. The introduction of charter schools in the state caused an approximate one percent increase in the score, which constitutes about one quarter of the average yearly growth. The gain was roughly two to five times greater than the gain from decreasing the student-faculty ratio by 1. This research could partially explain how other studies have found a small significant difference in comparing educational outcomes between charter and traditional public schools. It may be that in some cases, charter schools actually improve other public schools by raising educational standards in the area.
American Federation of Teachers study
A study performed by the American Federation of Teachers, which "strongly supports charter schools", found that students attending charter schools tied to school boards do not fare any better or worse statistically in reading and math scores than students attending public schools. This study was conducted as part of the National Assessment of Educational Progress in 2003. The study included a sample of 6000 4th grade pupils and was the first national comparison of test scores among children in charter schools and regular public schools. Rod Paige, the U.S. Secretary of Education from 2001 to 2005, issued a statement saying (among other things) that, "according to the authors of the data the Times cites, differences between charter and regular public schools in achievement test scores vanish when examined by race or ethnicity." Additionally, a number of prominent research experts called into question the usefulness of the findings and the interpretation of the data in an advertisement funded by a pro-charter group. Harvard economist Caroline Hoxby also criticized the report and the sample data, saying "An analysis of charter schools that is statistically meaningful requires larger numbers of students."
Caroline Hoxby studies
A 2000 paper by Caroline Hoxby found that charter school students do better than public school students, although this advantage was found only "among white non-Hispanics, males, and students who have a parent with at least a high school degree". Hoxby released a follow up paper in 2004 with Jonah Rockoff, Assistant Professor of Economics and Finance at the Columbia Graduate School of Business, claiming to have again found that charter school students do better than public school students. This second study compared charter school students "to the schools that their students would most likely otherwise attend: the nearest regular public school with a similar racial composition." It reported that the students in charter schools performed better in both math and reading. It also reported that the longer the charter school had been in operation, the more favorably its students compared.
Criticism
The paper was the subject of controversy in 2005 when Princeton assistant professor Jesse Rothstein was unable to replicate her results. Hoxby's methodology in this study has also been criticized, arguing that Hoxby's "assessment of school outcomes is based on the share of students who are proficient at reading or math but not the average test score of the students. That’s like knowing the poverty rate but not the average income of a community—useful but incomplete." How representative the study is has also been criticized, as the study is only of students in Chicago.
Learning gains studies
A common approach in peer reviewed academic journals is to compare the learning gains of individual students in charter schools to their gains when they were in traditional public schools. Thus, in effect, each student acts as his/her own control to assess the impact of charter schools. A few selected examples of this work find that charter schools on average outperform the traditional public schools that supplied students, at least after the charter school had been in operation for a few years. At the same time, there appears to be a wide variation in the effectiveness of individual charter schools.
Meta-analyses
A report issued by the National Alliance for Public Charter Schools, released in July 2005 and updated in October 2006, looks at twenty-six studies that make some attempt to look at change over time in charter school student or school performance. Twelve of these find that overall gains in charter schools were larger than other public schools; four find charter schools’ gains higher in certain significant categories of schools, such as elementary schools, high schools, or schools serving at risk students; six find comparable gains in charter and traditional public schools; and, four find that charter schools’ overall gains lagged behind. The study also looks at whether individual charter schools improve their performance with age (e.g. after overcoming start-up challenges). Of these, five of seven studies find that as charter schools mature, they improve. The other two find no significant differences between older and younger charter schools.
A more recent synthesis of findings conducted by Vanderbilt University indicates that solid conclusions cannot be drawn from the existing studies, due to their methodological shortcomings and conflicting results, and proposes standards for future meta-analyses.
National Center for Education Statistics study
A study released on August 22, 2006 by the National Center for Education Statistics (NCES) found that students in charter schools performed several points worse than students in traditional public schools in both reading and math on the National Assessment of Educational Progress test. Some proponents consider this the best study as they believe by incorporating basic demographic, regional, or school characteristics simultaneously it "... has shown conclusively, through rigorous, replicated, and representative research, whether charter schools boost student achievement ...", while they say that in the AFT study "... estimates of differences between charter schools and traditional public schools are overstated." Critics of this study argue that its demographic controls are highly unreliable, as percentage of students receiving free lunches does not correlate well to poverty levels, and some charter schools don't offer free lunches at all, skewing their apparent demographics towards higher income levels than actually occur.
United States Department of Education study
In its Evaluation of the Public Charter Schools Program: Final Report released in 2003, the U.S. Department of Education found that, in the five case study states, charter schools were out-performed by traditional public schools in meeting state performance standards, but noted: “It is impossible to know from this study whether that is because of the performance of the schools, the prior achievement of the students, or some other factor.”