Report: Ohio E-Schools Show Superior Results, Data: Academic And Cost Effectiveness

Above was the title of a message posted in one of the iNACOL forums sometime last week.  The complete message read:

From the Ohio Alliance for Public Charter Schools

Newly released state data reveal that e-schools perform better academically
than Big 8 districts at lower cost

http://www.oapcs.org/files/EschoolStudy_final6-24-09.pdf

COLUMBUS – Like charter schools and Big 8 urban schools, Ohio’s e-schools are obtaining gains in student performance despite what the state’s Local Report Card ratings show, according to an analysis of newly released state data by the Ohio Alliance for Public Charter Schools (OAPCS).

The new analysis, together with research released last week by the non-profit, non-partisan organization KidsOhio.org, demonstrates that the state’s Local Report Card, which weights average test scores over actual student gains from year to year, falls short in adequately describing overall school effectiveness – especially for schools serving students with high mobility rates who enter school grade levels behind.
KidsOhio research showed that both charter schools and urban schools on average rank low on static achievement test scores but rank much higher on the state’s “value added” measure of student improvement from year to year.

The KidsOhio report compared all charter schools as a group with the most similar district schools (Big 8 traditional schools) as a group.

“We decided to expand on the KidsOhio report by zooming in on the seven statewide e-schools,” said William Sims, OAPCS president. “Students attending e-schools are among the most challenged in the state. Not only do they typically enter these schools grade levels behind but the schools tend to operate with a very mobile student body. These factors have to be accounted for in accurately measuring their performance.”

The data show that average e-school achievement test scores rank in the 4th percentile of all 617 Ohio districts and statewide e-schools. But when value-added (student growth) scores are ranked, e-schools jump 29 percentile points in the rankings – more than Big 8 urban districts who serve a similar student population.

Currently, a school’s report card rating is heavily weighted on average test scores (measured by the Performance Index), which are highly correlated with incoming student performance. This method of evaluation is not a complete measure of a school’s real academic success.

“Penalizing schools based on average achievement test scores alone while ignoring that they are making significant year-over-year growth gains effectively discriminates against all charter schools and their urban counterparts in district schools,” according to Sims.

Sims indicated that “e-schools, like other charter schools, operate on significantly less funding than their traditional counterparts. In fact, e-schools’ per-pupil tax expenditure is about half the level of Ohio Big 8 schools. Along with distance-learning institutions of higher education, e-schools are at the cusp of revolutionizing education internationally. The bottom line is that e-schools are serving a vital public need; and, they are obtaining results superior on average to Big 8 schools at a fraction of the cost to taxpayers.”

As a researcher, the claim of “superior results” quickly caught my attention so I’ve been looking into it over the past two days.  The report itself is a very brief document that doesn’t provide much in the way of data, although it does provide a number of conclusions.  For example:

• Newly released state data show that Ohio’s Big 8 urban districts and statewide charter e-schools serve similar student populations, both comparatively low in preparation  and achievement, with e-schools having the added disadvantage of serving students with higher mobility rates (rate at which students move among schools).
• The state’s Local Report Card, which weights average test scores over actual student performance gains from year to year, does not adequately describe overall school effectiveness—especially for schools with high mobility rates and whose students enter grade levels behind.
• Both Big 8 urban districts and statewide charter e-schools rank higher when looking at their “value-added” progress over one year rather than simply measuring their one-time testing performance in the spring, as reported on the state’s annual Local Report Card.
• On average, e-school value-added results are significantly higher than Big 8 districts.
• Statewide charter e-schools spend $5,540 (44%) less in taxpayer funds per pupil than the average of Big 8 urban districts.
• On average, e-schools achieve better gains while spending less per pupil than Big 8 urban districts.

However, like most conclusions offered with little data included to support them, it is worth digging a little deeper on these issues.  The report was prepared by the Ohio Alliance for Public Charter Schools and the “non-partisan organization” KidsOhio.org – always love when they indicate the “non-partisan” portion, as it almost always means that they have a particular political slant but want to appear as if they weigh both sides equally.  It was designed to answer three basic research questions:

1. What student populations do these schools serve and how do they differ?
2. What does Ohio’s Local Report Card tell us about their comparative academic achievement and progress?
3. How do their average per-pupil expenditures compare?

The conclusions you draw for these three questions is largely based which report you read.  The conclusions listed above came from the report on the Ohio Alliance for Public Charter Schools website – an organization that obviously has a horse in this race that it wants to see do well.  When you actually try to answer the questions above using both reports, you get a chance to see exactly how selective the Ohio Alliance for Public Charter Schools report is.  As I believe the answer to question 1 is kind of obvious (i.e., e-schools have a population that is more consistent with what one often finds in urban areas, but is not representative of the state average) and not as contentious, let’s examine question 2 a little.

2. What does Ohio’s Local Report Card tell us about their comparative academic achievement and progress?

If you look at the Ohio Alliance for Public Charter Schools report, they concluded:

• The state’s Local Report Card, which weights average test scores over actual student performance gains from year to year, does not adequately describe overall school effectiveness—especially for schools with high mobility rates and whose students enter grade levels behind.
• Both Big 8 urban districts and statewide charter e-schools rank higher when looking at their “value-added” progress over one year rather than simply measuring their one-time testing performance in the spring, as reported on the state’s annual Local Report Card.
• On average, e-school value-added results are significantly higher than Big 8 districts.

There was one main graph and one main chart that was included in this section to support these conclusions.  One the other hand, KidsOhio.org concluded:

• Many of Ohio’s lowest-rated schools serving some of Ohio’s most disadvantaged students—both traditional Ohio 8 schools and public charter  schools—rise to near the middle of the rankings among schools statewide when ranked according to Ohio’s “value-added” measure of annual student progress.
• The Ohio 8 school districts and the state’s public charter schools serve students with similar needs and produce similar gains in student performance.

In trying to compare the respective graphs and charts from each report, we see:

Ohio Alliance for Public Charter Schools	KidsOhio.org

If you compare these two graphs, you can come to two very different sets of conclusions.  The Ohio Alliance for Public Charter Schools report clearly indicates that cyber charter schools do marginally better than schools in the 8 urban districts on all three measures.  However, when you look at the KidsOhio.org report it indicates that charter schools (which would include the cyber charter schools) did marginally better on the value-added rank and marginally worse on the performance rank compared to schools in the 8 urban districts.  Both groups performed lower than all remaining public schools (and I appreciate the inclusion of the additional data).

Ohio Alliance for Public Charter Schools	KidsOhio.org

Again, if you look at these two charts side-by-side they tell two very different stories.  The Ohio Alliance for Public Charter Schools report shows that e-schools meet or exceed expectations at a much higher rate than schools in the 8 urban districts.  The KidsOhio.org report shows that all charter schools (which again would include cyber charter schools) have a higher percentage of schools in the meets or exceeds category, although the use of only percentages hide some of the data – as there are twice as many urban schools as charter schools.  If you combine these two charts and provide all of the data, it would read:

	Cyber Charter Schools	Brick-and-Mortar Charter Schools	All Charter Schools	Schools in the 8 Urban Districts	All Other Public Schools	All Public Schools
Above Expectations	1 (14%)	10 (19%)	11 (19%)	5 (5%)	1 (9%)	6 (5%)
Meeting Expectations	4 (57%)	15 (29%)	19 (32%)	18 (18%)	0 (0%)	18 (16%)
Below Expectations	2 (29%)	27 (52%)	29 (49%)	79 (77%)	10 (91%)	89 (79%)
Total Schools	7	52	59	102	11	113

When you compare the raw scores like this in an honest and transparent way you still have to give both charter schools and, in particular, cyber charter schools their props.  I say raw scores because the percentage scores can be very misleading.  For example, the cyber charter schools have 14% of their schools that are above expectations.  But that is only one school.  If that school fell to the meeting expectation category, cyber charter schools would have 0% of their schools in the above expectation category.  By the same token, if one of the schools from the 8 urban school districts fell from the above expectations to meeting expectations that group would still have 4% of their schools that were above expectations.  Comparatively, when you account for the differences in the total population for each category, the group of schools that appears to be doing the best based on this data is the “all charter schools” group.  It would have been interesting if one or both reports presented their data in this fashion (i.e., broken down into the various groups), as this is a much more honest representation of the data – as compared to the more slanted (and I would argue politically motivated) presentations found in the reports.

In defence of the authors of the report, some might argue that comparing the e-schools against all schools in the state is unfair because the student populations are not that similar (i.e., that it is a comparison of apples and oranges – an argument that I often make myself), and that a comparison with urban schools who have greater similarities is more like comparing apples to apples.  But are these apples really the same?  If you read the Ohio Alliance for Public Charter Schools report, they report the following demographic data to support their apples to apples position:

This comparison makes it appear as if the students in the cyber charter schools are almost the exact same as the students in the schools from the 8 urban districts – at least in terms of their achievement profile.  However, in the KidsOhio.org they indicate:

Not quite the same picture.  The schools in the 8 urban districts have a higher percentage of economically disadvantaged students and a significantly higher percentage of minority students.  What are two groups in American society that tend to underperform in the K-12 school system?  Hmmm, I wonder…  So, are we really comparing apples and apples?

Overall, it is also worth noting the differences in the two reports release by the sponsoring organizations.

• Ohio Alliance for Public Charter Schools – E-schools Show Superior Results: Analysis of State Value-Added Data Confirms E-schools Students’ Progress
• KidsOhio.org – Analysis Shows Ohio’s 8 Large Urban Districts and Charter Schools Rank Higher on Educational Progress Than on Absolute Test Scores

Basically, the report released on the Ohio Alliance for Public Charter Schools is briefer, has many of the items that indicate that e-schools or charter schools in general do not compare well to schools state-wide removed, and has a more slanted title.  The report released on the KidsOhio.org website is longer, fairer in its portrayal, and has a more descriptive title.  The KidsOhio.org report also does a better job indicating that e-schools and the schools in the 8 urban districts are closer in terms of the students they serve and provides a comparison against those 8 urban districts, but at the same time provides the data for all of the other public schools in the state (which the e-schools do not do a good job against).

In terms of the overall conclusions, they are difficult to draw because of the limited data provided in either (and I would argue the politically slanted presentation of the data – which has been consistent with other charter school reports I have reviewed, see Selective Conclusions About Charters).  It appears that there are a greater number of charter schools that are meeting expectations or are above expectations, having said that it appears that the vast majority of traditional public schools in the state have a higher percentage of economically disadvantaged students and minority students.

Without the raw scores on a per student basis (so that socio-economic status, race, type of school, location of school, etc. could all be added as variables), it is impossible to tell if the type of school or the method of delivery is responsible for the differences seen between school type.  For example, while we know that 66% of students attending charter schools are economically disadvantaged, we have no idea what that percentage is for those 7 cyber charter schools.  The higher proportion of those schools in the meeting expectations and above expectations may indeed be due to the fact that the students are attending charter schools that offer instruction online, but it could also be due to the fact that they have among the lowest percentage of economically disadvantaged students.  We don’t know because these organizations have cherry-picked the data that best serves their own purposes.