EDTECH597 – Commentary Entry: Dissecting The Arkansas Virtual Academy Study

So this topic has been sitting in my draft folder for quite some time, waiting for me to find a few minutes to actually dissect this report. Anyway…

Back in January, a pair of researchers at the University of Arkansas released an external evaluation of student performance at the Arkansas Virtual Academy Charter School (ARVA). The accompanying press release indicated that University Report Finds Positive Academic Results For Arkansas Virtual Academy. This was actually a reasonable headline, as that was what was found – positive results. But as those who have followed this blog for anytime will know, those surface levels “results” often do not tell the full story. So let’s examine this study a bit more first.

The study used a “nearest neighbor” methodology to compare student performance of ARVA students with two students located in the resident school district that the ARVA student used to attend. The students were matched based on having a similar baseline standardized test score in mathematics and literacy, and the comparison was made based on their test scores two years later (to measure student growth in each environment from the same starting point). The researchers chose to use test scores two years later to allow for any spikes due to the transition from face-to-face to online to smooth themselves out. The researchers also made attempts to match for socioeconomic status (using free and reduced lunch status), race, and gender. Again, on face value all of this appears to be rather reasonable and makes perfect sense – until you begin to examine what these methodological decisions meant to the study, as well as ask a few questions about the actual control and treatment samples.

For example, by deciding to use data two years after the initial baseline data was collected it meant that any student who repeated a grade (in either environment) would be removed from the sample. This resulted in the four weakest students from the ARVA group being removed from the sample because they repeated a grade, while there were no students removed from the face-to-face sample. It also meant that any student who dropped out of the online environment and returned to the face-to-face classroom were also removed from the ARVA, a limitation that the researchers note but do not indicate how many students were in this position. Again, based on the research that we have about K-12 online learning in general – and, in particular, full-time cyber charter schooling – tells us that these students would be the weaker students and would have been among the lower performers on these standardized tests. So before we even start to compare the results we know that the ARVA sample is already skewed towards a higher performing group.

Next, we ask some questions about the information provided about the two samples. While the researchers “attempted to match for socioeconomic status (using free and reduced lunch status), race, and gender”, the limited information the researchers proved indicate that several of these factors may have further skewed the sample. For example, the researchers do tell us tat the face-to-face sample had 10% more students who qualified for free or reduced lunch in the both the mathematics and the literacy sample (with free and reduced lunch being the variable the researchers choose to use to measure socioeconomic status). As the research has shown there is a clear link between socioeconomic status and student performance, the study is already further skewed in favour of the ARVA sample. Additionally, the researchers also report that the face-to-face sample had a “significantly larger proportion of minority students” than the ARVA sample.

So let’s review….

1. The ARVA sample had several of its lowest performing students removed before they had repeated a grade or had dropped out over the two year period.
2. The ARVA sample was a more affluent group.
3. The ARVA sample had significant fewer minority students.

All of these factors call into question whether this study is actually comparing apples to apples, or if – like almost all of the student comparative research done in the field of K-12 online learning – this study is comparing apples and oranges.

So, what does this apples and oranges comparison reveal?

When comparing student performance in mathematics, the researchers found:

• students in the face-to-face group increased their performance by 1% more than the ARVA group from grades 3 to 5 (not statistically significant)
• students in the ARVA group increased their performance by 5% more than the face-to-face group from grades 4 to 6 (not statistically significant)
• students in the ARVA group increased their performance by 2% more than the face-to-face group from grades 5 to 7 (not statistically significant)
• students in the ARVA group increased their performance by 16% more than the face-to-face group from grades 6 to 8 (statistically significant at the p=0.10 level)

When comparing student performance in literacy, the researchers found:

• students in the face-to-face group increased their performance by 3% more than the ARVA group from grades 3 to 5 (not statistically significant)
• students in the ARVA group increased their performance by 11% more than the face-to-face group from grades 4 to 6 (statistically significant at the p=0.10 level)
• students in the ARVA group increased their performance by 2% more than the face-to-face group from grades 5 to 7 (not statistically significant)
• students in the ARVA group increased their performance by 7% more than the face-to-face group from grades 6 to 8 (not statistically significant)

So, the ARVA students – who were already a higher achieving, more affluent, and more white group of students – outperformed the face-to-face students in mathematics in the grade 6-8 cohort and in literacy in the grade 4-6 cohort. In all other areas they had statistically similar levels of performance.

But wait…

For those folks out there that know much about quantitative research and statistics in general might have a few questions about these measures. Typically speaking, when researchers are comparing means they tend to use an alpha level or p of 0.05 as the standard level for statistical significance. The alpha level is essentially the possibility that the results were due to chance or luck. An alpha level of 0.05 means that there is a 5% chance that the results were due to luck and a 95% chance that the results were due to the actual treatment involved (in this case the full-time online learning offered by ARVA). An alpha level of 0.05 is not only the standard for quantitative research, it is also what we generally see for surveys and polls reported in the media (i.e., 19 times out of 20 or +-2.5%/+-3%). If the researchers want to be extra sure that the results were due to the treatment and not due to chance, they will often use p=0.01 or a 1% chance that the result was due to luck (if you want more information on statistical significance, http://www.numberwatch.co.uk/significance.htm and http://stats.org/in_depth/faq/statistical_significance.htm are good primers).

Getting back to the results of this study, the researchers choose to use an alpha level of 0.10. This means that there is a 10% or one in ten chance that their results had nothing to do with the ARVA and were based simple on chance or luck. The researchers did not provide the actual result of the regression analysis, so as a reader – even an informed one – we are unable to tell if the result would have been statistically significant at the standard 0.05 level or the more cautious 0.01 level. When the treatment sample only consists of 55 students for the mathematics and 46 students for the literacy, even differences of 10% and more could be statistically insignificant at those lower levels.

So what can we really say about this study? We can say that a more academically advanced, affluent, and white group of online students performed at levels comparable to their face-to-face counterparts in six out of eight measures. On the remaining two measures, this more selective group of online students outperformed their face-to-face counterparts, but there is a 10% chance that this result was due to pure luck!

This is quite different from the claims made by those neo-liberals proponents of the privatization of public education through K-12 online learning when they claim Study Finds AR Virtual Academy Students Outperform Public School Students. Yet another example of the intellectual dishonesty that these neo-liberal proponents of K-12 online learning have to resort to as they try to convince more legislators to hand even more public money over to line the pockets of corporate executives and shareholders!