Virtual School Meanderings

July 2, 2009

US DOE Report: Evaluation of Evidence-Based Practices in Online Learning

inacolThis is another one of these reports that have hit me from a number of directions in the past few days. I suppose the first was from one of the iNACOL forums with this message:

Online learning has definite advantages over face-to-face instruction when it comes to teaching and learning, according to a new meta-analysis released Friday by the U.S. Department of Education.

The study found that students who took all or part of their instruction online performed better, on average, than those taking the same course through face-to-face instruction. Further, those who took “blended” courses — those that combine elements of online learning and face-to-face instruction — appeared to do best of all. That finding could be significant as many colleges report that blended instruction is among the fastest-growing types of enrollment.

The Education Department examined all kinds of instruction, and found that the number of valid analyses of elementary and secondary education was too small to have much confidence in the results. But the positive results appeared consistent (and statistically significant) for all types of higher education, undergraduate and graduate, across a range of disciplines, the study said.

A meta-analysis is one that takes all of the existing studies and looks at them for patterns and conclusions that can be drawn from the accumulation of evidence.

Read the full article at:

Then it came in my Facebook in the form of a message from the United States Distance Learning Association (USDLA).

“Blended Learning” More Effective than Face-to-Face (Source: Education Week, 2009)

Download Report:Evaluation of Evidence-Based Practices in Online Learning

A new
report released today by the U.S. Department of Education, which analyzed 46 studies comparing online learning to face-to-face education, concluded that “blended learning,” or programs that include elements of both face-to-face and online learning, is somewhat more effective than either approach by itself. The study also found that, by itself, online learning was more effective at raising student achievement than face-to-face instruction exclusively.

“This new report reinforces that effective teachers need to incorporate digital content into everyday classes and consider open-source learning management systems, which have proven cost effective in school districts and colleges nationwide,” said U.S. Secretary of Education Arne Duncan in a statement. “To avoid being caught short when stimulus money runs out, school officials should use the short-term federal funding to make immediate upgrades to technology to enhance classroom instruction and to improve the tracking of student data.”

I’m sure online education advocates are thrilled at the conclusions drawn by this report. But while it does put online education, especially in a blended environment, in a favorable light, there are a couple of significant disclaimers.

Researchers found that blended learning environments often included additional learning time and incorporated more instructional elements, which “suggests that the positive effects associated with blended learning should not be attributed to the media, per se,” said the report. Also, the analysis found very few studies conducted specifically with K-12 schools, therefore “caution is required in generalizing to the K-12 population because the results are derived for the most part from studies in other settings (e.g., medical training, higher education).”

In fact, the report goes so far as to say, “the most unexpected finding was that an extensive initial search of the published literature from 1996 through 2006 found no experimental or controlled quasi-experimental studies that both compared the learning effectiveness of online and face-to-face instruction for K-12 students and provided sufficient data for inclusion in a meta-analysis.”

That’s a pretty sad statement on the amount of research, or lack thereof, on K-12 online learning

Download Report:

And let’s not forget the blogsphere…

It is interesting to see the differing slants that folks have put on this issue.  Just looking at the various blog titles and message subjects tells a great deal (e.g., from ).

usdoe_reportIn looking at the actual report, a couple of interesting things from the Executive Summary:

  • “The most unexpected finding was that an extensive initial search of the published literature from 1996 through 2006 found no experimental or controlled quasi-experimental studies that both compared the learning effectiveness of online and face-to-face instruction for K–12 students and provided sufficient data for inclusion in a meta-analysis.” (p. xii)
  • “Among the 51 individual study effects, 11 were significantly positive, favoring the online or blended learning condition. Two contrasts found a statistically significant effect favoring the traditional face-to-face condition.” (p. xiii)
  • “Students who took all or part of their class online performed better, on average, than those taking the same course through traditional face-to-face instruction.” (p. xiv)

A couple of things that I take away from these statements.  The first is that these findings are based on K-12, undergraduate, and graduate students.

The second thing is that I’m a bit concerned that anyone in this day and age believes that “experimental or controlled quasi-experimental studies” could actually control for all of the variables that one finds in an educational setting – and even with random assignment, there are still too many factors that could account for any differences found in student performance.

The third thing I take away is that I’m not surprised that the authors found that students in online environments did better than students in the same course in a face-to-face format.  Consider the way online students are described in the literature based on this quote from Barbour and Reeves (2009):

The nature of students who are served by virtual schools has been a consistent discussion in the literature. Clark et al. (2002), in their evaluation of the IVHS found that students who were ‘‘highly motivated, high achieving, self-directed and/or who liked to work independently” typically did well in the online environment (p. 41). This was consistent with the characteristics of the student Stevens (1999b) described earlier. Both of these descriptions are also consistent with the characteristics more often attributed to adult learners, who according to Knowles (1970) are more self-directed and independent in their orientation to learning than adolescents.

These findings were supported by the work of SRI International and their five-year evaluation of the VHS. The VHS is one of the oldest and most researched virtual schools in the United States. In their first year evaluation of the VHS, Kozma, Zucker, and Espinoza (1998) found that the vast majority of students in their courses were planning to attend a four-year college. They also reported that two thirds of the teachers indicated that the VHS students were less likely to drop out of school than students in their classroom-based courses. These findings led the evaluators to conclude that ‘‘the current VHS curriculum [was] dominated by advanced courses that cater to students who are successful, independent, and college bound” (p. 49). The following year, Espinoza, Dove, Zucker, and Kozma (1999) reached similar conclusions when they stated that ‘‘VHS courses are predominantly designated as ‘honors,’ and students enrolled are mostly college bound” (p. 49). These finding were not surprising to the evaluators, as they indicated that the VHS’ own faculty handbook promoted this kind of selectivity when it stated: ‘‘although all students should have access to the VHS catalog, we recommend that the school site coordinator and guidance counselors select students who can work independently and handle responsibility” (p. 50).

In both the Clark et al. (2002) and the Espinoza et al. (1999) evaluations, the authors recommended that the virtual school take steps to increase the range of students served. During the third year evaluation of the VHS, Kozma et al. (2000) took a slightly different approach and focused upon four classes as a case study of the VHS model. The four courses selected were Advanced Placement Statistics, Modern Classics, Photographic Vision, and pre-engineering and design. The students from these classes were described by their teachers as very capable academically and college bound. However, even with this selectivity the evaluators still found a higher dropout rate for these four VHS courses than for the face-to-face comparison groups. To summarize their five-year evaluation of the VHS, Zucker and Kozma (2003) released the book The Virtual High School: Teaching Generation V. In this volume, they reported that students who were not expected to succeed in the VHS environment were discouraged from taking VHS courses and that more than four out of every five students in VHS courses were college preparatory.

Issues of student selectivity had also been found in evaluations of virtual schools in Canada. Haughey and Muirhead (1999), in their examination of online learning in the province of Alberta, described the preferred characteristics of K-12 students involved in virtual schooling to include the highly motivated, self-directed, self-disciplined, independent learner who could read and write well, and who also had a strong interest in or ability with technology. Later in their evaluation of student achievement and performance in online learning in Alberta, Ballas and Belyk (2000) found that the performance of virtual and classroom students were similar in English and social studies courses, but that classroom students performed better overall in all other subject areas (i.e., biology, chemistry, mathematics, and physics). The authors also indicated that the participation rate in the assessment among virtual students ranged from 65% to 75% compared to 90–96% for the classroom-based students. This led them to speculate, probably quite accurately, that the sample of virtual school students did not reflect the total population of these students. While not discussed by the authors, it is plausible that the 15–25% difference in participation rate reflected all of the low-achieving students, as was raised by McLeod et al. (2005). This would indicate that in the majority of courses examined, the virtual school students had lower achievement levels, even with a more selective group of students. At present, there has been little or no research into the reasons for the poor performance of these highly skilled and more motivated students. Speculation has been that the learning experience provided by virtual schools was not at the same caliber as the learning experience that classroom based students have received – although there is also no research to support this potential cause. Barker and Wendel (2001) reported a comparison of performance between students attending six virtual schools and three conventional schools from three different provinces over a three-year period. Their findings were that students in the six virtual schools performed no worse than the students from the three conventional schools at both the grade nine and grade twelve levels. Again, even with a more selective group of students in the virtual school, the performance of the virtual school students did not exceed that of their classroom counterparts. (p. 410)

This is not to say that all K-12 students who enroll in online learning are similar to this description.  But I do believe that the students who have been subjects in the research studies that were likely to be included in this report fit this description.  So it isn’t surprising that THESE online students do better than their classroom counterparts.

Finally, it is worth noting that this kind of scholarship isn’t that useful at this stage…  For example, let’s imagine that this report found that online learning was a poor imitation of what a student could experience in a traditional classroom.  Do you really think that would slow the growth of online learning at any level?  The fact of the matter is that online learning is here, is here to stay, and is likely only going to continue to grow.  Acknowledging this reality, the question shouldn’t be is it as good as face-to-face learning?  That is a red herring kind of question.  The question should be: how do we design and deliver online learning to maximize the strengths of the medium, while minimizing the limitations of the medium?  In my honest opinion, this report is largely a waste of paper, money and human resources which could/should have been put to better use!!!


Barbour, M. K., & Reeves, T. C. (2009). The reality of virtual schools: A review of the literature. Computers and Education, 52(2), 402-416.


  1. Great insights Michael. Always appreciate reading them. I particularly appreciate what the “real” question should be! Let’s hope we all continue to move in that direction.

    Comment by robdarrow — July 3, 2009 @ 2:15 pm | Reply

  2. Thanks Rob!!! Make sure you have read the entry that I wrote shortly thereafter (see A Quick Word About Reports), as it puts things into a bit of perspective.

    BTW, how is the dissertation topic coming? I know that I was a little critical of your last idea, so I wanted to check in.

    Comment by mkbnl — July 3, 2009 @ 6:40 pm | Reply

  3. […] about the nature of student required to be successful in an online environment (see the bottom of US DOE Report: Evaluation of Evidence-Based Practices in Online Learning for one of the more recent examples).  In my own writing I often cite Mulcahy (2002) who mused […]

    Pingback by iNACOL Offers Advocacy Documents « Virtual High School Meanderings — August 6, 2009 @ 5:22 pm | Reply

  4. […] I’ll leave the last word to the blogger on Virtual High School Meanderings, who concluded with the pithy statement that “this report is largely a waste of paper, money […]

    Pingback by A report on online learning | David's Occasional Blog — August 18, 2009 @ 2:35 am | Reply

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Blog at

%d bloggers like this: