Virtual School Meanderings

May 18, 2017

Validated Online Course Design Standards

A few weeks ago I saw this news item scroll across my electronic desk – Edmentum Plato Courses Receive iNACOL Standards Approval. When you go to the news item, you get access to this summary:

Edmentum’s Plato Courseware’s iNACOL Standard Review –

A former doctoral student of mine actually completed his dissertation research focused on whether the iNACOL National Standards for Quality Online Courses could be validated based on research. Here are the results.

Adelstein, D., & Barbour, M. K. (2016). Building better courses: Examining the content validity of the iNACOL national standards for quality online courses. Journal of Online Learning Research, 2(1), 41-73. Retrieved from

  • The first round focused on whether there was literature and, more importantly, the research literature support for the standards.  David was able to find something for just about every standard, but there was actually little in the way of K-12 distance, online, and blended learning research literature to support most standards as they were written.

Adelstein, D., & Barbour, M. K. (2017). Improving the K-12 online course design review process: Experts weigh in on iNACOL National Standards for Quality Online Courses. International Review of Research in Open and Distance Learning, 18(3). Retrieved from

  • The second round had two teams of experts review the standards to determine whether, based on their expert opinions, the standards were appropriate measures of the quality of online course design.  While there was some general support for most standards, many of them were revised or dropped based on this expert feedback.

Adelstein, D., & Barbour, M. K. (2016). Redesigning design: Field testing a revised design rubric based of iNACOL quality course standards. International Journal of E-Learning & Distance Education, 31(2). Retrieved from

  • The third round had several teams of two reviewers that were responsible for applying the rubric associated with the standards to a variety of online courses to determine if inter-rater reliability could be achieved.  The results of this application were that the rubric could not be reliably applied.

Adelstein, D., & Barbour, M. K. (2016). Redesigning design: Streamlining K-12 online course creation. MACUL Journal, 37(1), 20-21. Retrieved from

  • This final article was targeted to practitioners.  It was basically an attempt for David to translate what he learned from the three rounds of research that would be of interest and of use to a practitioner audience.

November 28, 2016

Article Notice – Redesigning Design: Field Testing a Revised Design Rubric Based of iNACOL Quality Course Standards

Notice of this article came through my inbox today…

David Adelstein, Michael K Barbour

International Journal of E-Learning & Distance Education (formerly the Journal of Distance Education) – Vol 31, No 2 (2016)


Designers have a limited selection of K-12 online course creation standards to choose from that are not blocked behind proprietary or pay walls. For numerous institutions and states, the use of the iNACOL National Standards for Quality Online Courses is becoming a widely used resource. This article presents the final phase in a three-part study to test the validity and reliability of the iNACOL standards specifically to online course design. Phase three was a field test of the revised rubric based on the iNACOL standards against current K-12 online courses. While the results show a strong exact match percentage, there is more work to be done with the revised rubric.

RésuméLes concepteurs ont une sélection limitée des normes K-12 de création de cours en ligne à choisir qui ne sont pas bloqués derrière des propriétés exclusives ou des péages informatiques. Pour de nombreuses institutions et états, l’utilisation des Normes nationales pour les cours en ligne de qualité iNACOL devient une ressource largement utilisée. Cet article présente la phase finale d’une étude en trois parties pour tester la validité et la fiabilité des normes iNACOL spécifiquement liées à la conception de cours en ligne. La phase trois était une mise à l’essai sur le terrain de la rubrique révisée établie en fonction des normes iNACOL par rapport aux cours en ligne K-12 actuels. Bien que les résultats montrent un fort pourcentage de correspondance exacte, il y a plus de travail à faire avec la rubrique révisée.


K-12 online learning, K-12 distance education, virtual school, cyber school, online course design

Full Text:


Full disclosure: I was a co-author on this article.

December 17, 2013

Validated Standards In K-12 Online Learning

I’ve spoken in the past about the need for validated standards to measure quality in various aspects of K-12 online and blended learning, and how the “national standards” presented by the International Association for K-12 Online Learning have not gone through the normal validation process.    Interestingly, the folks over at Getting Smart (of all places, a bunch of neo-liberals finally highlighting a research-based practice) have described such a process related to the Quality Matters framework.

Quality Matters in K12 Online and Blended Learning

By: Jim Snyder 

Quality Matters is a widely adopted program for quality assurance in online learning serving K–12, postsecondary, and other organizations involved in e-learning in the United States and internationally. The primary offerings and components are a set of design standards (or rubric) for the continuous improvement of online and blended courses, a peer review process for applying these standards, and related professional development.  The Quality Matters Rubric, with versions for continuing and professional (corporate) education, educational publishing, secondary education, and post-secondary education, is based on recognized best practices, built on the expertise of instructional designers and experienced online teachers, and supported by distance education literature and research.  The Rubric and web-based tools are complete with annotations that explain the application of the standards and the relationship between them.  The goal of the program is to increase student engagement, learning, and satisfaction in online or blended courses by implementing better course design.

Professional Development supports the QM program components and is an important part of a quality assurance effort.  After taking QM’s K-12 Applying the Quality Matters Secondary Rubric workshop – the most comprehensive approach to the K-12 Secondary Rubric that QM offers — Elliott Lemberg from New-Albany-Plain Local School District in Ohio stated that he learned how to use the Rubric to evaluate the quality of online and blended courses and, ”this course has been an edifying experience for me. Not only did it reinforce what I had already deduced about online learning environments through other workshops I have taken, but it also opened my eyes to the rigor of designing a blended or an online course. It showed me how the [K-12 Secondary] Rubric can be invaluable resource and tool for ensuring continuous improvement in those courses by guaranteeing they are aligned to the learning objectives. In my years as an educator, I have encountered many educational rubrics that purport to promote constant enhancement of design and pedagogy, but this rubric is the first one that truly achieves that goal. It is accessible and lucid and provides great insight and spurs tremendous professional reflection for all educators.”

The Education Service Center of Central Ohio turned to the Quality Matters Program (QM) to promote rigorous, uniformly applied standards for online courses for students and online professional development for educators.[1] Through the Quality Matters Professional Development Grants, eTech Ohio sought to identify agencies with the capacity and desire to deliver professional development to schools and educators throughout Ohio. “This grant provides key infrastructure to support Ohio’s K-12 teachers in their efforts to provide high-quality online instruction,” said Greg Davidson, interim executive director of eTech Ohio. “It is an important component of the state’s overarching support of digital learning initiatives.”[2]

To continue reading…

July 20, 2013

CLRN Online Course Training Videos

Notice of this entry came through my RSS reader a few days ago…

CLRN Online Course Training Videos

What separates a high quality online course from a flat and linear textbook are iNACOL’s Standards for Quality Online Courses. iNACOL’s standards, created by CLRN and variety of California and national stakeholders and published in 2011, outline specific expectations for content, instructional design, student assessment, technology, and course evaluation and support. California and Texas exclusively utilize iNACOL’s standards for course reviews and a variety of other states have adapted the standards for their reviews.

We understand that our reviews have an impact, both in the classroom and in the industry, which is why we strive to be transparent both within our published reviews and in our review process. Providing our stakeholders with both the standards and reviewer considerations, which define both the expectations and the evidence we must find, is an important step. To facilitate CLRN reviewer retraining, we’ve also created four instructional videos that provide more information about each standard and our thoughts about the validation process. Each of the four videos, which include Sections A-D from the online course standards, may be access here:
Section A: Content

Section B: Instructional Design

Section C: Student Assessment

Section D: Technology

Now in the past I have questioned the wisdom of using a set of standards that have yet to be found to be reliable and valid (at least based on the published research literature) as a way of measuring quality in any formal way.  So take these resources with a grain of salt in terms of what they actually help prepare you to measure.


May 27, 2013

Online Teacher Resource – Useful Resource Or Possibly Just Fluff?

This came through my inbox back in late October…

Illustrating iNACOL’s National Standards for Quality Online Teaching …
“Illustrating iNACOL’s National Standards for Quality Online Teaching” is an eight -part series highlighting the International Association for K-12 Online

Essentially, this was a resource created by the International Association for K-12 Online Learning that included a series of videos to illustrate the various “national standards for quality online teaching.”  At present, there are eight videos there – each one aligned with one of the iNACOL standards.

I ask the question “useful resource or possibly just fluff?” because of the nature of the standards that these video illustrate.  I think that this resource could have great potential, assuming it was based on standards that have been found by independent research to be reliable and valid.  The iNACOL standards have not been through this process yet – even though several states have adopted them as their sole measure of quality.

Either way, the videos are available at:

Next Page »

Blog at