Final Stages of Rob’s Dissertation Topic Journey

Well, it has been almost a month sinc I’ve posted on this topic – largely because I’ve been waiting to see what Rob’s next move would be (see Defining A Dissertation Topic – Which Aspect Of K-12 Online Learning? and Rob’s Continuing Dissertation Journey).  As a reminder, he began by sending out a message to a bunch of folks on the iNACOL research committee – and a number of other people, then he narrowed his dissertation topics to a list of four reasonable topics, and then he further narrowed those topics to two possible ideas.  A day or so ago, he indicated that his dissertation topic was decided.

The research questions he has decided upon are:

1. Are there a disproportionate number of at risk students attending online charter schools as compared to comprehensive high schools?
2. Are at risk students being more successful in online schools?

His plan is to “compare full time online students with students in a comprehensive high school…. hoping to find an online school in one region/county in California who will provide me with their student statistics regarding demographics, test scores, grades, etc.  And, then find a corresponding comprehensive high school in the same region with the same statistics.”

In looking at these questions, the first thing that comes to mind – particularly with question 2 – is some advice I provided Rob earlier (see Defining A Dissertation Topic – Which Aspect Of K-12 Online Learning?) when I stated:

However, the problem with this kind of analysis – in my opinion anyway – is twofold.  The first is the obvious objective that is raised based on years of media comparison studies in the field of educational technology (i.e., comparing student performance in a technology mediated environment vs. one where technology is not present).  This problem is best summarized – almost infamously – by Clark (1983), when he indicates that technology influences learning as much as the delivery truck affects the nutritional value of the groceries it carries.  His basic argument is that it is impossible to attribute gains or losses in student performance to the presence or absence of technology because there are too many variables that could influence that learning that simply can’t be controlled for.

Similar to that argument, one of those variables that I would argue affects K-12 online learning moreso than other media comparison studies is the nature of the students being examined.  Let’s take Cathy’s 2001 meta-analysis as an example.  As the article was published in the first issue of 2001, it was likely accepted in late 2000, revised in Fall 2000, reviewed in the Summer 2000, so likely written in early to mid-2000 (January to June 2000).  What kind of K-12 online learning programs were available prior to June 2000 and what kind of students tended to enroll in those online programs?  I would argue, quite rightly I believe, that the majority of K-12 online learners were students enrolled in Advanced Placement, second languages, or higher level mathematics and science courses.  These students would have likely been your better performing students.  However, Cathy found that there were no significant differences in K-12 student performance between online and classroom-based students.  Isn’t it odd that a cohort of higher performing students were only able to do as well as a wider ability range of classroom-based students?

This is simply a quick illustration.  If you want a more researched perspective, I suggest you take a quick look at the attached document (Barbour & Reeves, 2009), beginning with the paragraph that starts “The remaining two challenges outlined by Berge and Clark (2005) were student readiness issues and retention issues.” on page 409 to the end of that section on page 411.  This gives a better explanation of the problem with these comparisons, and even some of the rationale as to what it is problematic (also giving you the appropriate citations).

Barbour, M. K., & Reeves, T. C. (2009). The reality of virtual schools: A review of the literature. Computers and Education, 52(2), 402-416.

Clark, R. E. (1983). Reconsidering research on learning with media. Review of Educational Research, 53(4), 445–459.

Beyond that, however, my bigger question is “So what?”  Let’s say that you find a higher proportion of at-risk students in the online school and let’s say those at-risk students in that online school perform better than the at-risk students in the comprehensive high school, what does that tell us?  As a statistical research study, it tells us that this one online school served a higher percentage of at-risk students and students at that one online school out-performed a brick-and-mortar located in roughly the same geographic location.  That doesn’t tell us much and, more importantly, it doesn’t tell us why or how.  Why did those at-risk students in the online environment out-perform their brick-and-mortar counterparts?  Did it have anything to do with the fact they were online?  Did it have something to do with the fact that their parents cared enough about their educational opportunities and success that they pulled them from the brick-and-mortar system?  Did it have something to do with the control over their learning that they could exercise in the online environment and would have done just as well had we purchase a bunch of CD-Roms to deliver their educational experience?  Correlation doesn’t equal casuality, and in this instance without casuality there is little value in the results.

I’m also reminded of some of the work that one of my own dissertation advisors did on a notion called socially responsible research (and I particularly recommend Reeves (2005) from below).

• Reeves, T. C. (1995). Questioning the questions of instructional technology research. Peter Dean lecture delivered at the national convention of the Association for Educational Communications and Technology. Anaheim, CA. Retrieved on July 30, 2005 from http://www2.gsu.edu/~wwwitr/research/reeves1995.htm
• Reeves, T. C. (1999). A research agenda for interactive learning in the new millennium. Keynote delivered to the World Conference on Educational Multimedia, Hypermedia & Telecommunications. Seattle, WA. Retrieved on July 27, 2005 from http://it.coe.uga.edu/~treeves/EM99Key.html
• Reeves, T. C. (2000). Enhancing the worth of instructional technology research through “design experiments” and other development research strategies. Paper presented at the annual meeting of the American Educational Research Association. New Orleans, LA. Retrieved on July 25, 2005 from http://www.coe.uga.edu/~treeves/AERA2000Reeves.pdf
• Reeves, T. C. (2000). Socially responsible educational technology research. Educational Technology, 40(6), 19-28.
• Reeves, T. C. (2005). No significant differences revisited: A historical perspective on the research informing contemporary online learning. In G. Kearsley (Ed.), Online learning: Personal reflections on the transformation of education (299-308). Englewood Cliffs, NJ, Educational Technology Publications.

This is a polite – well, maybe not so much – way of asking what is the benefit of this study?  Once it has been completed – beyond the fact that it will earn Rob his doctoral degree, which is no small feat – but who will be better off and how will they be better off?  What will we know that will be of use to those working at these two schools?  Those working with at-risk students?  Those working in K-12 online learning?  It is a reasonable question to ask.  The purpose of academic inquiry is to add to the knowledge base, which Rob‘s study does.  However, I also think that as researchers we have a bigger purpose and that is to do some good for those with whom we study or those who come after and find themselves in similar situations.

So, in Rob‘s case while he may not be able to devise a study that will help the students who participate in it, the results of his study should be able to help the teachers and administrators at those schools do a better job with future students.  Based on the two questions Rob has proposed, I don’t see how that can be done because the presence or absence of the online environment cannot be said to cause the changes in performance.  This means that his results will be able to say there is a difference in the percentage of at-risk students served and there is a different in the student performance, but he won’t be able to say why – and that would be the critical question that could help future at-risk students.

Rob, I hope this critical feedback is taken in the spirit it is offered (and that would be to assit you in designing a study that will be of benefit to the population of students that I know you care deeply about).  Also, if other researchers and evaluators are reading this please feel free to provide Rob with any additional thoughts/advice.  Finally, if there are teachers out there that work with at-risk students, if you could have someone do a research study with your at-risk students what would you want to know?