This is the twenty-fourth session – and final one for Monday (and the conference) – that I am blogging from the 2014 annual meeting of the American Education Research Association (AERA) in Philadelphia. This session was a part of a symposium that was described as:
Virtual Schools in the United States 2014: Politics, Performance, Policy, and Research Evidence
In the past decade, virtual education has moved quickly to the top of the K-12 public education reform agenda. Though little is known about the efficacy of online education generally or about individual approaches specifically, states are moving quickly to expand taxpayer-funded virtual education programs. The main purpose of this session is to understand the specificities of today’s virtual school movement as it moves from novelty to mainstream. Drawing from a rich array of theoretical perspectives and content disciplines, we will examine the performance of full-time, publicly funded K-12 virtual schools, describe the policy issues raised by the available evidence, assess the research evidence that bears on K-12 virtual teaching and learning, and offer research-based recommendations to help guide policymaking.
The actual session is described in the online program as:
Examining Variation in Achievement Impacts Across California’s Full-Time Virtual Schools
Charisse Atibagos Gulosino, University of Memphis; Jonah Liebert, Teachers College, Columbia University
Perhaps the most significant current trend in education reform is the growth of virtual (online) schools (Watson et al., 2011, 2012). While these schools offer the potential to radically restructure the way that teaching and learning happens, they also present challenges for researchers and policymakers who want to know whether they work. Specifically, the extent to which virtual schools depart from traditional brick-and-mortar schools creates difficulties with respect to assessing what these schools are doing in terms of teaching and learning and how well they are doing it.
This study uses longitudinal student-level data covering all full-time virtual schools (thirty-two total) in California from 2010-2012 to study the effect of virtual schools on student performance. Based on our web-based research, all full-time virtual schools in California are contracted to run as charter schools. Full-time virtual schools are defined as those schools in which instruction is delivered entirely or primarily through online methods (Watson et al., 2012). However, students self-select into virtual schools, making it difficult to estimate the effects of these schools on achievement. This study addresses this challenge using propensity score matching (PSM). Following the counterfactual framework (Rosenbaum & Rubin 1983; Rubin 1974), PSM matches virtual school students (“treatment” group) to those who are non-virtual school students but similar in all other preexisting observed characteristics (“control” group), based on a propensity to attend a full-time virtual school. In addition, this study addresses selection bias due to both observed and unobserved covariates. Previous studies employing the PSM approach have focused mainly on selection bias due to observed covariates (Chevalier & Viitanen, 2003). Using the Rosenbaum bounds method (Rosenbaum 2002), we evaluate the extent to which selection bias on unobserved covariates would nullify propensity score matching estimates of the effects of virtual schools.
The data for this study come from the California Department of Education (CDOE). The Department maintains longitudinal records on all public school teachers and students, including test scores (CALPADS), demographic data, enrollment and attendance information. This study supports our larger project’s focus on heterogeneity in the achievement effects of charter school attendance across demographic groups in California. We focus on the difference in impact between virtual school students and non-virtual school students in our sample, primarily because of the finding of large positive impacts in urban charters and non-significant or negative impacts in non-urban charters has been noted in our prior analysis.
Although virtual schooling is gaining ground in the K-12 classroom (Molnar et al., 2013), its impact on academic performance remains largely unexplored. Considered one of the largest markets of virtual school programs in the United States, California offers a fertile context for the study of virtual school impacts and thus serves as the focus of our study. Ultimately, our study focuses on discovering which policy-amenable aspects of virtual schools—their characteristics and conditions— are related to their ability to maximize student learning and close the achievement gap.
While a part of our symposium, this portion was not part of the National Education Policy Center’s report Virtual Schools in the U.S. 2014: Politics, Performance, Policy, and Research Evidence.
This was a difficult one for me to follow, as it was highly statistical. The other difficulty was that the presenter was not able to present the results. Basically, the California Department of Education has indicated that they are unable to verify the data that Charisse and her team have collected – even though the data has come from the California DoE’s own website – they have forbidden her from presenting the data (something I was led to believe came about after she submitted her AERA proposal). She – and others in the room – were hopeful that this might be worked out at some stage.
Some notes that I was able to capture. Her data was from the 2010-11 school year to the 2012-13 school year. It used a PMS feeder school model to address selection bias (a technique commonly used by CREDO in their studies of charter schools and cyber charters). She used a multivariate analysis strategies, that had three matching procedures, as a way to try and compare apples to apples. That’s about all I was able to get from the procedure – and, as I mentioned, she was unable to present the actual data.
A couple of resources that she did mention that helped inform her work included:
Cassandra Guarino, Ron Zimmer, Cathy Krop, and Derrick Chau. Nonclassroom-based Charter Schools in California and the Impact of SB 740. RAND: MG-323-EDU, February 2005.
Ron Zimmer , Richard Buddin, Derrick Chau, Brian Gill, Cassie Guarino, Laura Hamilton, Cathy Krop, Dan McCaffrey, Melinda Sandler, and Dominic Brewer. Charter School Operation and Performance: Evidence from California. RAND: MR-1700, July 2003.
So I wanted to share those as well.
This is me officially signing off from AERA 2014…