This is the sixteenth session that I am blogging from the 2014 annual meeting of the American Education Research Association (AERA) in Philadelphia. This session was a symposium that was described as:
Many current education reforms are touted as having the potential to substantially enhance student achievement at scale and, thereby, contribute to the nation’s economic competitiveness. This symposium will offer a critical examination of the economic rationale, as well as the claims for two salient reforms: (i) data-driven improvement and accountability; and (ii) on-line, virtual schooling. Presenters will demonstrate that, contrary to the assertions of proponents, the evidence-base for these reforms is quite narrow and the prospects for success are much dimmer than promised. The symposium highlights the fact that policy makers’ desire “to do something” often leads to a rush to implementation built on an over-interpretation of the existing evidence — or even a willful misreading of that evidence.
I wasn’t able to attend this session because it conflicted with the special education and cyber charter session. But I did want to highlight the three sessions in this symposium. The first session in this room was:
Will Contemporary Reform Strategies Provide the International Economic Progress and Educational Equity Gains They Promise?
William J. Mathis, University of Colorado – Boulder
1. Purpose – Since the 1983 Nation at Risk report, the most prominent educational reform strategy has been top-down, test-based, accountability accompanied by punitive consequences and frequently including various forms of privatization. The primary rationale for this approach, according to President Obama, is to make the nation economically competitive in a twenty-first century world. A second claimed benefit is the achievement of educational equity. By separating test scores by wealth and racial groups, a light will be cast on the nation’s educational disparities which will, in turn, have the salutary effect of forcing remediation of the inequities.
The purpose of this paper is to examine whether these education reform strategies produce or hold promise of producing these economic and social benefits.
2. Perspectives / Theoretical Framework – The primary determinants of national economic growth are first established through the economics literature. Then, these elements are compared to the focus of various education reform strategies. In terms of social equity, the primary approach is to explore whether the achievement gap has closed, stayed the same, or widened during the time frame of the reforms.
3. Modes of Inquiry – The methods are correlational and descriptive. The work of various labor economists is examined with a particular emphasis on current and projected job requirements in terms of types, levels and skills. The effects of education standards (and their relative difficulty) are reviewed in both a national and international context. Funding equity and adequacy are examined as relates to the identified break-out groups of interest with a focus on whether achievement gaps are opening, closing or staying about the same.
4. Data sources – World Economic Forum international economic criteria, international rankings of this data, National Assessment scores across groups and over time, National Bureau of Labor Statistics projections, identified skills for the twenty—first century, the Common Core State Standards content, and national testing consortia test specifications (to the degree they are available) are the primary data sources employed.
5. Results – Preliminary evidence indicates that contemporary reforms have led to a narrowed essentialist curriculum, a relative neglect of the broader purposes of education, the relative omission of soft skills, a failure to close the achievement gap, negligible gains from privatization efforts, a lack of relationship between achievement and difficulty of standards, and only a weak relationship between test scores and international economic competitiveness.
5. Significance of the Study – If the contemporary test-based accountability reform strategies advance national workforce and economic needs while also advancing democratic and equity goals, then we can conclude that this reform strategy is sound. If the indicators do not align or have not moved in the desired direction, then scholars, practitioners and policy-makers must more closely examine the efficacy of these reforms and whether they should be continued.
The second session in the room was:
Data-Driven Improvement and Accountability
Andrew Hargreaves, Boston College; Henry I. Braun, Boston College
This presentation offers a critical examination of data-driven improvement and accountability (DDIA): the policies and practices concerning the use of data to inform school improvement strategies and to provide information for accountability. DDIA can lead either to greater quality and equity, or to deterioration of services and distraction from core purposes. Hargreaves and Braun (2013) address the focal question: What factors and forces can lead DDIA to generate more positive and fewer negative outcomes in relation to both improvement and accountability?
Although educational accountability is meant to contribute to improved student learning, there are often tensions and even conflicts between improvement and accountability. Drawing on data from business and sports, as well as education, the presentation identifies five key factors that influence the success or failure of DDIA systems in public education:
(i) The nature and scope of the data employed by the improvement and accountability systems;
(ii) The types of indicators used to track progress or to make comparisons among schools and districts;
(iii) The interactions between the improvement and accountability systems;
(iv) The kinds of consequences attached to high and low performance and how those consequences are distributed;
(v) The culture and context of data use .
In general, we argue that over more than two decades, through accumulating statewide initiatives in DDIA and then in the successive Federal initiatives, DDIA in the U.S. has come to exert increasingly adverse effects on public education, because high-stakes and high-threat accountability, rather than improvement and accountability together, have become the prime drivers of educational change.
Contrary to the practices of countries with high performance on international assessments, and of high performing organizations in business and sports, DDIA in the U.S. has been skewed towards accountability over improvement. It has focused on what is easily measured rather than on what is educationally valued. The high-stakes, high-pressure environment of educational accountability, in which arbitrary numerical targets are hierarchically imposed, has led to extensive gaming and continuing disruptions of the system, with unacceptable consequences for the learning and achievement of the most disadvantaged students. Instead of being informed by the evidence, educators become driven to distraction by narrowly defined data that compel them to analyze grids, dashboards, and spreadsheets in order to bring about short-term improvements in results.
The presentation concludes with twelve recommendations for establishing more effective systems and processes of data-driven or evidence-informed improvement and accountability. The significance of the research presented is that, drawing on data and experiences from different sectors and different countries, it offers a credible and coherent prescription for resolving the tensions between improvement and accountability. These are most likely to be resolved when there is collaborative involvement in data collection and analysis, collective responsibility for improvement, and a consensus that the indicators involved in DDIA are accurate and fair. When these conditions are absent, improvement efforts and outcomes-based accountability can work at cross-purposes, to the detriment of all.
Hargreaves, A. & Braun, H. (2013). Data-driven Improvement and Accountability. Boulder, CO: National Education Policy Center.
And the third session in this symposium:
The Virtual Evidence Base for Virtual Education and the Implications for Policy
Jennifer K. Rice, University of Maryland; Luis Alberto Huerta, Teachers College, Columbia University
Over the past decade, virtual education has become a focal point among policy makers interested in expanding education choice options and improving the efficiency of public education. In particular, full-time virtual schools, also known as online schools or cyber schools, have attracted a great deal of attention. Proponents argue that the individually tailored nature of online curriculum has the potential to promote greater student achievement in comparison to traditional brick-and-mortar schools. Further, lower costs – primarily in the areas of instructional personnel and facilities – make virtual schools appealing from a fiscal resource perspective. Assumptions about the cost-effectiveness of virtual schools, coupled with their alignment with policies to expand school choice and the market incentives they present to for-profit companies, have fueled a fast-growing movement of virtual school expansion in the U.S. To date, 30 states and the District of Columbia allow full-time virtual schools to operate (Watson, et al. 2011) and estimates of student enrollment in virtual schools nationwide range between 200,000 to more than 250,000 (Molnar, et al. 2013). The majority of these students are enrolled in a small number of large for-profit EMOs.
This presentation will critically examine the extant evidence base for virtual education, and will discuss a number of key policy measures that should be considered as this education reform movement continues to unfold. While many states have adopted legislation that permits full-time virtual schools and increasing numbers of parents and students are choosing these options, little is known about the effectiveness of this approach to education. Existing evidence on the effectiveness of virtual education is mixed, and a recent U.S. Department of Education meta-analysis found that “few rigorous research studies of the effectiveness of online learning for K-12 students have been published” (Means, 2010, xiv).
Given the insufficient evidence to support online initiatives in elementary and secondary schools, scaling up virtual school reform presents significant implementation and accountability challenges (Molnar, et al. 2013). A continuing challenge for states will be to reconcile traditional funding mechanisms, governance structures, and accountability demands with the unique organiza¬tional models and instructional methods found in virtual schools. Drawing on recent reports and our own research on virtual charter schools, we consider relevant policy issues in the areas of finance and governance, instructional program quality, and high quality teachers. For each topic, we present the critical issues, common assumptions, and unanswered empirical questions. We conclude with a set of policy recommendations.
Barbara Means, et. al. (2010). Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies, U.S. Department of Education, Office of Planning, Evaluation, and Policy Development.
Molnar, A. (Ed.); Miron, G., Huerta, L., Cuban, L., Horvitz, B., Gulosino, C., Rice, J.K., & Shafer, S.R. (2013). Virtual Schools in the U.S. 2013:Politics, Performance, Policy, and Research Evidence. Boulder, CO: National Education Policy Center.
Watson, J., Murin, A., Vashaw, L., Gemin, B., & Rapp, C. (2011). Keeping pace with K-12 online learning: A review of state-level policy and practice. Evergreen, CO: Evergreen Education Group.