REL Mid-Atlantic Winter Newsletter

Note the remote learning items below.

Click here to view in browser

From the Director  │ Highlights │ RELevant Blog Posts

Latest News from the Director

As the current contract cycle for the RELs comes to a close at the end of 2021, we have lots of projects wrapping up—and lots of products coming out, with relevance across the Mid-Atlantic region and beyond. In collaboration with the Pittsburgh Public Schools, we’ve examined remote learning during the pandemic, using fine-grained data on students’ use of online learning systems to identify learning lags and highlight the strong relationship between student engagement and course failure. This process exposed the need to re-engage students who checked out while school buildings were closed—a need that surely exists in schools all over the country. We’ve also examined the engagement of families, with a study of a teacher home-visiting program in the District of Columbia showing that summer home visits reduced disciplinary incidence and improved attendance the following school year. We’ve shown how teacher surveys on school climate can shine a light on the performance of principals. To cap off a large portfolio of work on school performance measures, we’ve proposed a framework designed to make sense of the ways that different measures are useful—or not—for different policy, operational, and instructional decisions. These and other products are described below. With so much here, I’ve never been more confident that you’ll find something useful in this newsletter to inform your work. As always, we at REL Mid-Atlantic are grateful to all of our partners in the field who have given us the opportunity to inform your work toward promoting better and more equitable student outcomes across the region.

 

Brian Gill

Director, REL Mid-Atlantic

Highlights

Understanding pandemic-related learning lags and identifying disengaged students

To determine how student achievement was affected by the pandemic and disruptions to instruction, we partnered with Pittsburgh Public Schools to analyze their remote learning data, looking at both test scores and course grades. Test scores indicated that most students experienced growth, but less than in a typical year. These lags were largest for the youngest students, but course failures increased most in middle and high school. Attendance data indicated that many students were likely failing courses because they were not showing up for their remote classes. Using data from Pittsburgh’s learning management system, we looked at how frequently students logged into the system during remote instruction and how many course materials they opened and submitted. We identified how patterns of use changed during the school year and examined how these measures differed across different groups of students. In this webinar staff from Pittsburgh Public Schools and the REL discussed the findings, and how students engaged with online learning applications during remote instruction to help other educators consider policies and interventions that help re-engage students and address lags in learning. Check out this fact sheet to learn more.

Improving student outcomes with teacher home visits: Evidence from a REL Mid-Atlantic evaluation in DC

Read a new report and infographic examining the benefits of home visits. Using a rigorous quasi-experimental methodology, the research team studied a schoolwide program for elementary and middle grades in the DC Public Schools. The program trains teachers to conduct structured relationship-building home visits with parents and provides financial incentives to conduct the visits. The team found that summer home visits significantly reduced the likelihood of a student having a disciplinary incident, and also slightly improved student attendance the following school year.

Leveling the playing field to identify high school effectiveness

An important indicator of a high school’s effectiveness is whether its students graduate and enroll in college. But graduation and college enrollment rates are affected by factors outside of the school’s control, which means that simply comparing those rates may not fairly assess the school’s contribution to student learning. Read an infographic and report describing a new approach that measures the “promotion power” of individual high schools, putting them on a level playing field and more fairly measuring their effects on those student attainment outcomes.

Photo by Rich Clement

Surveys of staff can add perspective to principal evaluations

Many school districts assess the performance of their principals by looking at students’ test scores and having a supervisor evaluate the principal’s leadership. We worked with the District of Columbia Public Schools to understand how teacher surveys can provide valuable additional information for principal evaluations. Read the full report on what we found. Read the full report.

RELevant Blog Posts

Making sure school performance measures provide the right diagnosis to improve student outcomes

Measuring school performance has been an important component of state and federal policies for two decades. Measures based exclusively on reading and math proficiency have given way to more sophisticated approaches. But to avoid drowning in data, educators and policymakers need to understand how different measures inform different decisions. Read this new blog from our director, Brian Gill, for a new classification framework that makes sense of different measures of school performance, drawing on a wide range of REL Mid-Atlantic’s work with stakeholders in the region. Listen to a related webinar discussion.

Engaging in equity: Alternatives to suspension and expulsion

REL Mid-Atlantic partnered with the Maryland State Department of Education to better understand discipline disparities–instances where students in certain demographic groups are subject to disciplinary actions at a greater rate than the total population in the school—and help the state make progress in addressing these disparities. Read a new blog that examines how districts across the country are using professional development, community partnerships, mindfulness training, and other techniques to address these issues with an equity lens. Check out the related infographic.

Identifying students at imminent risk of academic problems

Identifying students at risk of near-term academic problems helps districts target services to prevent problems before they occur and lead to more serious consequences, such as dropout. A recent REL Mid-Atlantic study found two approaches—a simple early warning system and a more sophisticated computer algorithm—had similar predictive accuracy, but the algorithm gives districts more information and therefore more opportunity to develop different kinds of responses for students at different levels of risk. Relatedly, read a blog from a second study that explored what we found out about the benefits of incorporating data on SEL competencies and school climate into early warning systems.

Getting a clear picture of incoming enrollment

Predicting incoming enrollment is a challenge for many districts with school choice systems, substantial student mobility, or both. Inaccurate predictions can disrupt learning as districts adjust by reshuffling teachers and students well into the fall semester. A recent study and infographic with the School District of Philadelphia explored how machine learning algorithms might improve predictions of incoming cohort size.

Two new videos on culturally responsive teaching and trauma-informed instruction

Culturally responsive teaching means understanding that what students experience outside the classroom can affect the way they learn inside it. In this video, REL Mid-Atlantic shares tips on what it takes to build a culturally responsive classroom.

 

Trauma-informed education means teaching with an awareness that children go through serious, challenging events, and those experiences can affect learning. In this video, REL Mid-Atlantic shares tips on what it takes to create a trauma-informed classroom.

Please reach out anytime at RELmidatlantic@mathematica-mpr.com to share your ideas about important issues the REL Mid-Atlantic could address, request free technical assistance, or ask questions about how we can help you in your work.

Follow Us on Twitter for Latest Updates

This work was funded by the U.S. Department of Education’s Institute of Education Sciences (IES) under contract ED-IES-17-C-0006, with REL Mid-Atlantic, administered by Mathematica. The content of the newsletter does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government.

REL Webinar: Measuring Educational Performance for Improvement

This webinar may be of interest to some readers of this space.

Educators and policymakers are often implored to be “data driven,” but in practice they are drowning in data that may be of limited value for the decisions they need to make. REL Mid-Atlantic has worked with state and local education agencies to develop, refine, and stress-test measures of school performance that provide actionable diagnostic information. This free webinar will present a new framework for making sense of school performance measures, informed by our collaborative work with state and local agencies. We’ll discuss new measures to examine student achievement growth in grades K-3, school climate, social-emotional learning, and the impact of high schools on postsecondary outcomes—all among a range of measures that have the potential to be used purely for informational purposes or as part of a formal accountability system. The framework will help policymakers identify the ways that different measures of school performance are useful—or not—for different policy purposes. Our panelists include: Carolyn Phenicie, Press Secretary, Council of Chief State School Officers (panel moderator) Donna Johnson, Director of Accountability, DC Office of the State Superintendent of Education Evan Kramer, Assistant Superintendent of Data, Assessment, and Research, DC Office of the State Superintendent of Education Tonya Wolford, Chief of Evaluation, Research, and Accountability, School District of Philadelphia Gene Pinkard, Director, Education and Society Program, The Aspen Institute Brian Gill, Director, REL Mid-Atlantic Measuring Educational Performance for Improvement: A Framework for Diagnostic Use of Data December 17, 2021 10:00‒11:00 a.m. ET Register here    ***** The Regional Educational Laboratories (RELs) build the capacity of educators to use data and research to improve student outcomes. Each REL responds to needs identified in its region and makes learning opportunities and other resources available to educators throughout the United States. The REL program is a part of the Institute of Education Sciences (IES) in the U.S. Department of Education. To receive regular updates on REL work, including events and reports, follow IES on Facebook and Twitter. To provide feedback on this or other REL work, email Contact.IES@ed.gov. The Institute of Education Sciences, a part of the U.S. Department of Education, is the nation’s leading source for rigorous, independent education research, evaluation, statistics, and assessment.

By visiting Newsflash you may also sign up to receive information from IES and its four Centers NCES, NCER, NCEE, & NCSER to stay abreast of all activities within the Institute of Education Sciences (IES).

REL Southeast Director’s Email—December, 2021

Note the report below entitled Exploring Teachers’ Influence on Student Success in an Online Biology Course, which should be of interest to readers of this space.

View this email in your browser

REL Southeast Director’s Email

December 2021 Greetings from the REL Southeast, This December marks the last year of our five-year REL Southeast contract period. As such, I’d like to take the opportunity to reflect on our time serving the region. Though the pandemic presented challenges none of us had experienced, the committed staff of the REL Southeast managed to pivot and produce relevant, evidence-informed materials parents could use at home with their students. Similarly, the work of our Blended and Online Learning alliance became a critical resource for many educators who were wading into digital waters for the first time. All the while, our longitudinal studies, reports, and coaching activities diligently continued to take shape in the background. All this amazing work would not be possible without the determination and resiliency of our talented staff and research partners. To them, I say thank you. Below you will find a selection of highlighted products produced by REL Southeast over the past five years. We encourage you to share these resources with colleagues and continue visiting for regular updates. The REL Southeast team looks forward to serving you in the future as we continue the critical work of improving outcomes for all students. With gratitude, Dr. John Hughes Director, REL Southeast Integrating Reading Foundations: A Tool for College Instructors of Pre-service Teachers This resource complements and extends the practice guide, Foundational Skills to Support Reading for Understanding in Kindergarten Through 3rd Grade, by providing lessons to assist college instructors in building pre-service teachers’ knowledge of evidence-based strategies for helping students in kindergarten through grade 3 acquire the language and literacy skills to succeed academically. Development of Mathematical Reasoning Developing mathematical reasoning is a process of constructing mental relationships and connections in increasingly sophisticated ways. This infographic provides an overview of the five major components of mathematical reasoning: counting strategies, additive thinking, multiplicative reasoning, proportional reasoning, and functional reasoning. Exploring Teachers’ Influence on Student Success in an Online Biology Course This study of an online high school biology course offered by Florida Virtual School examined the amount of variation in course completion, students’ final exam scores, and time to completion that is attributable to the influence of teachers. Preparing a Career-Ready Student This infographic presents four practices that were selected by REL Southeast Florida Career Readiness Research Alliance members after reviewing and discussing a literature review on effective career counseling practices. Preparing a School-Ready Child This infographic discusses the four dimensions that must be considered when preparing a child for school and links to valuable additional resources for early educators. Impact of the Developing Mathematical Ideas Professional Development Program on Grade 4 Students’ and Teachers’ Understanding of Fractions The purpose of this study was to assess the impact of the Developing Mathematical Ideas (DMI) professional development program on grade 4 teachers’ in-depth knowledge of fractions as well as their students’ understanding and proficiency with fractions. Systematic review of the literature on the effectiveness of early childhood education curricula and instructional practices on language and literacy development This study reviews 20 years of research on early literacy interventions aimed at improving language, phonological awareness, print knowledge, decoding, and early writing skills for preschool students. Using a process modeled after the WWC methodology, REL Southeast identified 109 studies, representing 132 interventions, conducted from 1997 to 2017 that it determined were high-quality experimental or quasi-experimental studies. Professional Learning Community: Improving Mathematical Problem Solving for Students in Grades 4 Through 8 Facilitator’s Guide REL Southeast developed this facilitator’s guide on the topic of mathematical problem solving for use in professional learning community (PLC) settings. The facilitator’s guide is a set of professional development materials designed to supplement the What Works Clearinghouse practice guide, Improving Mathematical Problem Solving in Grades 4 Through 8 (Woodward et al., 2012). The practice guide provides research-based recommendations for teachers to incorporate into their classroom practice. The facilitator’s guide is designed to complement and extend the practice guide by providing teachers in a PLC setting with additional, step-by-step guidance for the best ways to implement some of these evidence-based recommendations. Professional Learning Community: Emergent Literacy The goal of emergent literacy instruction is to teach the building blocks that will, in later grades, provide children the foundation needed to become proficient readers, writers, and communicators. Emergent literacy building blocks, or skills, include print knowledge, phonological awareness, vocabulary, and oral language. Preschoolers who learn these skills are less likely to develop future reading problems and more likely to read with ease, understand what they read, and succeed in school. Supporting Your Child’s Reading at Home The videos and activities on this website provide families with information about how to support children as they practice foundational reading skills at home.

You are receiving this email because you opted in at our website or provided your information at a REL Southeast sponsored event. Our mailing address is: REL Southeast at Florida State University 2010 Levy Avenue Suite 100 Tallahassee, FL 32310

Home Visits Reduced Disciplinary Incidents and Improved Attendance

I’ve said this lately when it comes to some of these REL items, but I wonder what this would yield if it were done in a K-12 distance or online learning context (even what it might look like).

Click here to view in browser

Home Visits Reduced Disciplinary Incidents and Improved Attendance

Engaging parents in their children’s education can improve education outcomes. Although home visits have emerged as a promising way to boost parent engagement, there is limited evidence on their effectiveness.

REL Mid-Atlantic partnered with the District of Columbia Public Schools to evaluate the impacts of teacher home visits conducted through the district’s Family Engagement Partnership program. The program trains teachers on engaging with families, including how to conduct relationship-building home visits, and provides financial incentives for teachers to conduct the visits. The study measured the impacts of the teacher home visits on the disciplinary incidents and attendance of students in grades 1–5.

 

Key findings include the following:

• Home visits reduced the likelihood of a student having a disciplinary incident later in that school year. In the school year immediately following a summer home visit, 9.27 percent of visited students had a disciplinary incident compared with 12.22 percent of non-visited comparison students, a statistically significant difference.
• Home visits slightly improved student attendance. Attendance rates averaged 95.28 percent for students who received a summer home visit and 94.93 percent for comparison students who did not, a statistically significant difference.

Read the infographic and report.

 

This work was funded by the U.S. Department of Education’s Institute of Education Sciences (IES) under contract ED-IES-17-C-0006, with REL Mid-Atlantic, administered by Mathematica. The content does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government.

Calling All Districts and Schools: Here’s How to Collect Rigorous Evidence to Know Whether Your ESSA Tier 4 Intervention Works

This REL item may be of use to some readers.

Click here to view in browser

Calling All Districts and Schools: Here’s How to Collect Rigorous Evidence to Know Whether Your ESSA Tier 4 Intervention Works

By Gina Lucchesi

The Every Student Succeeds Act (ESSA) encourages schools and districts to use evidence-based interventions and practices, with tiers of evidence based on the strength and rigor of the research supporting the effectiveness of the intervention or practice in improving student outcomes. The American Rescue Plan (ARP) has boosted the importance of the evidence tiers by requiring that 20 percent of the associated funds districts spend must meet one of the tiers. An intervention backed by research might meet the requirements of one of the top three evidence tiers, depending on the rigor of the research.

 

But ESSA also recognizes that many promising interventions have not been rigorously studied, and that requiring rigorous evidence for all interventions would make innovation impossible. That’s why ESSA includes a fourth tier. Tier 4 interventions do not need to be supported by results from a previous study so long as they include a logic model based on high-quality research. This gives educators the flexibility to innovate, developing approaches they believe will be successful in their local context. Because lots of innovative interventions have not yet been carefully studied, Tier 4 is potentially very useful. But because Tier 4 interventions are not supported by strong prior research, they should include a plan to study their effectiveness. By including a plan to study the interventions they are implementing, schools and districts not only can meet expectations of the American Rescue Plan, but also contribute valuable knowledge to the evidence base of what works in education.

Tier 4 interventions must be supported by a well-defined logic model. Refer to this REL Pacific tool and slides 23-30 of this ESSA tiers training from the Institute of Education Sciences to learn about logic models and how to create them.

Collecting evidence on whether an intervention is succeeding can be a challenge. One of the biggest stumbling blocks is a study design that can give you confidence that any change in outcomes results from the intervention itself (rather than from other factors). If reading proficiency improved after you implemented a supplemental reading program, how do you know the improvement wasn’t caused by something else?

 

Using a comparison group is the best way to demonstrate that any change in student outcomes results from the intervention itself. The biggest challenge is finding an appropriate comparison group, so that it is possible to compare results for those who participate in the intervention with those who do not. This can pose operational challenges because educators and policymakers typically want to give promising interventions to everyone. But there really isn’t any good way to know whether an intervention has worked without creating some sort of comparison group that does not get the intervention.

 

Here are three ways you can create a comparison group:

A.

Randomly assign students or teachers to treatment and comparison groups before you begin to implement the intervention.

 

A randomized experiment—the gold standard in research design—uses a lottery to determine who is in the treatment group and who is in the comparison group. After the treatment group receives the intervention, you can compare student outcomes across the treatment and comparison groups to analyze whether the intervention is working. The treatment group receives the intervention first, and other students may receive it later. (This is helpful if you would like to use a small-scale test to see whether your intervention is effective before rolling it out on a larger scale.) The free Evidence to Insights Coach, or e2i Coach, is a digital tool that helps educators generate rigorous evidence and can help you with the randomization process. Using a randomized experimental design can also make your work eligible for addition to the What Works Clearinghouse and inclusion in one of the higher tiers of evidence!

B.

Use a matched comparison approach.

 

If you can’t randomize—or if your intervention is already underway—you can identify a “matched comparison” group. This involves looking at administrative data to select a group of students who are not receiving the intervention but are similar to the students who are. The comparison group students should be in the same school or district as students receiving the intervention and should be similar to them in terms of demographics and achievement levels before the start of the intervention. Like randomization, comparing outcomes in the intervention group to those in the “matched comparison” group helps ensure that any changes resulted from the intervention rather than from other factors. The e2i Coach can help with analyzing data for matched comparison groups as well as for randomized studies.

 

To identify your matched comparison group, we suggest partnering with district or state education agency personnel with data or statistics experience. The e2i Coach can also identify matched comparison groups.

C.

Compare outcomes for students just above and below a threshold that triggers an intervention.

 

Let’s say your school district refers students to an attendance intervention if they are absent for 10 or more days in a single semester. You can compare outcomes for students who just barely made it into the intervention group (10 or 11 absences) to similar students in the comparison group (8 or 9 absences). This method is called a regression discontinuity design.

Although collecting rigorous evidence may be challenging, it pays off over time. It will help you determine whether your interventions are working and decide how to invest in the most effective approaches for your students. Along the way, you may produce research that falls under ESSA’s Tiers 1, 2, or 3, spreading evidence on what works in education across the country.

Click here to tweet about @RELMidAtlantic

Contact Info

For more information, please contact us at RELmidatlantic@mathematica-mpr.com or visit our website.

Follow Us on Twitter for Latest Updates

This work was funded by the U.S. Department of Education’s Institute of Education Sciences (IES) under contract ED-IES-17-C-0006, with REL Mid-Atlantic, administered by Mathematica. The content of the newsletter does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government.