News + Ideas

Exciting New Evidence about the Promise of Personalized Learning

The number of schools designed for personalized learning is growing rapidly. At NewSchools, we’re helping teams of educators launch many such schools through our Catapult program.

Why did we decide to focus here? One reason is early evidence of the positive effects personalized learning has on students. What do we know so far? The RAND Corporation recently released the findings of the third year of a study of schools implementing personalized learning.[1] The quasi-experimental study follows 11,000 students in 62 schools for two years and includes additional analysis on a subset of 21 of the schools that have three years of data. Among the 11,000 students, 75% are Black or Latino and 80% are low-income. It’s the largest and most rigorous evaluation to date of schools designed explicitly for personalized learning.

Below are some observations about a few of the study’s findings, organized as a set of questions most interesting to me:

  • Do students in personalized learning (PL) schools demonstrate more learning growth than students like them in traditional schools?
  • Do students in PL schools outperform students like them regardless of their incoming performance?
  • Given that 56 of 62 schools in the study are charters, is their superior performance actually associated with choice rather than personalized learning approaches?
  • What do we know about how students in PL schools fare on indicators of student success beyond reading and math scores?

Question #1: Do students in PL schools demonstrate more learning growth than students like them in traditional schools?

Yes. The study uses a virtual comparison group (VCG) approach, the most rigorous methodology possible for this group of schools. In all grade levels, students in PL schools did better than students like them in traditional schools. The effect sizes in elementary and middle school math and reading are positive, large and statistically significant. In high school the results are positive and moderate, but not statistically significant due to small numbers of students in some of the high schools in the sample.

chart1

As in prior years of the study, the authors struggle to convert the effect sizes into plain language. They estimate what the effect size means in terms of the median percentile gain on the MAP exam, then show the difference between this and an estimate of what the median gain would have been in a non-PL school. These differences are impressive, but the metric is still pretty abstract for most of us to really grasp the magnitude of the results. The study misses a chance for easy-to-understand comparisons to other well-known studies. CREDO and Mathematica convert effect sizes for school evaluations into “additional days of learning.” RAND does this in other studies, too. It’s a little more complicated since the PL results are based on growth rather than absolute scores. But we just need an estimate. I hope some enterprising researcher will take the PL data set and convert the effect sizes into “days of learning” so we can more easily compare the results to other improvement strategies. Anyone know an ambitious economics of education doctoral candidate who is up for the challenge?

Question #2: Do students in PL schools outperform students like them regardless of their incoming performance?

Yes. Students in every performance quintile outperformed students like them in non-PL schools. Students who started out furthest behind gained the most, which is important if they are to catch up and eventually surpass grade-level expectations.

chart5

But what about students who aren’t behind to begin with? In prior years of this study and in some other PL evaluations, fewer than half of students at the top did better than similar students in non-PL schools. But not this time – more than 60% of students in the top quintile outperformed other students like them in the VCG. This is an important finding. One of the aims of personalized learning is to meet every student where they are and help them reach their full potential. It’s possible that as the field matures from the earliest pioneers to more established approaches, the models are strengthening their ability to serve all students better.

Question #3: Given that 56 of 62 schools in the study are charters, is their performance actually associated with choice rather than personalized learning approaches?

Probably Not. For each of the 11,000 students in the study, RAND researchers created a VCG of 50 students from the MAP database who are as similar as possible in starting achievement and demographic characteristics. This design allows for the kinds of comparisons I described earlier and for a variety of sensitivity analyses. To control for the fact that so many of the PL schools are charters, the RAND team created an additional VCG comprised only of students from schools of choice.

choice

Students in PL schools out-performed the choice-only VCG in math and reading. In fact, the 3-year sample did a little better against the choice-only VCG (blue) than the baseline VCG used throughout the study (red).

These data don’t address every question about whether the charter schools in this study would be superior to the baseline VCG even if they were not designed for PL. But as the RAND team put it, they “do not cast doubt” on the main findings that students in the PL schools did better than students like them in non-PL schools, including other schools of choice.

Question #4: What do we know about how students in these PL schools fare on indicators of student success beyond reading and math scores?

Nothing. This might be the biggest limitation of the study and is a huge opportunity for future studies. Most of the teams launching new schools or redesigning existing schools for personalized learning are aiming for an expanded definition of student success. One that goes beyond helping students score better on reading and math tests. They are focused on helping students develop the agency, skills and mindsets that will help them succeed over the course of their lives. If the RAND study continues beyond year three, I hope they will incorporate these kinds of measures.

As we think about our research and evaluation agenda at NewSchools, generating useful evidence about this expanded definition of success is a top priority.

 

[1] Disclosure: I led the team at Gates that funded these kinds of schools and commissioned the RAND study 3 ½ years ago. I left Gates 16 months ago and the study has continued under new leadership. I did not have access to the study’s data or analysis before its public release on November 10th, 2015.

2 Responses to “Exciting New Evidence about the Promise of Personalized Learning”

  1. I strongly agree with your statement “As in prior years of the study, the authors struggle to convert the effect sizes into plain language.” One of the major problems in education research, especially as funded by large philanthropy, is that the findings are inaccessible to so many on the ground – working day to day – in schools around the nation. It has been clear for decades that self paced personalized learning is effective. That is why the private sector adopted it back in the late 80s and 90s. I was one of the pioneers back then.

    But in K-12 education adoption of personalized learning would go a long way to closing skills gap for many children of color and providing enhanced rigor for many beyond grade level children. But the pace of adoption is slow because the process of adoption is misunderstood and the real definition of personalized learning not well formed. There are also may Ed Tech charlatans pitching tools without understanding true impact or integration into the daily practice of schools and teachers.

    Although this study shows what many in the field already know – personalized learning, blended learning and appropriate use of technology can produce significant gains – using standard deviation units of growth and impact goes way over the heads of most readers. In past CREEDO reports, they translate their results into days of learning but that too is a fuzzy number for many.

    Having run a pilot (New Orleans) where we introduced blended learning in an experimental design with 1 target and 3 control schools, we captured the results in 2 ways. Using standard education research measures but also using rich media like video and blogs to make it accessible for those closest to the kids. (http://edinquiry.com/?p=572).

    There is a major chasm between education research and the communities of color that need to understand what works and what does not. My hope is that by adding more diversity to these education research teams and by enhancing the community of education research with people of color the message will be transformed from impact scores, standard deviations unit and the like to something meaningful for school leaders far removed from statistics but on the front line of change in our lower income school.

    We need more pilot projects, well documented, to help people understand what works and does not. We need more diversity in research teams to help with the translation of findings. I continue to hope that education leaders will realize the importance of messaging and hearing perspectives beyond those of the research team too often devoid of people of color.

  2. This is good news and has come at the right moment to give food for thought to the sceptics. Not everyone believes in the power of adaptive or personalized learning. A few have never seen tech as the way to solve the personalization problem in education and hopefully these results will help them re-analyze their beliefs.

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>