Resources

So thanks, Monica, very much for joining me today. I'm going to begin this conversation by acknowledging that I'm joining you from Aboriginal land, from the land of the Gadigal people of the ER nation, and pay my respects to elders past and present and my respects to all First Nations people. So we're here to talk about your study which explored scaling intensive tuition through the addition of a computer based learning application. And my first question to you is what prompted you to undertake this study?

Sure. Well, first, thank you so much for having me. I'm, I'm, I wish I could be there in person, but alas, the world is large. And I am thrilled that technology is facilitating my ability to be there in the room with you today. And it, it really is at the heart of this because technology can facilitate the so many things in our society, including the advancement of teaching and learning so that all students benefit. And yet there are some guardrails that I think we we want to place along that

we wanted to explore that as part of the study. So for the past decade or so at the University of Chicago Education Lab, we have been looking at the effects of what we call high dosage tutoring. So this is a tutorial where one tutor works with a few students in the context of the school day as part and parcel of how instruction is delivered during the school day.

And we wanted to understand the impacts of that intervention on student learning. So about 10 years ago now, we did a series of randomised control trials, which is the most rigorous type of study that you can do to understand the causal impact of one sort of intervention or programme on outcomes. And we saw that when students meet with a tutor in ratios of two students to one tutor, it is enormously beneficial to them. And in this case, it was a study of 9th grade algebra, which seems to be a gatekeeping course, at least here in the United States and I imagine in Australia as well, to later life outcomes. We saw that it was so beneficial and very cost effective. So we went running for the hills and told all of our public sector partners in Chicago, across the country, at the Departments of education, at various States and districts. And we said, you know, we found the educational equivalent of nuclear fission. This is how we're actually going to address these longstanding disparities in achievement that we see along race and class lines that have been very persistent in the US educational context. And they said that's wonderful. It is cost prohibitive because it costs $3000 per student, which is about 1/4 or 25% of the per pupil expenditure of Chicago Public Schools annually.

That's too high to spend on one student and one grade in one subject. And so they sort of tasked us with trying to find an intervention that was just as effective, but cheaper and less costly. And we said to our partners at the time, Saga Education, who was the programme delivery partner, there's sort of a best in class nonprofit tutoring provider. We said, how can, how can we do this? And they went off and did some design work, and they came up with an interventional design that was for students sitting at a table with one tutor, you know, similar sort of structures. It was an elective course called Math Lab. The kids went every day. It was part of their schedule. So none of that changed. But you, for the same amount of money for the same tutor, could serve twice as many students. And in order to facilitate that larger group, the students spent every other day on an educational technology platform. In this case, it was Alex. But you could sub in any kind of adaptive educational technology platform that helped you differentiate instruction. And we said, does this work? Now oftentimes in social science, when you reduce the cost of something, you reduce the efficacy as well. And so we wanted to understand, could you cut the cost in this case? It cut the cost of this very effective tutoring intervention that doubled or even tripled what a student learned in mathematics in one year compared to their peers down the hall in the same school in the same grade. We said, can you cut the cost of that by 30 to 50%, let's say 30% just to be really conservative here and maintain the effectiveness. And what we found was that it was just as effective. So I know we'll get into the the study findings in a minute, but we were really motivated by whether you could sort of use this design recipe of increasing the ratio of students to tutors just slightly and then leaning on the educational technology to facilitate that scale and still mimic the differentiated instruction. We wanted to know, could that interventional design work?

I thank you. I love that phrase. Leaning on the educational technology, helping it to leverage for equity, which you have described as being at the very heart of of this study. Now is it fair to say that boiled right down what your study found was that by introducing the technology platform you could effectively double the number of students that the programme reached, keep the same the number of human tutors, which is a really big consideration in a kind of resource constrained environment. Reduce as you've just said, the per cost student significantly and yet not compromise the significant learning outcomes that you've seen in the previous human only study. Is, is that correct?

That's right. So we saw increases of .23 standard deviation. So I apologise for using a term like that. But we saw basically no difference in the amount of learning for a 9th grade student in algebra, whether they had the sort of SAGA technology enabled version or the prior version where it was two students working with one tutor. And so again, we were really thrilled by this because it provided a road map or a recipe, if you will, of how we could maybe think about taking this very effective intervention delivered by a boutique vendor at modest scale, maybe a few thousand students and really think about how we could reach millions of students, you know, across the country and even across the the world. And you said you were thrilled, which I can absolutely understand. Were you surprised?

Um, you know, I think something that people don't appreciate as much is that there was a lot of design work and iteration that went into the, the intervention itself done by Saga Education prior to us studying its efficacy. And so personally, Saga had sort of given their stamp of approval and said, OK, this is the version to test. We had done other iterations that ended up on the cutting room floor. So we had tried 1/2 dose version where kids met every other day.

But in the context of a school building that's very confusing for kids because they have a new schedule every other day and principals didn't like that very much, neither did the teachers. But in this version it was on the student schedules and it was expected by teachers, tutors and students alike what what was going to happen. And so they liked this version better than the sort of alternate day. So we tried lots of different iterations. And so I don't know that I was surprised that it was effective. I am surprised that it was just as effective that there was no deterioration in treatment effects just because that's something we we never see in social science. And it's, it's really heartening because it means that technology can facilitate scale. And now we have to get even more sophisticated in our research questions and our practise and understand under what conditions. So that segues very nicely. The next thing I'd like you to talk about is, in your view, what were the primary factors that supported the success of this implementation?

Sure. So oftentimes when we think about introducing technology, there are no standards attached to that phrase. So we could mean everything from enabling kids to have a login from the administrative side, like, oh, everybody is able to get a login and a passcode, all the way to what this saga technology intervention was, which is it is a course that had to be scheduled into a student school day. That means time was found, dedicated time for the student to go to a classroom where there was a teacher of record and six tutors. So 24 kids were scheduled into this class period. They met, you know, with a tutor.
The tutors had to be sourced, staffed, hired, managed, trained, and the kids sat at the table with with the tutor. So there was still 4 kids sitting at a table with one tutor. And so even when they're on the Ed tech platform, they would start off by the tutor checking in with the kids both interpersonally. And they had this thing called the do now where they would all work on a math problem is kids filtered in. They would check in with the students about halfway through the class period when the the two students that they were working with we're working on a math problem, They would lean over and say, hey, how's it going over there? Are you, you know, making sure they're not texting their girlfriend or, you know, doodling or, or just staring at the computer screen and not actually engaging. And by having that sort of motivational piece that the human tutor could do, but allowing the platform to do the differentiated instruction and get kids to practise math problems, which is miraculously what gets kids to learn more math is, is that practise? I think that interventional design and those kind of component pieces were really important because the kid wasn't just, you know, on a zoom with a black screen. They were, they had a, a human tutor who expected them, who was motivating them and they had access to this technological platform. I think that's just a very different interventional construction. Then I'm going to give you a slip of paper with a login and you can log in on your own time. I worry a little bit about that use of technology in that construction because we know that the kids who are more likely to log on are the kids who are already doing better in school, or at least we have mounting evidence that that's true. Would you just that that's effect is described in your paper, we sometimes call it the Matthew effect of, you know, increasing advantage, building on advantage. Would you describe a little bit more about what you did find about those different patterns of edtech usage?
Sure. So I should be really clear here that there is a lot more work to be done on this topic. So I'll try to stick to what this study showed and then try to differentiate that from my opinion. So in this study, we had the technological usage of the kids who logged on to the Alex platform. So by definition, these were all the kids who in our random assignment trial were assigned to receive tutoring. And the tutoring was inextricably linked to logging on to the Alex platform. And even in that constrained design setting, when we looked at the relationship between students baseline characteristics and Alex usage, we found that kids who had higher test scores and higher GPA's were more likely to log on to Alex and use it more, learn more topics than students who didn't have those characteristics. Now, is that causal evidence? Absolutely not, because this is just descriptive in that sort of one, one use case from our study, but I think that is the pattern that is replicated in other studies as well. And we're starting to learn that, you know, left in the wild. And now I'll sort of veer into my hypothesis. We see that kids who sort of log on and do their work either already have the study skills, the motivation, the attachment to schooling and also the inclination to do that. And so they're the kids who kind of had higher baseline test scores and GPA's to begin with. And I think we have mounting evidence of that from the COVID-19 pandemic in which we have a lot of data coming from remote instruction and you can see who logged on and who dropped off. And and so I, I would sort of count that evidence from the remote instruction during the pandemic as as substantiating this hypothesis.
Thanks very much for that, Monica. And I think that moves us kind of naturally to the next question that I'd like to ask you, which is were there any aspects of this particular study that you feel warrant now further exploration? You already mentioned the interest in evolving studies. What are they? What would you still like to know in this space?

Sure, I think there's so much to know. So I I think I'm heartened by the fact that we know that this pedagogical practise of tutoring, individualising instruction really is the best way for anyone to learn anything. So if I were to take a piano, I could sit in a class, I could take a watch online YouTube videos, I might learn the piano. I'm much more likely to learn the piano in a way that I get better faster if I had a coach or a teacher come to my home and teach me one on one. Take what I already know from my, I don't know, 30 years ago piano tutorial experience, and then build on that and teach me in a consistent way. I think what we've never been able to figure out in public education is how do you take that pedagogical practise and make it so that you can fit it into the parameters of what the public education system can afford and how the system is set up where we, you know, typically have one teacher with 30 students. I think we have some evidence and and I would say that this study provides a proof point that technology can facilitate scale. I think what we don't really have are practical answers to policymakers questions about how do I do this in my own district? What is the first thing I stand up? How do I find these tutors? How do I make it so that they're allowed to be in a classroom with students, even legally without a teacher of record there? Or maybe with a teacher of record, allowing that teacher to, you know, be the sort of architect of a classroom where the the kids are getting some individualised instruction tutors, some are getting an individualised instruction from platforms. And then the teacher can use his or her pedagogical expertise to really work on the the more difficult problems of practise. That's the classroom of the future. I think we're still building towards that. And we don't know how to take all of these effective practises in their pockets and put them together so that they create a system that makes sense for students, you know, regardless of a creator background. And so from my vantage point, I want to understand what are those ratios of students to tutors at which you can't rely on technology anymore to facilitate scale? You know, can you have six kids? Can you have eight kids? Can you have 10 kids? Then you start to get into some of those classroom management problems that you had earlier. And I think a lot of the focus that we're, you know, really, really concentrating on for the next few years is what is the right mix of human capital and technology to facilitate that individualised instruction.

So I don't know the answer yet, but maybe I can come back in a few years. Thank you. I like that phrase. The the best mix of human capital and technology. One thing that did strike me from your paper was, was is there as the question of is there a sweet spot for the amount of tech technology used by either students in general or individual students? Could you say something about that?

Sure. So you know, I think during the pandemic, we saw there was too much technology. So the sort of like log on to Zoom for 8 hours a day and you know, have it teacher talk at you is not the kind of technology that seems to facilitate genuine learning opportunities. And what we hear a lot is, you know, kids are burnt out on technology. Now, if you think about the, the a different situation, maybe just prior to the pandemic, where we actually do see that technology helps kids learn. If you use the tech platform in the right way, it actually can help kids learn. And so if you're not using it enough, we're actually leaving learning on the table. We're we're using technology in the sub optimal way. And that's, you know, malpractice because we have all these students who are just begging to learn and we have very inequitable outcomes kind of not just within our our individual countries, but sort of globally that we need to kind of think about. So we can't afford to leave technology on the table or sorry, learning on the table that technology could facilitate that's opportunities and advancements that we are leaving on the table. And so I think what we're trying to do is find that sweet spot where, where it's the, uh, the optimal mix again, of both the human capital that you have at your disposal and then all also the, the technology and the technological modalities and platforms that you might be able to deploy. And something that I think is likely obvious to everybody in the audience, but should be stated is that I think that that right mix is highly context dependent. So if you're in a rural area, you might have to lean on the modality of technology more, but then you really need to think about how you're going to use your human capital to kind of overcome the Zoom fatigue that we saw during remote instruction. That might not be as true if you're in a more densely populated area, but you might have other challenges as well in terms of equitable, equitable distribution of that human capital and those opportunities. And so I think we just have to get a lot more granular, not just about what works on average, but sort of what are the overarching design principles as we've been talking about? And then how do you adapt them in your local context and still be confident that they're yielding the treatment effects that that we all want on student learning?

Well, thank you so much, Monica. My last question was going to actually be about why is, why is your study and why is technology more broadly important, particularly for students experiencing disadvantage and the schools that serve them? But I think your whole, everything you've had to say and share with us has been from that foundation of, of equity and needing to do to do better. And I think we all know that we can't leave any of the things on the table when we're trying to do that the the problems two pressing and to to large to overlook any of the potential levers. So that just leaves me to say thank you so much once again for your insights and for your time. Thanks, of course. Thank you for having me. Thank you.