The edtech sector has lots of generous benefactors, including charitable foundations funded by tech titans like Facebook founder Mark Zuckerberg and Microsoft founder Bill Gates.
But do edtech products actually, provably help kids learn?
Anecdotally, the answer is yes. Scientifically, though, it’s more of a maybe. Randomized or quasi-experimental studies supporting edtech products are a rarity, and most edtech products hit the market before their impact on students has been rigorously tested.
In 2015, President Barack Obama moved to change that by signing the Every Student Succeeds Act, or ESSA, which laid out research-based criteria for gauging the effectiveness of edtech and other classroom measures.
The highest rating, “Strong,” indicates that a product or curriculum has statistically significantly improved student outcomes in at least one randomized and controlled study. Nonetheless, only a select few edtech products have received that “Strong” rating, according to Johns Hopkins University’s Evidence for ESSA site.
Meanwhile, personalized learning software like the Summit Learning platform (funded by the Chan Zuckerberg Initiative, named for Facebook's Zuckerberg and his wife, Priscilla Chan) has no ESSA rating. It provides students with personalized curricula and weekly mentorship sessions, a format backed by academic research, Zuckerberg notes. The research to which he refers, however, suggests that humans learn best with one-on-one attention from a tutor — not with Summit specifically.
That doesn't make Summit unique. Plenty of other edtech products — including those used for student evaluation — are supported by anecdotal case studies, but haven't been subjected to controlled and randomized trials. So while the sector continues to evolve, edtech skepticism abounds.
What are the potential pitfalls of fusing education and technology? We spoke with three edtech experts to learn more.
Vincent Quan
Associate director of policy at the Abdul Lateef Jameel Poverty Action Lab focused on the intersection of education and technology, and co-author of "Education Technology: An Evidence-Based Review."
In your work, you focus on translating academic research into evidence-based policy. Do you feel that the way edtech is implemented in American schools right now is typically evidence-based?
Not so much. I don't know the exact statistic on how many school administrators consult rigorous evidence before adopting a new education technology program, but I recall that a very low percentage of them consult rigorous evidence before making an adoption decision. Generally speaking, my sense is that most school administrators rely on word of mouth from other administrators. So they rely more on anecdotes so far than on evidence.
Why do you think that is?
I have some theories about the barriers to evidence. First of all, evidence can be hard to access, and even when it’s available, it's often not written in a way that's accessible to people who don't have a PhD. Academic researchers who publish edtech studies in journals aren't writing for policy makers or practitioners, they're writing for a research audience. So they will use terminology, like “p-values,” that’s not necessarily understandable to folks without a particular background.
Another barrier is that a lot of education technology companies don't feel like they have incentives to actually have their programs evaluated and the school administrators often feel like they have no choice. A lot of these programs are not backed by evidence, and they feel like they have to implement an education technology program.
Where do you think that perceived pressure comes from?
I'm not sure. My sense is there might be pressure from the edtech companies themselves. They’re often faced with incentives to scale up rapidly. And then I think that there is sometimes a belief that any technology is better than no technology at all.
When you were working on the evidence-based review of edtech, did you feel that there was an overall dearth of evidence?
We actually found a decent amount of evidence around some form of the education technologies. The review ultimately looked at over 120 experimental studies so there's definitely a decent amount. But I think that a lot of very popular forms of education technology were not included in that review because we couldn't find any rigorous evidence about that specific program. One example is Khan Academy [an online education resource]. We didn't find a randomized evaluation specifically looking at the impacts of Khan Academy versus no Khan Academy.
Do you think that the lack of research on popular edtech has anything to do with funding? Do investors fund edtech more than research on edtech?
Yes, I definitely think that there is some disparity between investment in all of those edtech programs versus investment in evidence of their efficacy. That's definitely a cause for concern in the edtech industry more broadly.
Folks who are investing in for-profit ventures want to be able to see financial returns from the investment, so I think they look more closely at a product’s ability to scale rapidly than evidence of impact. I also think that there's a bit of a misconception that rigorous evaluation takes too long for companies to be able to pull off.
In your review, did you find any particularly promising technologies?
Educational software designed to help students develop particular skills have shown enormous promise in improving learning outcomes, particularly in math. Technology based nudges, such as text message reminders, can have modest but meaningful impact on a variety of education-related outcomes, often at a low cost.
Kentaro Toyama
Professor at the University of Michigan School of Information and author of Geek Heresy: Rescuing Social Change from the Cult of Technology.
Since you first got interested in edtech, back in 2004, what are some issues you’ve seen proponents of edtech overlook?
I think people who are optimistic about educational technology tend to start from this assumption that education is about absorbing information and knowledge and skills—which makes it seem like technology is the ideal tool for improving education.
But I think education is primarily about motivating kids to do the hard work that is the absorption of information or knowledge or skills. If you think of it as a motivational problem rather than as an informational problem, then technology has some things it can do—but no single technology, or even combination of technologies, has what it takes to keep one child motivated for the full length of an education. You need qualified people teaching, too.
The best analogy I can give is exercise. All of us know we should be getting moderate amounts of exercise, but most of us aren’t. It doesn't make a difference how cool a treadmill we have. We need somebody that's going to get us motivated to do the running. Running is something you can't have any machine do for you.
What do you think of these companies that try to gamify learning? In their apps and software, kids earn stars or some other sort of in-game reward when they master a skill. Do those games motivate kids effectively?
I think gamification is interesting, but we are nowhere near getting a gamified system to the point where it's able to generate, let's say, six hours a day of motivation for 12 years of what might be a basic education. I believe gamification can help motivate a kid for maybe a couple of hours, or maybe even a couple of weeks, but kids get bored very fast.
And then on top of that, I think there's another, more difficult challenge. Let's say gamification succeeds, and somebody gamifies the entire K through 12 educational system. Kids who come out of such a program can only do work in a gamified environment. It’s not generating students who have their own curiosity and want to learn on their own.
Shifting gears a little bit, in a 2015 story you wrote for the Atlantic, you refer to “the Law of Amplification,” which basically says that technology amplifies existing human tendencies. Have you seen technology amplifying any darker human tendencies in the classroom?
Absolutely. Just about any teacher who's tried to use technology in the classroom will have experienced the funny paradox, which is that even if you're trying to teach a class on the technology itself — about, let's say, computer programming — your worst enemy is the technology. Most students have an instinctive desire to learn, but they also have an instinctive desire to distract themselves with unproductive stimulation. Technology amplifies that.
One time I was teaching an after-school class that was specifically about digital literacy for kids in Seattle, and as soon as they opened their laptops, they would go and find games to play. This was after I thought I had removed all the games from the computers and disconnected the internet. They would still somehow find these games. I learned that even in a context where the goal is to teach digital literacy, you had to be very careful about when students were allowed to access their computers.
In the Atlantic story, you mentioned working in rural Uganda. A lot of edtech is made in the U.S., though, and especially in Silicon Valley, by people who don't send their kids to schools that use a lot of technology. Do you feel like there's a disconnect between the people making and using educational technology, and does that pose any problems?
I'm not sure if there's a disconnect, exactly, but there are different sides to people who work in Silicon Valley. When they put on their parent hat, they have a different view of technology in education than when they put on their engineering and marketing hats.
One funny thing about Silicon Valley is that everybody there knows that what makes a startup succeed in the long term is the management team. It’s all about the quality of the team, the human beings that run the company. I've never heard of a venture capitalist investing in a startup because the startup used the right technology. And yet, everything that comes out of Silicon Valley sends this message that “If you use our technology, whatever you're doing will succeed.”
Audrey Watters
Independent edtech scholar and ‘17-’18 Spencer Fellow at Columbia University, writing a book on education technology, Teaching Machines, and running the website Hack Education.
Can you tell me a little about Teaching Machines?
The book tells the story of the education psychologists who, in the mid-twentieth century, designed machines — not computers — they claimed would automate and individualize education. Their efforts weren’t the first time technologies were used in the classroom — film and radio, for example, were introduced decades prior — but they did help establish some of the ways in which textbook publishers and instructional designers began to think about organizing lesson material and assessing students.
I wanted to write a book about the history of education technology in part because so many of today’s edtech entrepreneurs and advocates seem to know nothing about the industry’s past. One of the claims you’ll frequently hear these days is that computers will help students move at their own pace through educational materials, giving them immediate feedback. But that’s precisely what the builders of teaching machines in the 1950s and 1960s said their devices would do too.
The subtitle of your book is, “How automatic education became ‘personalized learning.’” Personalized learning is a huge concept in the edtech industry. Why the airquotes? Can you talk a little about your wariness of the phrase?
"Personalized learning" has a lot of appeal, no doubt. It’s often positioned as a counterbalance to what’s perceived as the "mass education" of our public school system. The idea of individualizing education can be traced in part to John Dewey and other early twentieth-century progressive education reformers who sought to encourage more student inquiry in the classroom as an alternative to practices that gave teachers (and textbooks) total control.
Personalized learning, as designed and marketed by Silicon Valley companies today, might try to invoke some of this legacy. But often, the software has simply shifted the control over what is learned and when from the teacher or textbook to the algorithm. It’s no more “personalized” than are the recommendations that Netflix gives you over what movie to watch next.
In your ideal classroom, what role would technology play—if any?
I think this is the wrong question. When we talk about technology, we often just focus on specific objects or products, but technology is much more complex than that. It involves practices and values and systems and ways of thinking. Technology is also a site of power and control. We need to talk about what it means to hand over the future of school instruction and school administration to industry.
When people talk about the role of technology in the classroom, promoters of edtech talk about a handful of mostly digital tools. They do not talk about the ways in which windows are a classroom technology, as are metal detectors at the school door or surveillance cameras in the hallways. We should ask more questions about what sorts of practices and what sorts of values lead to certain students having windows and others having no natural light in their school buildings; some students walking through metal detectors each morning and having their every movement tracked and monitored; certain students doing "drill and kill" on computers in large classrooms and other students having small classrooms, human teachers, and opportunities for inquiry—with or without computers.
What do you think drives people to try to create “teaching machines?" What makes them want to mechanize or replace human teachers?
Since the public school system was established in the nineteenth century, it has been pressured — mostly by the business community and politicians — to run more efficiently. The idea of automating education should be seen in this light — the demand to educate more students, more quickly, for less money. Reformers have often seen teachers, who are mostly female and highly unionized, as standing in the way of school efficiency. Often the arguments made for teaching machines are full of promises that automation will free up teachers’ time from the drudgery of repetitive tasks like drilling and grading. But few of these arguments seem concerned that students’ work is still mostly drudgery. Just because it’s a worksheet on an iPad doesn’t mean it’s transformational or exciting. It’s still a worksheet.
