Why “No Difference” Matters to Us All

Sign up

  • Required*

    Read our privacy policy here:

Thank you.

You will be hearing from us shortly

May 22, 2012

Ithaka S+R study tests potential for interactive online learning systems to improve learning outcomes and reduce costs in higher education


“Interactive Learning Online at Public Universities: Evidence from Randomized Trials”

William G. Bowen, Matthew M. Chingos, Kelly A. Lack, and Thomas I. Nygren


May 22, New York, NY
Higher education, especially in the public sector, is increasingly short of resources, and students continue to bear the cost burden. Can sophisticated, interactive online courses using machine-guided learning help educate more students successfully at a lower cost? “Interactive Learning Online at Public Universities: Evidence from Randomized Trials,” a new ground-breaking Ithaka S+R study, finds that student learning outcomes in a sophisticated interactive online course (using machine-guided learning and limited face-to-face instruction) equaled outcomes in the traditional classroom. 


Seven instances of a hybrid prototype introductory statistics course were offered alongside traditional courses at six public university campuses in fall 2011. The hybrid course was developed by Carnegie Mellon University and relies heavily on machine-guided instruction (what the authors refer to as an “ILO”—or “Interactive Learning Online”—type course). The study included a diverse population of 605 students from the various campuses. Results were consistent across campuses and sub-groups, a population in which half the students were from families with incomes less than $50,000 and half are first-generation college students. 


The report notes that there have been thousands of studies of “online learning,” but the great majority of them are deficient in one way or another—often for reasons beyond the control of the principal investigators. Very few look directly at the teaching of large introductory courses in basic fields at major public universities, where the great majority of undergraduate students pursue either associate or baccalaureate degrees, presumably because very few ILO courses have been offered in these settings. It is also rare that the studies use random assignment techniques to create “treatment” and “control” groups—the approach used in the Ithaka S+R study—which  can reduce otherwise ubiquitous selection effects that make it hard to interpret findings.


Referring to the coefficients measuring the difference between outcomes in the hybrid and traditional courses, Matthew Chingos, one of the report’s authors and a fellow at the Brookings Institution said, “These are well-estimated zeros, and the randomization enables us to be confident that the results accurately reflect the impact of the ILO course relative to the traditionally taught course.” Another one of the authors, William Bowen, added, “This seemingly bland finding is in fact highly consequential. Generalized worries that all kinds of online systems will inevitably hurt learning outcomes do not appear to be well-founded.”

"I know of no study of online learning that equals this one in its rigor and its relevance to practical application,” stated Michael McPherson, president of the Spencer Foundation. “It definitely shows the impressive potential of truly interactive learning systems to reduce educational costs while sustaining quality in subjects that lend themselves to such treatment."


Potential is the operative word. While the research demonstrates that ILO-based learning systems need not negatively impact learning outcomes, the report also cautions that these systems are in their infancy and encourages vigorous efforts to explore uses of both the simple systems proliferating every day and the more sophisticated ILO-type system studied here. “I know of no online learning system available today that can deliver significantly improved educational outcomes across the board at scale,” noted Mr. Bowen. “Yet there is every reason to expect these systems to improve over time, perhaps dramatically, and it would not be foolish to believe that learning outcomes will also improve.”


"Because of its rigor,” commented William (Brit) Kirwan, chancellor of University of Maryland System, “this study offers great hope that at long last technologies are becoming available that will provide high quality education and simultaneously bend the cost curve.  After reading the report, I have renewed optimism about the potential to address our nation's higher education challenges."  


About Ithaka S+R

Ithaka S+R is a strategic consulting and research service that focuses on the transformation of scholarship and teaching in an online environment, with the goal of identifying the critical issues facing our community and acting as a catalyst for change. Ithaka S+R is part of ITHAKA, a not-for-profit organization that also includes JSTOR and Portico.


Media Contact

Marita La Monica
Senior Marketing Manager

More Info