The value of MOOCs must be measured by more than just completion rates
Kennesaw State University (KSU) in Georgia has declared its entry into the world of massive open online courses (MOOCs) a success, not by the traditional standard of course completion, but based on branding, student access and return on investment. According to eCampus News, MOOCs have increased the brand visibility for the university and expanded student access to KSU course materials. Researchers also measured the ROI, defined by the number of MOOC participants who later enrolled in a university programme, and in the first year, KSU far exceeded its goal to simply break even.

University sees measurable success outside of traditional standards of academic course success.

Significant return-on-financial-investment (ROFI), massive social networking momentum, and thousands of downloads of one University’s MOOC have led experts to conclude that their MOOC initiative is a home-run success—a success that has nothing to do with completion rates.

Having the benefit of jumping into the MOOC game more recently than a number of other institutions, Kennesaw State University (KSU) began their exploration into MOOCs as the University System of Georgia (USG) entered into a state-university partnership with Coursera to leverage technology as part of Complete College Georgia. KSU simultaneously began an exploration into what the University calls MOOC-Institutional-Fit to better identify the role of MOOCs at the institution.

However, leadership in charge of evaluating MOOCs for the University realized something that perhaps other institutions may not have considered: completion rates are not necessarily the main indicator of success.

In a recent report published in the Online Journal of Distance Learning Administration, authors Dr. Elke M. Leeds, associate vice president of Technology Enhanced Learning and associate professor of Information Systems for KSU, and Dr. Jim Cope, executive director of the distance learning center at KSU, mention that in Stephen Haggard’s “The Maturing of the MOOC,” many state and public institutions have not yet been able to qualify or quantify the value proposition in order to justify engagement with the open course market.

Therefore, leadership at KSU made sure to ask the central questions of: “What resources and capabilities do we need? Why us? Why now? What will be the value to the institution and our students? How will we measure success?”

The authors mention that even though faculty’s main concerns focused primarily on completion rates, academic integrity, and the equitable value of a MOOC to a for-credit course, these traditional standards of success may not indicate a MOOC’s success.

“Measures of anticipated success are based on measures of academic course success, when in actuality one is not necessarily a substitute for the other,” say the authors. They mention that in Reich and Ho’s “The Tricky Task of Figuring Out What Makes a MOOC Successful,” course drop-outs are not a breach of expectations, but rather a natural result of the online, open environment.

Measuring Success Outside of Completion

ROFI

Launching the University’s first MOOC in January 2014—a K12 Blended and Online Learning MOOC for teacher professional development and continuing education that offered completion certificates at the community level, professional learning units (PLUs) at the professional level, and fee-based credit transfers at the credit-seeking level—leadership conceptualized a financial break-even assessment that was conducted to determine the number of learners that would be required to enroll in a downstream program to offset initial investment. The number was determined based on current tuition rates, costs of production, faculty release, and graduate teaching assistant costs. The number of students enrolled needed in the associated graduate endorsement program (a subsequent two-year commitment) to cover all costs is four.

Though the first iteration of KSU’s MOOC saw enrollment decrease weekly (of the close to 6,000 learners initially enrolled, 40 percent were active in week 2, which dipped to 6 percent at the conclusion of the 8-week guided course), 100 professional learning unit recipients and 14 program applicants emerged after course completion, and 12 subsequently enrolled, exceeding break-even expectations.

Branding

MOOC leadership also evaluated the number of unique visitors, social media mentions, new Twitter followers, and course-institution awareness through grass-roots efforts. Over 25,000 Twitter hashtag tweets and re-tweets were documented, say the authors, and this year the University measured the number of learners who were affiliated or previously affiliated with KSU, heard of KSU and/or never heard of KSU. More than half had never heard of the institution, and another 25 percent had heard of the institution but were largely unfamiliar with it.

“Yet, each of these learners meaningfully engaged with materials produced and made available by KSU,” note the authors.

Student Access

At the PD level, KSU evaluated the number of video streaming views, unique viewers, material downloads, first-level completers, PLU applications, and course completers. In 2014, the MOOC recorded over 80,000 video lectures viewed and close to 4,000 unique viewers over a 10-month period. The 2015 offering, after only 6 weeks, has already recorded over 25,000 unique lectures viewed, 28,000 streaming views, and over 6,000 downloads, say the authors.

KSU also measured the number of registrants, portfolio submissions, completion certificates awarded, and program applications and enrollments at the credit-seeking level, and “determined that leveraging the value of this MOOC included all of the following: re-credentialing Georgia teachers, increased access to education; scaled learning of traditional online classes; student-student, student-instructor, and student-content engagement; innovative technology testing;” progress in research, application of design to credit hour courses, and proficiency in open educational digital resources, the authors explain.

These “unexpected measures of success [in the initial 2014 MOOC offering] result[ed] in the subsequent re-launch in 2015 with a different perspective in mind,” say the authors. “The three level framework of community, professional, and credit-seeking were left with lasting digital impressions, branding, and reach beyond expectation, and notable benefit to the university, college, faculty and students.”

Success Going Forward

While KSU’s original MOOC resulted in 12 students enrolling in graduate programs, a survey of participants in the MOOC’s second iteration show that over 100 participants plan to enroll in the Bagwell College of Education’s graduate program, which the authors believe to be a “very positive recruitment strategy” of the MOOC, and will “bring in substantial revenues to the University and College while extending the brand and fostering the successful implementation of new pedagogical models.”

The authors also note that outside of projected revenue increases and cost savings for students, the MOOC initiative had yielded educational innovations. For example, the MOOC produced an award-winning open educational resource currently under review by MERLOT, and made possible from a collaborative partnership with a leading international consortium on online education.

Also, “The K12 [MOOC] created a research-rich environment that yielded a textbook transformation award, conference proceedings, peer-reviewed journal articles, promotional videos, along with 13 presentations accepted and delivered in a one-year span,” say the authors.

“The traditional measures of success based on participation, retention, and completion tell only one side of the MOOC success story,” they conclude. “While they bring the potential for significant disruption, they can be a legitimate and strategic element of a university’s offerings…”

For more information on KSU’s MOOC offerings, as well as further research, read the full report, “MOOCs: Branding, Enrollment, and Multiple Measures of Success.”