Scaling Content Production

SITE NAME
2016-01-10T22:13:34+00:00

One of the challenges we face at Coursera is scaling our content production to help us onboard hundreds of great new courses and specializations. In fact, just a few months ago, we launched over 30 new specializations on our platform, which amount to over 200 new courses. Supporting and launching all of these courses is in line with our mission to provide universal access to the world’s best education. But this process requires coordination between the course team at our partner universities and our internal operations team to ensure that the courses we provide to learners are of high quality.

In consequence, we’ve spent the last few months thinking about and working on various improvements to our platform, with the goal of scaling our content production and helping our internal operations team launch even more courses while maintaining the same high level of quality. In this post we’ll highlight some of the interesting technology and product enhancements we introduced over the last few months to streamline the pipeline.

Course states

Believe it or not, a course on Coursera is a lot more than just videos and assessments. A course is modeled and stored with various associated properties. These properties are then used by our platform to determine how learners are able to interact with the course. Over time, as we’ve launched new functionality, the number of such properties have expanded, which means they may sometimes interact in unexpected ways.

To make it easier to deal with all of these properties, we enumerated a list of valid ‘course states’ and added validations to ensure that courses always have properties corresponding to one of the expected states. This has created a single source of truth, allowing our platform to automatically derive the state of a course from the values of the properties. At the same time, this allows us to easily communicate about and find courses in different states, as the course state is condensed into a single field we can query – a big win in terms of getting visibility into the states of our courses.

Internal raw course editing

Behind the scenes, Coursera often needs to make edits to properties of a course that aren’t exposed through our authoring tools. This includes editing course properties for new features that haven’t been developed completely or are still tested. In an effort to help our internal operations team we introduced a new internal raw course editor. This editor remains flexible enough to allow changes to many course properties exposed through authoring, but also leverages as many of our existing authoring tool course validations as possible for safety. This enables us to iterate quickly and with confidence when making changes to courses–a big plus when dealing with a large number of courses.

Pre-launch experience and production tool

Producing a course on Coursera also takes a lot of work and time – often a few months. Since there is such a large period of time between when a course is first created to when it actually launches, we now automatically set milestones (attached to ‘course states’) leading up to the planned launch date to help keep instructors on track. Notably, one of these milestones is to enable ‘beta testing’, which allows a small subset of learners to review your course content before it has launched; we’ve found this feedback process to be a great resource in helping improve course quality. We will describe our ‘beta testing’ process in more detail in a follow-up blog.

Building out this infrastructure also gives us insight into how far along a course is in its developmental process as well as when each milestone is planned to be met. In the unfortunate event that an instructor falls behind schedule, we can automatically notify instructors of the missed milestone and ask them to consider updating their planned launch date due to the slip. This helps us avoid advertising to learners that a course will start on an earlier date than is realistic.

By leveraging all of these improvements, we were able to build a course production tracking tool. This tool lists all of our courses alongside all the different properties associated with each course, and gives us visibility into our content pipeline by allowing us to filter courses by different states and their progress on completing pre-launch milestones. This lets us quickly answer questions such as ‘which courses are nearly ready to launch’ and ‘which courses are open for pre-enrollment.’ Having such granular control and visibility helps keep us organized and informed as we add more courses to our platform.

Roles validation

Aside from the regular learner roles hierarchy, Coursera maintains quite a few other roles reserved for partner staff members both at the course and partner level. For example, teaching staff, instructor, partner administrator, etc. These roles determine one’s ability to access certain courses and to make edits. As with course states, our methods for determining whether or not a user could edit a course became more and more complicated, leaving us vulnerable to inconsistencies that could potentially block legitimate users from accessing the authoring tools for their course. When this did happen, we relied on user reports to take corrective action.

As we continue to onboard new courses, we recognized that this approach simply would not scale and instead opted to be more proactive in terms of addressing these inconsistencies. We introduced a regularly scheduled validation job that reporting any red flags found in user roles. This gives us insight to existing problems so that we can proactively fix these user roles, ensuring that our partners have the correct access to their content.

Conclusion

Our work to help our internal operations team iterate with confidence and track the statuses of all of the courses in our production pipeline has greatly improved our workflow. As we enter the new year and subsequently a new phase of growth, we are now prepared to onboard a lot more new content in our ongoing effort to provide universal access to the world’s best education.

Happy new year!