Re-engineering the past for the future

SITE NAME
2015-11-11T16:00:00+00:00

Coursera’s take on bringing online education to the Apple TV

When someone asks what I do, I tell them I develop for mobile. Generally, this is met with interest and curiosity — after all, we are tasked with the interesting challenge of resizing and rearranging content built for the web on a small screen while trying to make education as engaging as possible. With billions of phones all over the world, developing for mobile is key to achieving Coursera’s mission of providing universal access to the best education from leading universities.

Last month, however, this answer slightly changed. A sample conversation may be like this:

“What are you up to these days?”

“I’m trying out development for the TV.”

Awkward silence ensues.

“TV…like the thing that you can watch the news on?”

“Yes.”

At this point, the topic dies and we stare at each other for a while before they ask why a mobile developer would work on a non-mobile device.

"So, what is a tv?" Questions milennials ask at #svios. pic.twitter.com/GUzamaE7nJ
— Tim Burks (@timburks) November 4, 2015

The 4th Generation Apple TV, in some senses, came pretty late. The original was announced in 2007, and did not change very much for the next eight years, even as the iPhone changed forms and the Jony Ives decided skeumorphism was the devil. For years, it was virtually impossible for developers outside major companies to get their apps onto the Apple TV without jailbreaking and performing a variety of other hacks.

Here are some of our discoveries as we worked to reengineer our iOS application for tvOS.

The Future of TV…is Apps

Any change after years of silence is going to be perceived as groundbreaking. Still, opening up the SDK to all developers is truly an important milestone — graduating from a side project that Apple is making for kicks to something that is supposed to be around much longer. It is still a streaming box like its competitors, but it aims to be something more than that. It’s now up to developers and users to see if its app ecosystem will bloom to the same degree as iOS’s app store did.

The original Apple TVs and its competitors often ran Javascript and XML-based applications, which made sense with their relatively low hardware specifications. In fact, many manufacturers essentially redressed their mobile operating systems and shoved it into the TV. Samsung moved to the Tizen operating system on its TVs, and LG televisions often use WebOS. The Apple TV was no different, with its second generation switching to an iOS-based operating system. However, the fact that these TVs required a substantial amount of time and effort to port existing native mobile applications to these Smart TVs, compounded by the relatively poor hardware and the abysmal fragmentation of the platform, meant a lot of developers simply did not have the resources nor reason to write software for the big screen.

The current Apple TV differentiates itself from the market by not only upgrading its hardware significantly to match the specs of the iPad or iPhone, but also offering developers to code for it not only in TVML, a revamped version of the Javascript-XML template-based system, but also in Swift and Objective-C. Additionally, while the operating system, tvOS, is technically separate from iOS, it shares many of the underlying fundamental frameworks, including, but not limited to, the following:

AVFoundation

CloudKit

CoreData *

CoreFoundation

CoreGraphics

Foundation

GameKit

Metal

SpriteKit

StoreKit

UIKit

** CoreData exists, but does not guarantee that your data will actually persist. More on this later.*

Suddenly, it is much easier to bring apps over to the big screen. Indeed, at Coursera, we were able to ship for the Apple TV after 4 weeks of work. That said, there were still large challenges we faced from both a design and engineering standpoint.

Extending the iOS app to tvOS

At Coursera, we make use of a modified version of the VIPER architecture for our mobile applications, which can be thought of as an extension upon the MVC structure. In short, VIPER offers us to separate our business logic from our view logic, and thus gives us the flexibility to add an Apple TV target to our app. (For more information, you can check out two of the talks we have given on VIPER here or here, and an overview of how we use the architecture with Swift here.) This way, we can leverage our existing networking and models, leaving us more time to work with adapting layouts for the TV. For parts of our app that use frameworks that are not supported by tvOS, we used preprocessors to if-def them out.

// Handle course information
#if os(iOS)
  // Persistence code
#endif
// return course

For many apps, in-app purchases are key for generating revenue. As the StoreKit APIs are available on tvOS, developers can enable purchases for the big screen — for example, for premium features, or for game upgrades. For developers who develop subscription-based purchases, it is definitely worth considering making a universal app, as this means users are able to bring their purchases made on a mobile device to and from the TV. To do this, you should make sure that “Devices” is set to “Universal” inside your project deployment settings, and confirm that you use the same bundle identifier and sign with the same provisioning profile for both iOS and tvOS.

One of the biggest surprises was when we read Apple’s stance on persistence:

Your app can only access 500 KB of persistent storage that is local to the device.

With caching, for all intents and purposes, disabled, a good portion of our focus was devoted toward finding ways to minimize both the size and frequency of network requests in order to provide the same performance on the TV as the iPhone and iPad. Although we still have a ways to go, one relatively cheap, but important, fixes include making sure assets are optimized before downloading. It will be interesting to see how developers will work around the restriction in the coming weeks.

In Cupertino, “Menu” means “Back”

One of the problems with the Siri Remote (and to be fair, all of the Apple TV remotes), is that the “Menu” button often does not give you the menu. Instead, you get sent back. It’s almost as if “back” is so connected to Android that Apple had to choose a different word, but for some inconceivable reason, decided “Menu” was the way to go. If you launch the app to its home screen, and the user thinks: “Oh, I should go to the menu to see what else there is” and presses “Menu”, they’ll be booted out as if they clicked exit. Although users can (and will) get used to this behavior, it still poses navigational challenges for developers.

One such example includes how we display screens with a payment flow. Since we allow users to purchase Course Certificates on the Apple TV, the user will generally tap “Enroll” on the course description page, choose whether they would like to pay for a certificate or enroll without, log into iTunes as necessary, before arriving on the course outline. However, by pressing “Menu”, users would not be taken into the course description page, but instead to the payment page. While this offers the user another chance to repurchase an item (and unfairly inflate revenue), this is an undesirable user flow.

If a user purchases a course (marked 2), they are sent to the course outline (marked 1). However, if they press menu, by default, they would end up back in the payment screen. We override the menu button to unwind back to the dashboard (marked 3).

To fix this, there are three ways to intercept button presses and override their default actions. On the highest level, you can use UITapGestureRecognizer to check for the following two press types:

UIPressType.Menu

UIPressType.PlayPause

As their names imply, these press types fire when the Menu or Play/Pause buttons are pressed, respectively. One implementation may be as follows:

For us, we intercepted the menu press and redirected the user back to the dashboard instead of to the purchase flow via an unwind segue.

For a lower level approach, you can either use GameKit to use the remote as a controller, or override four methods in UIResponder:

pressesBegan

pressesEnded

pressesChanged

pressesCancelled

The last two methods offer greater control that is needed in applications such as games. Since Coursera is not, unfortunately, a game, we could use the first method to redirect users back to the course description page.

Other uses for overriding these buttons include using it for one click actions. One example that we’ve seen includes AirBnb’s bookmarking feature, where users can mark listings they like to view later on their computer by simply pressing the play-pause button. A future feature we may look at include 1-click enrolls by using this method.

AVPlayer: Great for 95% of use cases

Accessibility has always been a big priority for us at Coursera. On both the web and mobile, we have added support for subtitles for various languages in lecture videos so as to make the content more accessible for all. Apple’s AVPlayer is surprisingly well suited for most video needs, and by default, integrates with a lot of the new features in the Apple TV for free. With HLS videos with WebVTT subtitles, the client can easily parse the URL and load into the player to enable subtitles. As a bonus, you can perform a lot of the basic options offered by the Siri Remote as well: users can easily scrub to times with the touchpad, or use voice commands to seek to a different time.

There are still cases where you would need to write your own video player though, especially if you need to get to the details of the video data itself — for example, adding real time filters to video. However, for the most part, AVPlayer makes it very simple to add rich media into the Apple TV.

Don’t Touch the TV! Designing for the remote

The Apple TV, for the most part, does not support continuous scrolling. Instead, developers need to make use of the Focus Engina, where you hop along focusable and selectable elements to navigate within and between views. This simplified navigation allows users to quickly move to different parts of the app, but comes with a few gotchas to watch out for. Imagine that you are carefully picking your way across a river by hopping on rocks. If there is a waterfull beyond the bend in the river, and there are no rocks for you to keep your shoes dry, regardless of how amazing it is, you will not be able to see it. Just like how the rocks are the stepping stones in the river, the focusable elements are the stepping stones of the view. Opt for designs that mainly move horizontally and vertically as opposed to diagonally, and check that there are no “dead ends” where it appears you can move in a direction, but actually can not.

Another awkward “why would you do that” design is the linear keyboard on Apple TV. It looks nice! But that’s about the extent of its good qualities. Here are ways to drive a person insane:

Use the Apple TV keyboard to type in a 32 character WiFi password

Use the Apple TV keyboard to type in an e-mail you’ve had since 5th grade

Use the Apple TV keyboard

A few frustrating seconds of attempting to name my TV “Alex’s Apple TV”, I’ve compromised and now let my phone AirPlay to “Ale”

The lesson we’ve learned is that typing absolutely sucks on the Apple TV. Even typing in short search terms is as enjoyable as listening to nails on chalk. One way to mitigate using typing to search is to create a browsing experience. For us, we broke down our catalog content into overall domains of knowledge, and further subdivided it into fields. This way, if a user were to look for “iOS development”, they would only need to check “Computer Science” and “Mobile Development”, rather than arduously pick out the letters in a search bar.

One feature we are planning to migrate to includes the PIN-based login system. Instead of typing out your password for everyone to see on the TV, you would either:

1. Show an onscreen code and URL for the user to type in on their personal device.

2. Display a onetime short token on the app to enter on the TV.

These two strategies allow users to seamlessly login without the frustration of trying to type out long or complicated passwords.

TL;DR — A Few Final Notes on Design

It’s a sad fact that people today dislike reading more than ever. However, there is one thing that people hate more than reading: reading on the TV. Perhaps the most important point to remember while developing for the Apple TV is that less is more. This should be reflected in both the content and layout of your app. Pictures are great! Text sucks. A good balance of both creates a more immersive experience.

Space is king on the Apple TV: moving our Social Media Marketing Specialization from the Web to Apple TV

The Human Interface Guidelines lists out the important things to look out for, but here is a quick summary of things to look out for:

**Don’t just scale up. **With AutoLayout, it’s too easy to assume that if the app scaled well from the iPhone to the iPad, it will work out on the Apple TV. Unfortunately, this is not the case. The TV is a uniquely different experience from a mobile device, and should be treated differently.

Having small text on the Apple TV is like having small text on a sign on the freeway. Seattle drivers are already bad enough at getting around — they don’t need to deal with additional challenges.

Design big. Big text and generous spacing helps greatly in creating a more relaxed experience. Prioritize content so the most important items are on top, and everything else can be hidden below the fold. Make sure your minimum font size is above 28pt, and titles are large (72+pt). You know how highway signs are big so they could be read from far away? The same concept applies for the TV.

Keep in the safe zone. Much like print, TVs have what are called a “safe zone” — an area inset roughly 90 to 120 pixels from the edges of the screen. Not only is content at the edges of a screen harder to read, it can often be cut off if a TV is not configured quite correctly.

Reduce typing, and optimize the experience around the remote and Focus Engine.

Coursera on the Silver Screen

Our goal at Coursera is to bring the world’s best education to everyone. From our beginnings on the web in 2012, to mobile in mid-2014, and now to the Apple TV, we hope that expanding onto more platforms can help us get closer to our goal of universal access to education from the world’s leading universities. We are super excited to see how developing for the Apple TV will evolve and see where this path will go.

Many thanks to Paul Robinson, David Le, Brandon F. Tram, Deepkanwal Plaha, Daniel Shusta, as well as the Mobile Team at Coursera for making Coursera for Apple TV possible!