We recently released an amazing short film entitled "Resignation" that turns the superhero myth on its head. See the full experience at resignationsuperhero.com, or just enjoy it below. The film runs on the Multipop video platform which layers in additional content, backstory, social interaction and click-to-purchase functionality. This is the first in a series of technical dives into how the Multipop player works behind the scenes.

Let's focus on the content presented in sync with the film itself. Look at the panels surrounding the video and you'll see a lot going on. As our hero takes photos in the live action, images pop up in the right siderail, then a newspaper article with that photo appears beneath in the detail panel. Similar things happen with beautiful graphic novel style artwork filling us in on the backstory. Even links to purchase books related to the story appear. And on the left siderail, a Mission Control tab streams a feed of DEA agents' conversation in sync with the live action. How is all this being done?

HTML5 In. Flash Out.

Let's start by pointing out this is not some elaborate monolithic Flash movie with all the interactivity and content programmed in at great effort and expense. Rather, the content is assembled dynamically at runtime and rendered as HTML in a series of widgets that surround the core video player. Two benefits of this approach are quick and easy production and the ability to change content and timing continuously. So even after the film is released, the experience can be tuned based on audience reaction. Let's break down how it works.

Multipop Authoring

There is a whole back end authoring environment for assembling Multipops from the original media. The tool is web-based, and includes the usual features to upload and manage video files. Synced content is also defined within the tool. This includes image files of the photographs, newspaper articles & artwork, plus text of the DEA messages and URLs to purchase products on Amazon.com. In aggregate, we call this the Item Catalog

Multipop timing editor

Multipop timing editor

Items from the catalog are placed on a timeline in sync with the video (see the screenshot of the editor above). The author watches a preview of the video, then just clicks a button to set the timings. The timing data is stored in a database in what we call a Multipop Manifest that can be changed at any time. Let me state that again: because the timing data is not stamped into the video itself or baked into a Flash application, it can be changed at will with ease.

The Multipop Manifest

When someone goes to view the Multipop, the first thing the player does is request playback instructions in the form of the Multipop Manifest. It does this via an internal REST API which serves the manifest in JSON format like this:

Among many other things, the manifest describes the synced content and when to display it. In the example above, we see URLs to the images for the photograph and newspaper article, plus the timings (in seconds) for when to display this content. Our back end does plenty of hard work to serve this from a high performance cache to ensure speed and scalability, but to the player, it's just clear and concise JSON available on demand.

The Multipop Player

The Multipop Player uses the manifest to marry up the video with the content. The player is entirely brower-based, and consists of the following:

  • the Flowplayer HTML5 video player
  • our own custom HTML5 UI surrounding and overlaid on top of the Flowplayer (for the siderails, detail panel, clickable regions, etc.)
  • a sophisticated Javascript/JQuery application that communicates with the Multipop back end to retrieve the manifest, animates the UI, and reports user interactions back to our analytics engine

Without going into a ton of code, the concept of syncing content with the video is easy. Flowplayer throws a "progress" event every 250 milliseconds or so during playback with a parameter indicating the current position of the playback head. All Multipop has to do is subscribe to that event and show any items scheduled to displayed at or before the current time. Here's a snippet from the actual source code:

// responds to video playhead moving
progress: function (event, api) {
var tick = api.video.time;
// show items tied to timeline

The function showItemsForTime() does all the complex work of updating the UI to display the item, but the act of syncing it up is amazing simple. The result looks like this:

Synced Content in Action

Synced Content in Action

That covers the basics of how Multipop implements synced content. In upcoming installments I'll look at other aspects including media encoding and distribution, analytics, click-through purchasing, hotspotting and more.