Studies show that embodied learning - learning by involving the whole body - helps kids grasp new concepts. For decades, Sesame Street has drawn its young audience in with audience participation - songs that encourage singing and dancing along, answering Kermit's questions, or counting with Count von Count.
The project aimed to push the boundaries of the Kinect platform to create a fun, engaging and visually stunning experience that got kids off the couch to enhance learning.
The result was two seasons of Kinect Sesame Street TV, plus a season for the Surface called Sesame Street Touch & Learn. Using the Kinect, kids are dropped in the middle of Sesame Street to jump, move, wave, and play with Elmo & friends.
I hopped across the pond to London in 2011, and joined the small Creative Media team as a motion designer - alongside the game designers who worked closely with Sesame producers to weave interactivity into new Sesame stories, the video team who worked magic with green screens, and engineers who worked with hundreds of video segments to create interactive video branching magic. Needless to say, I learned a lot.
With video as our medium, motion graphics functioned as a layer providing immediate feedback to the child in front of the screen. A character on screen might ask a child to get up and jump or wave or shout in order to help them complete a task - without visual or auditory feedback, kids would lose interest, feel overwhelmed, or might not feel involved in the story.
Another good example of this was in a scene where the child is asked to wave in order to give a dog a haircut in a certain shape (like a frisbee...). In usability sessions with kids, they weren't sure what to do, or if they did wave, their wave wasn't long enough or didn't span wide enough to be detected by the Kinect. We wanted a big, enthusiastic wave. After several experiments, we added a paw animation with particles in an arc shape to mimic the action the child should do with their own (human) paw. It worked!
Once the child waves their arm back and forth, as detected by the Kinect, the pup gets their haircut - reinforcing the shape that they're learning.
An interesting challenge came with taking Sesame segments from past seasons and transforming them into an interactive experience. These were filmed on set with human cast members and muppets, sometimes featuring a celebrity guest (like Jason Schwartzman 💖).
To keeps kids engaged during the roughly 7 minute stories, the team decided on a hide & seek style game. The idea was to show a specific object to the child before the segment, and when they saw one of the 12 hidden objects, they were to point at it and say 'picture!'. Cooper, a completely digital muppet, would pop up, take a picture of the child using the Kinect camera, and at the end, the child could view a scrapbook of all the images of themselves with all the objects they found.
There were a number of technical and usability challenges that came with this - how to blend a graphic into complex video footage, how to balance the frequency of the objects, the position and size of the object, the length of time on-screen, the level of visibility - all to balance varying levels of difficulty in finding the object to keep the child engaged.
This involved meticulous camera tracking, rotoscoping and compositing in After Effects. Think of all the muppet fur! The objects fit in the scenes seamlessly, often blending in with backgrounds, hiding behind characters or objects, only to be revealed when a character or camera moved.
Our User Research team tested segments on a weekly basis with preschoolers - revealing objects that were too difficult to find, too dark or too small or onscreen for too short of a time, so we could go back to the footage and iterate.
Another interesting challenge came up when Soho Productions began work on a touch version for Windows Surface called Sesame Street Touch and Learn. The challenge was to use the same content, but the interactions and voiceover prompts need to be redesigned - rather than asking a child to wave, jump or talk, designed for the Kinect, not a tablet, were replaced with tap and swipe.
I worked with our game designers and engineers to restructure and redesign the video content and graphics to correspond with the tap and swipe gestures.
An example of this was a segment called 'Letter Tree', that originally required the child to jump along with Cookie Monster to shake potential edible items out of the tree.
For the Surface, this was adapted to tapping to release the objects. To indicate that something is hiding, ready to be released, I animated each object bouncing (a bowling ball), swinging (Baby Bear's legs), or rolling (a kumquat) in the tree branches. The challenge was that we only had the original footage and assets - so this required taking a still image, slicing and dicing to animate it, and compositing it behind the leaves.
Kinect Sesame Street TV was released in September 2012. Microsoft made a few snazzy promo videos, it won a BAFTA Children's Award, got positive reviews from game critics and Amazon critics alike and a fancy Xbox game now sits on my shelf. It wasn't quite the rip roaring success I had anticipated for such an inventive TV experience—owning a $300 Xbox is a barrier for a lot of families and 3 year-olds aren't so great at navigating TV and Xbox menus themselves—but it was definitely a success for those of us working on it and we had a lot of fun and pride in doing so.