All articles

How Snapchat makes mini movies in augmented reality

By Reggie Ugwu
The New York Times

A.R. may change entertainment as we know it. For Snapchat, the revolution starts with 15-second cartoons.

LOS ANGELES — On a cool, sunny day in April, a short Frisbee toss from Venice Beach, a team of seven animators sat around a conference table, workshopping a storyboard from an unusual cartoon series.

In the storyboard — rudimentary, comic-striplike panels tracing the arc of a scene — a barefoot man in rolled-up jeans stands at the edge of a bed of flaming hot coals. The man gathers his nerve, walks across the coals without incident and does a little celebratory dance. But his celebration is premature; in a plot twist in the final frame, his feet burst into flames.

Some of the notes in the room weren’t far from what you’d expect in any introductory art class. But other feedback was more peculiar. In lieu of a setting for the story, the animators discussed only how they could “indicate an environment,” perhaps using sand or grass. Later, there was a curious debate about whether or not the character — otherwise recognizable as a human male — had any toes.

Everyone agreed that they didn’t know for certain.

Rather than a film or television studio, the animators worked for Snapchat — the camera app and social media platform beloved by teenagers and cautiously monitored by pretty much everyone else. The figure in the storyboard was what’s known as a 3D Bitmoji, part of a deceptively powerful Snapchat feature that uses augmented reality to drop cartoon characters — dancing hot dogs, twerking chipmunks, Ed Sheeran — and other digital objects into a camera lens’s field of view.

Excerpts from the hot-coals storyboard. Credit: Snapchat

A year ago, many of the animators at Snapchat worked for companies like Disney and DreamWorks. But the balance in their industry is shifting. As with radio and television in the last century, augmented reality  is paving the way for a transformative new kind of storytelling. The race to define its early days has aligned trained artists, giants of Silicon Valley and billions of dollars in venture capital. The outcome for legacy media may be profound.  

3D Bitmoji debuted on Snapchat just over six months ago, descended from the digital cartoon avatars that proliferate in texts and social-media profile pictures. (Snapchat acquired Bitstrips, creator of Bitmoji, in 2016.) As with most things on the service, describing what it’s like to use 3D Bitmoji is a little like humming what it’s like to jump out of an airplane. But imagine a cartoon replica of yourself — endowed with Charlie Chaplin’s gifts for physical comedy and a 13-year-old YouTuber’s choreographic repertoire —  standing on your desk, or in your bed or on your Labradoodle.

New 3D Bitmoji episodes are released two to four times each week, so one day you might open the app and find your avatar doing back flips on a park bench. The next, it’s dancing woozily to “Gucci Gang” or staring defiantly down a bed of red hot coals. The feature is any modern narcissist’s dream come true: serialized, scripted entertainment in which you, finally, are the star.

Snapchat isn’t the only company hastening the augmented reality revolution. Though not yet heavily used by consumers, A.R. has, in recent years, become an obsession of many of the most powerful firms in the world. Their aims are not modest. Tim Cook, the Apple chief executive, has said the medium will “change the way we use technology forever.” Mark Zuckerberg of Facebook has called it the platonic ideal of computer interfaces. Rony Abovitz, founder of the A.R. start-up Magic Leap, which has raised more than $2 billion in investments   from Google and others, talks of building “a new internet of presence and experience.”

Snapchat isn’t the only company hastening the augmented reality revolution. Though not yet heavily used by consumers, A.R. has, in recent years, become an obsession of many of the most powerful firms in the world. Their aims are not modest. Tim Cook, the Apple chief executive, has said the medium will “change the way we use technology forever.” Mark Zuckerberg of Facebook has called it the platonic ideal of computer interfaces. Rony Abovitz, founder of the A.R. start-up Magic Leap, which has raised more than $2 billion in investments   from Google and others, talks of building “a new internet of presence and experience.”

Unlike virtual reality, which demands that the natural world be forfeited for seductive ones and zeros, the allure of augmented reality is a kind of perceptual truce, with our digital preoccupations layered harmoniously atop the natural landscape. The holy grail is a scenario in which all of your screens —  phone,  laptop, television set — are no longer necessary, replaced by interactive holograms that simulate, or improve on, their essential functions.

In light of the medium’s sci-fi potential, Snapchat’s 3D Bitmoji animations are relatively primitive. Philip Lelyveld, director of immersive media at the University of Southern California’s Entertainment Technology Center, said the cartoons were “one step ahead of Pokémon Go,” the mobile game that  first created mass awareness of A.R. in 2016. Helen Papagiannis, an A.R. researcher and author of the book “Augmented Human,” likened 3D Bitmoji to silent films in the early days of cinema.

The equivalent of blockbusters, which will rely on special headsets like Magic Leap’s Lightwear and Microsoft’s HoloLens, are in development. And Facebook (which owns the virtual reality company Oculus), Apple and Google have all moved in the last year to make it easier for third-party developers to create new kinds of A.R. experiences for mobile phones.

But Snapchat’s animators here in Los Angeles have already created the medium’s first hit series, thanks to uniquely appealing main characters that are flipping, dancing and pratfalling their way into the hearts of the company’s 191 million daily users. While Snapchat doesn’t share viewership numbers for 3D Bitmoji, a spokeswoman for the company told me that 70 percent of its daily users are using their Bitmoji avatar on the app.

According to Snapchat, more than half of 13- to 34-year-olds in the United States are using its A.R. lenses every week. The selfie filters are just the beginning. What comes next may affect everything from your local theater to whatever you’re reading this story on.

The Liberman siblings — from left, Maria, Daniil, David and Anna — at work at Snapchat. Credit: Jake Michaels for The New York Times

What became 3D Bitmoji originated on the other side of the globe, in Moscow, with a satirical animated sketch show modeled after “Saturday Night Live.” The  2009-13 series,  “Mult Lichnosti” or “Cartoon Personalities,”  gingerly spoofed Russian politicians and celebrities, including Vladimir Putin. It was known for distinctive 3D character modeling (eerily detailed, with enormous bobble heads) and for  up-to-the-moment topicality — a feat in a medium where a single episode typically takes months to produce.

“Mult Lichnosti” was created by four lanky Russian siblings, Daniil, David, Anna and Maria Liberman, none of whom had any particular gifts for animation. Instead, they specialized in  creative operations — developing pipelines and systems that demystify and accelerate the creative process. Before “Mult Lichnosti,” they’d honed their operational skills running a successful massively multiplayer online video game company, which required keeping large teams of coders, writers and artists working together in lock step.

In augmented reality, the ability to work across disciplines is especially valuable. The medium represents a mash-up of modern technological marvels — computer vision, haptics, artificial intelligence — and more conventional creative endeavors, like screenwriting and character animation.

In late 2016, the Libermans’ unique background caught the attention of executives at Snap, which, one year earlier, had demonstrated A.R.’s potential as a creative platform with the rollout of animated overlays on selfies. The company jointly hired the foursome to develop animated A.R. characters based on Bitmoji. “They’d been there and done that,” said Eitan Pilipski, vice president of camera platform at Snap, Snapchat’s parent company. “There was an amazing opportunity to take their creative thinking and put it in the hands of our users.”

Sharing an office with three of your siblings could be a recipe for disaster, but it seems to have the opposite effect on the Libermans. In Venice Beach, they were like different limbs of the same chestnut-haired, fair-skinned organism, finishing one another’s sentences when not cracking up at inside jokes.

In Russia, growing up in a house with their parents and two other siblings, they fell down the rabbit hole of a Disney home video collection (“Beauty and the Beast,” “Duck Tales,” “The Lion King”) and never came out. “Russia can be quite melancholy,” said Daniil, product lead, 35. “We were always trying to bring more fun into the gloominess of the environment around us.”

The Libermans shared a house for a time when they moved to Los Angeles, too, then spread out to their current arrangement in three apartments on the same street. “Before college we were together constantly, and afterward it’s been the same,” said David, technology lead, 34.

At Snapchat, the siblings oversee an in-house animation studio consisting of a few dozen animators, scriptwriters, visual effects artists, character riggers, engineers and others. Though bite-sized animations for a social media platform may seem like a step down from  feature films, Maija Burnett, director of the character animation program at CalArts, said many in the industry view A.R. as a bold new frontier.

Daniil, left, and David Liberman taking videos of each other. The animators often film themselves to help portray body movement. Credit: Jake Michaels for The New York Times

“It has the feel of being a little under the radar but on the cutting edge,” she said. She also pinpointed something else that might explain Snapchat’s appeal. “A Pixar movie takes five years to make,” she said. “If you’re an artist, that’s two films in a decade, whereas at a place like Snapchat you can create a lot more volume.”

Each of the Libermans handles different responsibilities, which Anna, production lead, 36, said were based on their individual personalities: “Daniil is very creative and helps us to write jokes. Maria is verbal and good at understanding story structure. I’m very good at talking to people and getting them committed to specific dates and times. And David is very technical and has a strong engineering background.”

3D Bitmoji episodes follow a production cycle that is modeled after that of a feature-length animated film, only dramatically condensed. It begins with a script — 7 to 15 are written each week narrating 15 to 20 seconds of action. About half of those are deemed worthy of advancing to the storyboarding phase, which is followed by multiple rounds of animation and review, the addition of visual effects, quality assurance testing and, finally, publication.  

After the storyboard review, the hot-coals episode was ready for animation. The animators designed a special costume of T-shirt, rolled-up jeans and bare feet to fit the logic of the story. (As to whether Bitmoji have toes, they do, it turns out — feet with discernible digits had been developed for an earlier yoga episode.)

Unlike traditional cinema, in which everything you see on screen represents the will of a filmmaker, augmented reality experiences are “directed,” in effect, by the viewer, whose physical location and line of sight determine the setting of the characters and the angle of the camera. “We have no close-ups or any of the tools a director would normally use,” said Maria, content lead, 39. “The only thing we have control of is the story, which is why we try to make sure that every second counts.”

In the animation stage, the animators aim to “sell” the idea of the story, which should be easily “read” by the viewer without the benefit of dialogue (recording multiple languages and voice styles would be too resource intensive). The animators often film themselves role-playing as the character, or pull reference videos from YouTube, in order to realistically portray body movement.

With “Hot Coals,” they  focused especially on a sequence of movements right before the character steps onto the coal bed. The character takes a deep breath, shakes out its arms and enters a Zen position,  eyes closed and  index fingers pressed to its thumbs. “We really wanted to show that the character is nervous, that it’s not something easy to do,”  Anna said.

Because 3D Bitmoji episodes are designed to be sent between friends, they aren’t just entertainment — they’re custom memes, conveying the sender’s mood or status. The most successful tend to be those that capture a common experience or sentiment, sometimes indirectly.

Anna described “Hot Coals” as a universal story about facing your fears, leavened with the twist at the end (what Snap’s animators call a “bonus”) when the character catches fire.

That made Maria recall a sort of spiritual precursor, an earlier animation involving a pregnant pause at the top of a diving board. One user’s caption: “Me getting into a new relationship.”

When the technology is sufficiently mature, experts expect that most forms of entertainment will be reimagined for augmented reality. Mr. Lelyveld, of the Entertainment Technology Center, predicted a surge in sophisticated narrative A.R. experiences in 18 to 24 months, when glasses allowing users to circumnavigate fixed virtual objects in physical space are projected to be widely available.

Hands-free A.R. (which may one day rely on contact lenses or cortical implants) could unleash the potential for stories that blur the line between fiction and lived experience. They may share elements with linear film and television, or task-oriented video games. “You could be watching a sitcom like ‘Friends,’ but instead of the ‘Friends’ kitchen table, the characters are all sitting around your kitchen table,” Ms. Burnett said. But the medium’s capacity for contextual awareness allows for hybrid forms unlike any that exist today.

Experts cautioned that it was still too soon to say what shape they may ultimately take. “Our language has always been 2D, and if your language can’t express it, you can’t imagine it,” Mr. Lelyveld said. But Hollywood studios are already girding themselves for the new reality.

Last year, 20th Century Fox started FoxNext, a division focused on developing V.R. and A.R. experiences by raiding the company’s vast vault of intellectual property. Warner Bros. and Legendary Pictures are both investors in Magic Leap, which has also partnered with Disney’s Lucasfilm and Peter Jackson’s Weta Workshop. And Time Warner recently led a $27 million investment in a company that aims to make virtual clones of celebrities.

“It’s no longer science fiction,” said Ms. Papagiannis, the A.R. researcher. “It’s here.”

After making it through the gantlet of animation review, “Hot Coals” was ready for the final stage of its production cycle before testing and publication: visual effects. The Libermans’ VFX artists added dancing flames, floating embers and, for the climactic moment when the character catches fire, pillars of white smoke. “The characters are very cartoony, but because they inhabit a real-world environment, atmospheric elements need to be quite realistic in order to bridge the gap,” Anna said.

After it’s published, every 3D Bitmoji episode spends at least one day in Snapchat’s digital lens carousel, where metrics like number of shares, posts to other social media platforms and length of play time are closely monitored by the Libermans’ team. The better the lens performs, the more likely it is to remain in one of about  20  carousel slots, which the company refreshes on a rolling basis. The average life span for an episode is one week.

After a battery of quality assurance tests, Snapchat planned to publish “Hot Coals” on June 14.

Mr. Pilipski, the vice president, said 3D Bitmoji was just the beginning of  Snap’s broader A.R. ambitions. “We’re trying to take the user on a journey from creative expression in 2D, to interaction in 3D,” he said.

While Mr. Pilipski declined to outline the course of that journey, Snapchat recently rolled out an in-app product, called Snappables, which allows users to play simple A.R. games (How many digital strawberries can you catch in your mouth in 10 seconds?) and invite friends to join. The company also makes sunglasses with embedded cameras, called Spectacles, though they’re not currently A.R. capable.

One path Snap isn’t likely to pursue is making 3D Bitmoji more closely resemble their flesh-and-blood analogues. Asked if the company would pivot to lifelike virtual avatars as technology advances, the Bitstrips chief executive, Ba Blackstock, a former cartoonist who joined Snap after the acquisition, balked.

“If you want a super-realistic version of a person,” he said, “you should go and have a coffee with them.”