Nonny de la Peña is pioneering immersive journalism
Nonny de la Peña is exploring immersive journalism and storytelling with virtual reality as a senior research fellow at the University of Southern California’s Annenberg School of Journalism. De la Peña’s most recent piece, “Project Syria”, premiered at Sundance 2015. The following interview was conducted by Lisa Jamhoury and Greg Moyer. The piece was written by Julianne Parker and originally published on Ochre, a digital journal published by Blue Chalk Media.
The singing is what you remember about the moments just before; an upbeat, repetitious melody carried by a young girl who runs alongside as you walk a street bustling with pedestrians, honking cars and market vendors. Then comes a deafening explosion.
Faces are digitized, reminding you that your presence in war-torn Syria is not real. Yet when the bomb goes off and the dusty, wailing chaos erupts, your knees shake and your stomach knots in genuine fear.
You have just experienced a virtual reality reconstruction of an actual news event, created by the University of Southern California’s Nonny de la Peña. De la Peña, who formerly worked in print and documentary film, has redirected her career to focus on new territory with a new name: immersive journalism.
Amidst a humanitarian crisis ever more dangerous for conventional reporting, de la Peña’s “Project Syria” draws wrenching attention to the plight of some one million children victimized by the ongoing civil war. Commissioned by the World Economic Forum, the piece—one part the bombing, a second part set in a refugee camp—is being shown at 2015’s Sundance Film Festival.
De la Peña’s world stirs you in ways you’ve never experienced in other news media. Physical fear, coupled with a convincing sense of being present in the bombing’s aftermath, evokes deep empathy for people whose reality you share from behind a set of goggles.
For de la Peña, that’s the goal: creating environments and stories that position participants as witnesses, not mere readers, listeners or viewers. Witnessing, she believes, yields new levels of understanding.
We recently spoke with de la Peña about immersive journalism, her perspective on the future of her craft, and being a woman in a field dominated by men.
Immersive journalism is really new and really exciting. Tell us about it.
Immersive journalism is a novel way to utilize virtual reality gaming platforms and digital technologies to put people in the middle of stories, to make them feel like they are present on scene. Much like the journalists, you become a witness. The participant, the reader and the viewer—some of this vernacular, we don’t know how it’s going to shape up—get a deeper, more visceral connection to the story, and that allows them sometimes to have a better understanding of what’s happening in their world.
I think the real point of journalism is always to create informed global citizenry, and at a moment in time where newspapers and broadcast media maybe aren’t attracting younger audiences, immersive journalism can.
It’s [also] about how we as content makers are going to decide to utilize this incredibly evocative and emotional platform to tell the stories accurately and with best practices.
You say virtual reality is going to be everywhere. Obviously, this is no longer the stuff of science fiction.
It’s getting less and less expensive to make the content. The ability to go out and scan your street and the buildings around you—that moment has already arrived.
What does immersive journalism bring that other communications technologies don’t?
I could never get behind my computer screen and be in the room with you, right? I could never feel like if I turned around and walked around the computer screen I’d see your back, right? But in the technologies I’m building with, you can. When someone is scanned you can walk around.
I did a little bit of an experiment with that with my “Use of Force,” which puts you on the ground at an event where the border patrol beats and Tases a man to death. More than a dozen officers were involved in this homicide and two grainy videos captured what happened. We rebuilt it from that.
One of the witnesses who took that footage, I brought her to the lab and facial-scanned and body-scanned her, and then I facial-captured and body-captured her motions [in] recreating her own memories of the night. So when you’re standing next to her in the virtual environment, it’s not me who’s recreated her memory—she’s recreated her own memory and she’s represented as herself from this space. I’m not sitting down and interviewing somebody. I’m letting them tell their own story in this way, and I think that’s really going to become much more common.
You’re talking about what all journalists, especially visual journalists, aim for: to put us on the scene, to connect us to what’s happening. But immersive journalism is intimate. Can you address its emotional impact?
I made “Hunger in L.A.” in the [economic] downturn, and food banks were particularly strained. I wanted to give people a real sense of what was going on, because the hunger was quite invisible in society. We recorded audio at food banks many hours until a day where a man waiting in the long line who had diabetes didn’t get food in time, and his blood sugar dropped too low, and he collapsed into a diabetic coma.
When that premiered at Sundance [in 2012], I had no idea the power it was going to have. I had so many people coming up crying—some would be down on the ground with the man trying to hold his hand and talk to him. I did surveys afterwards to check both the sense of presence and realism that they felt, about the profound sense of wanting to help and not being able to. People said over and over that they were carrying that off with them and back into the physical world.
I’ve worked in plenty of traditional journalism platforms, but never have I seen the kind of change it brings in people as when I put them through these pieces. When “Project Syria” went to the Victoria and Albert Museum and had a five-day run, we had 54 pages of guest book notes and comments. The curators said they had never seen such an outpouring.
So what’s your responsibility as a journalist to offer people the paths to act? For “Hunger in L.A.,” I put up links to Mend and the Los Angeles Regional Food Bank.
As journalists we’re trained to not offer advocacy positions, yet at the same time I questioned what is my responsibility when I put people through this incredibly emotional experience. That, to me, is more the unwritten territory than the journalistic part of this.
Walk us through the creation of “Project Syria.”
“Project Syria” is made from one particular video, and then we had to source a whole bunch of videos from this day when a bomb went off. We found all kinds of cell phone videos, used probably eight videos altogether to verify and support the event. Then I sent a team to the border in the Kurdish area and recorded real material in a refugee camp there.
I did exactly [what] I would have in making a documentary film or a good visual journalism news report, but then I had to take all that material back and recreate it in a digital way. I had this extra added layer of carefully creating a map of every aspect of the street so that every texture, every brick building, every bench, every bit of trash, every sign, everything was created one-to-one in the digital environment from the real source material. The textures, the cloth, all of that came from the imagery in the photographs. Everything was built as exact as possible.
It might be traumatizing for some people to watch events in Syria as virtual reality. Do you decide what to show, or do you let it play out as it happened?
I did make an editorial choice, I confess. There were some pretty bloody scenes around this event that I didn’t include. I included one piece of a boy who gets picked up and carried away, but there’s a lot more graphic stuff that happened that I didn’t include for various reasons. Some of it happened around the corner, and I can only have you stand in one place. That said, just like on the Web, you can see the most graphic stuff you could possibly want to see. I’m very clear with people about what they’re about to witness.
[Many humanitarian groups] tell you [Syria] is the worst crisis of our lifetime. … I think it’s appropriate to give people that deeper understanding. So I think I’m doing the right thing, I think I can warn people. At Sundance we have what we call a decompression area where we allow people to write their thoughts down and at least offer a post-experience moment.
What is it about virtual reality that makes it such an immersive, even physical, experience?
As you know, it doesn’t matter how many pictures of a beautiful beach you see, when you’re there, it feels really different. That isn’t to say I make you get all the way to the beach, but I certainly get you a lot further with this technology than anything else I’ve ever experienced.
I had a national reporter say to me [after an immersive journalism experience] that “two weeks later, the memory is still in my body, not just in my mind.” Because you jump and you react. There’s something really weird that happens, that it touches, that can’t be ignored.
You’re a pioneer in this field. Some of what you’ve done—mixing gaming with journalism, for instance; inducing trauma, as we’ve discussed—has met resistance. What keeps you going?
I’m a bit one of those people who, when they say no, I’m going to say, “You can’t say no to me!” It’s part of my character. But the second part of what really motivated me, as you’ll see consistently, were human rights issues. People who are invisible and voiceless offer huge motivation to me for whatever crazy reason.
I’m Mexican; my mom will tell stories—she was scared by her experiences to a degree. We would walk into a store and the shopkeeper would be rude and she would say, “It’s because of the color of my skin,” and I would tell her, “It’s O.K., Mom, it’s not because of that.” I’ve maybe incorporated some of that.
But I also have this deep love for technology, I just do. I have a comfort level with it.
You’re a woman in a very technical, male-dominated field. What difference has that made?
I know that I’ve been scared out of rooms when I shouldn’t have been, especially when I was much younger, when I’d walk into a class and I’d be the only woman of 32 guys. Last spring I was the only [woman] speaker at a virtual reality conference. There were 300 members of the audience, about five of which were women. I held up through the whole thing, and then after two days came home and sat at the dinner table and just melted down to my husband.
Why is it so traumatizing? I can’t even figure out what it is about being the only woman that makes you think, “I’m going to say the wrong thing,” or “I’m not speaking their language,” or “They’re not looking at me as their equal.” Virtual reality is definitely still a guy’s world. I’m doing my best to bring women in and encourage women and hopefully hire more women. That’s one of my goals and dreams, to really encourage women to come into this field.
Does being female influence your storytelling?
It wasn’t until I got obsessed with [virtual reality] that we started getting narrative about things like hunger. So women will just have a different desire to tell different stories, I guess. I don’t even know if it’s the gender piece, but certainly we know that there are going to be different kind of stories and different kinds of experiences and that we want it to be an inclusive place so that all the stories can be told.
Let’s look to the future. Do you see The New York Times or Los Angeles Times doing a virtual reality piece here and there, or do you think it will always be independent shops?
Absolutely the news orbits are going to get involved. […] The way it’s going to work almost certainly is this: three tiers.
You’re going to have your IMAX version where you’re going to go to a location where you’re going to walk around and have this incredible experience. Then, you’re going to have your living room experience sitting down in your home with the best audio. And then you’re going to have your mobile experience.
Some things that will be designed for your mobile experience never will have to go to the IMAX experience. Sometimes we’re making IMAX things and we bring them down to the mobile version, which is what we’re trying to do with “Project Syria” right now.
For example?
For example, for a mobile version, I’ll use the Trayvon Martin case. We could have had that done in hours, literally. It’s a condo complex, it’s dark and rainy, so you don’t need hugely lit, perfectly detailed everything. It doesn’t need to be perfect to get a detailed understanding of what happened.
Essentially you’d be sitting in the car next to a George Zimmerman model. You’re hearing the audio—him calling, the police saying “stay in the car.” And you’re seeing a shadow in the corner out in the dark—a hooded figure in the stairway. You don’t have to see it very well. George gets out of the car, and boom—you dissolve. Now, you dissolve into one of the interiors of one of the apartments. You’re hearing three different people calling on the phone. You’re hearing the conversations with the police. You’re hearing the audio of whatever was recorded. But even if you look out the window, it’s dark, it’s rainy. They couldn’t even see what was happening. It’s not that I’m going to show you what happened, but I’m going to put you on scene along with 911 tapes to give you a more visceral sense of what happened in a linear way.
We could have had that out in a day, and for not a lot of money.
You’ve called all of this “crazy exciting,” and you seem crazy excited.
It’s been amazing to watch how the fire in my belly could become a useful tool to help innovate in journalism. And really, it’s just good for all kinds of storytelling—visual storytelling. The technology is going to get better and better, and the storytellers who put their mind to this are going to become a greater number, and that means more ideas and beautiful work are going to come out of it.
Nonny’s Virtual Reality Toolbox
When Nonny de la Peña creates an immersive experience, she starts with a gaming platform. One, Unity, provides the tools necessary to create interactive 3D and 2D content. Another, Unreal, provides game development tools without the need for coding.
Her next step is building the environments in which the virtual reality experience takes place. For stories requiring less detail, such as the Trayvon Martin example explained in her interview, she might start with Turbo Squid, where she buys stock 3D models for characters, architecture, landscapes and more.
For more complex stories built with original design, de la Peña uses Maya or AutoCAD Map 3D. Maya provides “high-end character and effects toolsets along with increased productivity for modeling, texturing, and shader creation tasks.” AutoCAD Map 3D offers “access to GIS and mapping data to support planning, design, and data management.”
The tough part, de la Peña says, is rendering accurate human faces. Face Shift analyzes motion to create realistic and emotional facial animations. Mixamo, a 3D graphics technology company, sells character animation from a library of motions.
