A Guide to AR/XR in Journalism

Last June, Yahoo News began an ambitious project: Launching immersive content experiences from a wide variety of partners. While the groundwork had been laid, what we were going to do with these partners was still unclear and would require navigating the complex infrastructure of a large organization (Verizon Media Group) and an in-house production company (RYOT), not to mention the inner workings of a dozen partner news organizations.

Download this GuideMany news organizations were burned after investing heavily in VR over the past few years — while several made pieces that were powerful and award-winning, they failed to build an audience or have the reach they hoped for. As a result, there’s been a pull back in immersive content creation and it’s understandable — fool me once. But with the XR Partner Program, we see an opportunity to push these organizations and others who haven’t dipped a toe in yet to explore what’s possible with the advances in AR and spatial audio projects. These initiatives, which can be just as ambitious storytelling-wise, are less of an upfront financial or time investment than VR projects — and have the benefit of wider distribution options.

Regardless of the size of your organization, over the past six months we’ve found that exploring immersive opportunities puts us as an industry at the cusp of defining a new workflow — not just for immersive content, but for digital content as a whole. We currently create content using a platform that is being rolled out in apps across Verizon Media Group, but we are quickly innovating on WebAR and firmly believe this will be the key to growing immersive content across content organizations. WebAR allows the experiences we make to be shared both in apps and across mobile web, leading to projects we can share seamlessly with partners.

Right now, immersive content is divided into two categories:

• Daily immersive content, like 3-D images and simple data visualizations in AR that blend with websites and article pages using interscroll technology that can be taken to the next level and put directly in someone’s environment using an augmented reality app.
• Tentpole immersive content, story-driven projects that take the viewer on a journey exclusively when they have access to the latest phones and apps, bringing them out of their traditional news consuming space and into a new frame of reference. These projects often have an offline or location-based component as well.

Verizon Media Group is a bit of an anomaly in this space, in that we benefit from being part of a large telecom that has showcasing 5G and next generation technology as a pillar of its overall strategy. The XR Partner Program is housed under RYOT, an immersive content studio with a staff of technologists, creatives, and CG artists.

By no means is this a structure that exists at most news organizations, but learning in a place with those resources has helped us understand what is worth taking to news organizations of a variety of sizes and what’s worth investing in generally —and overall how workflow across the industry will shift in the next two to five years.  It’s important to keep in mind that the technology is only half the battle. The other half is design, distribution, and buy-in from your organization to promote these new types of stories.

Being a news and content aggregator comprised of brands including Yahoo News, Yahoo Finance, Huffington Post and AOL, our goal is to empower our external partners as well as our internal news brands, giving organizations access to RYOT so they can learn to create and share immersive stories across the Verizon Media Group network and their own platforms.

We currently create content using a platform that is being rolled out in apps across Verizon Media Group, but we are quickly innovating on WebAR and firmly believe this will be the key to growing immersive content across content organizations. WebAR allows the experiences we make to be shared both in apps and across mobile web, leading to projects we can share seamlessly with partners. Most recently in response to the COVID-19 pandemic, we’ve added a WebAR execution that maps the spread of the virus in 3D.

Why and when to test something ambitious

The guiding principles for each type of immersive project are the same: Story always comes first. The following guide will explore examples of content projects where we’ve experimented with a variety of augmented reality executions and how we’re focused on using this new technology because it enhances a story, not because it is the story.

Larger, more ambitious projects have several benefits. They can garner attention for awards and funding opportunities, like the Journalism 360 challenge and can potentially be submitted for festivals and live event executions. They can also be part of overall story packages around news-driven events that already have buy-in and investment, tentpoles like elections and the Olympics, and big anniversaries like the 50th anniversary of the moon landing.

Given the benefit of having the resources of RYOT to try out larger scale projects, we are using our internal owned & operated properties to create immersive stories that we can use as the basis for templates, making these types of projects easier to do down the road and easier to create future immersive story opportunities for our partners.

Case study: Landing on the Moon (TIME)

Our first project through the XR Partner Program was on a popular topic for AR experiences — it seemed so many news organizations were already working on a moon landing anniversary project when we began talking with TIME about incorporating theirs into the Yahoo News platform. This raised a number of questions for us:

• How would this project be different from the ones the New York Times and USA TODAY were releasing?
• As an aggregator, how would we promote this project on top of all the other moon landing anniversary coverage that was sure to come through the feed in late July?
• How would we hit an exact date for launch, given our tight build deadline?

This was the first time we’d used the proprietary platform that RYOT had originally developed for ad experiences across the Verizon Media Group network to tell an editorial story from a partner. While TIME had already developed most of the experience for their own immersive app, the lead audio engineer at RYOT had been part of the project since the beginning, so our team already had a good sense of the story and what might be feasible in our app.

RYOT adapted the content into a lighter version for the Yahoo News audience that would be incorporated directly into Yahoo News article pages, so readers wouldn’t have to download anything additional to see the experience. There were features we had to let go of for this to work, but in the end this created a unique and complementary experience to the one TIME included on their app and showed at their live events.

Case study: Rebuilding Paradise (Yahoo News)

Last summer, Yahoo News experimented with creating an immersive project looking at firefighters during the California wildfires and giving users a way to learn more about the tools they used (Rebuilding Paradise). While it was an interesting use of the technology, the ultimate execution was lacking a storyline and we ultimately decided not to publish this first attempt.

In the fall, we were quickly approaching the one-year anniversary of the Camp Fire in Paradise, California, which leveled the town and destroyed the livelihood of many of its residents. We felt a need to tell this story and thought there must be a way to use our AR technology to really bring people in to this story in a new way. While many of us are based in California and have a good sense of what these fires meant to the state and what it feels like to live near this kind of natural disaster, our audience is everywhere and wildfires if you’ve never been near one, are a unique type of disaster that lend themselves to a story that you need to be in to understand.

We had just hired our first dedicated XR project manager on the Yahoo News editorial team, Henry Keyser. He was able to quickly rally the Yahoo News editorial staff around the wildfires XR concept and worked with Yahoo News journalist David Knowles, who had already been covering several climate change pieces related to the wildfires to concept what his reporting might look like in an immersive capacity. Producer Rebecca Corey found Bert and Jo Clement, who had lost their home in Paradise and were rebuilding and brought their story to the immersive team at RYOT. From there, we worked to create parameters for an immersive story that would bring readers to Paradise.

Our goals: 

• Use wildfires coverage as a tent pole to highlight immersive content
• Create a template that we could use for future immersive stories
• Tell a human story, not a technology story

Total time: 8 weeks

Case study: Political Soapbox at the Iowa State Fair (USA Today + Yahoo News)

In mid-August we began work on our first truly collaborative immersive project. Ray Soto at USA TODAY was interested in capturing the Democratic candidates for president when they spoke at the Iowa State Fair. Partnering with USA TODAY and Gannett’s Des Moines Register, we brought a producer from RYOT as well as the USA TODAY team to the fair and worked together to get access to the campaigns, interview the candidates, and imagine what it would look like to put these larger-than-life individuals in your living room.

We learned that coordination with the campaigns on a tight, in the field schedule is no simple task. We succeeded in capturing 8 of the presidential hopefuls in AR, but were not able to get the front runners, whose time was extremely limited and harder to schedule.

We learned about collaboration — what it takes to have organizations with different and sometimes competing goals working together. The Des Moines Register team had a system that they’d been using for decades to usher the candidates on and off stage at the fair and throwing a new ask into that mix was daunting. On the Yahoo/RYOT side, we had never worked with a news organization in this way and capturing a volumetric asset in the field — let alone eight of them — was a tall order that required significant CG clean up back in LA afterwards. In addition, we were creating projects that were mirror images of each other — USA TODAY has their own AR app with its own technical specifications that use Unity and are different from what we needed to make the captures work on the Yahoo News app.

We also learned why Iowa is not known for its wine and that corndog beer is a thing. In other words, hanging out with your collaborators is the best way to make a project successful. The time we spent together in Iowa made the coordination we needed to do on the technical end much easier once we were all back at our offices.

Why and when to start creating smaller-scale immersive projects

While tentpole experiences may be the stuff of shiny award submissions, they aren’t the day-to-day future of newsroom immersive integration. What is the future is a change in workflow that incorporates 3D images and next generation content on a near-daily basis. As smartphones become ubiquitous, news consumers stick to mostly consuming content on their devices, and 5G technology eliminates latency issues, immersive content is soon to be everywhere. Some newsrooms are already experimenting in this space and it’s these organizations that we hope to emulate within Verizon Media Group and help with distribution for our partners.

Case study: McClatchy New Ventures Lab workflow

Credit: Melanie Hogue

When we approached McClatchy about participating in the XR Partner Program, we knew they were already lightyears ahead of most news organizations in integrating simple 3D objects into news articles as well as experimenting with longer form stories through their artist-in-residence program. It was a matter of solving for size issues — converting the assets created by McClatchy for their Sketchfab library and cleaning up those 3-D images to fit into the Yahoo News platform. There isn’t yet a jpg of the 3-D world, but we are getting closer to finding quicker, cheaper ways to create files that work in multiple places.

But the biggest learning was what is really necessary to build in 3D. McClatchy had captured several stories in Paradise following the Camp Fire and these assets were a great complement to our larger-scale Rebuilding Paradise story. One of the most powerful images was of the murals by local artist Shane Grammer. These murals were destroyed in the rebuilding and the 3-D assets McClatchy created using drone footage were a moving monument to a particular moment in time that would be impossible to experience as fully any other way. It was so much simpler than a long, involved narrative project, but coupled with the Sacramento Bee’s reporting, the 3-D asset incorporated into the text brought the story to life in a new way.

Total time: 3-4 weeks

Case study: Creating XR templates for Yahoo News

Our first forays into quick-turn content creation focused on a carousel experience that lets users launch individual stories with audio quickly. They don’t require a lot of explanation on launch and they provide a deeper level of context than a written article.

Iowa first time voters

Ahead of the Iowa caucuses, Henry coordinated with journalists on the ground in Iowa to meet a number of first-time voters. He then went to Iowa and captured them using photogrammetry using just his iPhone and a recorder to get audio interviews. Within a few weeks, those images were cleaned up and optimized by a 3-D artist at relatively low cost and he was able to build and integrate the stories with simple graphics made on Blender.

The story launched with the Yahoo News Iowa coverage the first week of February.

Total time: 6 weeks

Case Study: LA Times Oscar dresses

We’re based in LA, so of course awards season was on our radar as we began talks with the Los Angeles Times and their new immersive editor, Nani Walker, who was previously an artist-in-residence at McClatchy’s New Ventures Lab. Together with the LA Times’s fashion editor and TiltBrush artist Micah404, we reimagined a series of iconic Oscar dresses that work like a Snapchat filter, allowing the user to try them on.

We landed on Tiltbrush after brainstorming through a number of options including volumetric captures of the dresses themselves or using the fashion designers’ original sketches. When those options proved too challenging for the short timeline, we thought about some of the really cool VR art projects we’d seen drawn using TiltBrush, like the Left Field series from NBC or the Sundance darling Dear Angelica. Why couldn’t we draw the 3-D asset in VR and then size it down for an AR implementation? With some experimentation and patience, we were able to reduce the file sizes and create some really beautiful interpretations of some of our favorite Oscar dresses. This workflow will serve us well in the future — we’re already thinking about ways that illustration done in VR can help us with AR news storytelling for other projects.

Total time: 8 weeks (including holidays when lots of folks were off!)

Stumbling Blocks

• Technology options and file format limitations. Creating AR is getting easier every day, but it’s still not seamless. No one has decided on the file format to rule them all, so making a choice for your organization and sticking with it, whether that’s using Google’s Model Viewer or testing out AFrame and WebAR, or building something custom is key for now. But being flexible and being able to change as the industry changes is also important.
• Getting buy in from your newsroom powers that be. The Journalism 360 site and these guides are a great place to start helping illustrate to newsroom leaders the opportunity and impact of immersive storytelling. Budget and resources are also key to making your point. The cost of these experiences has gone way down, most photogrammetry can be done on just an iPhone. Working with the XR Partner Program may also be an option to offset costs, especially if you are already have a content license deal with Verizon Media Group.
• The biggest question: How can this content generate revenue? We approach immersive content as part of a larger package for our stories — an additional way for our sales team to support gettign content sponsored. As the technology improves, ads embedded within the experiences will also soon be an option. For RYOT’s news organization partners, the main benefit of working with an aggregator like Yahoo News is the expanded distribution opportunity, which will ultimately lead to revenue.
• Working with creative ad sales teams and packaging XR content. It’s important to think outside the box when pitching XR content from a sales perspective. TIME had great success with this on the moon landing project, thinking creatively about retro brands that would want to build off the 1960s theme of the experience. Ultimately, they landed Jimmy Dean as a sponsor.
• Avoiding the question: Why is this in “AR”? We are constantly asked ‘why is this in AR instead of a flat video?’ This isn’t the right question. Of course any type of story can lend itself to any type of treatment — just think about movies based off books and the reasons one medium works sometimes better than the other. Instead we should be asking “How could XR help bring this story to life in a new way?” When working with reporters, it’s better to ask where they are having trouble or what they want to illustrate but can’t yet, to get to a more productive brainstorm about what makes a good immersive story.

Journalism 360 is a global network of storytellers accelerating the understanding and production of immersive journalism. Our mission is to help news organizations, journalists, technologists, content creators and journalism educators experiment with all forms of immersive storytelling, including but not limited to 360 video, augmented reality, virtual reality and mixed reality.

This guide was created by the Online News Association with the support of the John S. and James L. Knight Foundation