Immersive Case Study: American Food Post-Production
Published July 10, 2020
This case study is part of the broader Guide to Immersive Ethics. Learn more in the overview, including what ethical criteria to consider when starting a new immersive project.
American Food, McClatchy’s AR tour of cultural food enclaves around the United States, centers on the iconic dishes on local restaurant menus.
In post-production
The producers had returned from the field after capturing an array of Vietnamese food from Sacramento’s Little Saigon using photogrammetry. Complex light information from shiny and transparent surfaces confuses the photogrammetry software (in this case, we’re using RealityCapture), leaving holes in our 3D images. What’s more, this bowl was white, providing no textural detail for the software to use to connect the images into a complete model.
The first step should always be to reshoot, if possible. But with such a challenging subject, and little time for a reshoot, we were faced with a decision: We can either leave the rough spots or make edits to ensure that the surfaces are as faithful to the original as possible.
For guidance, we looked at industry precedent for visual media, in particular these two rules:
From the NPPA: “Editing should maintain the integrity of the photographic images’ content and context. Do not manipulate images or add or alter sound in any way that can mislead viewers or misrepresent subjects.”
And from NPR’s visual ethics code: “When reporting on news events, the photographs we take and use depict them truthfully, honestly and without bias. They are only enhanced for technical clarity — to correct color or improve contrast, for example. We are careful in how we crop them to ensure that the scene is in proper context.”
For 2D photos, this means limiting edits to modifications like adjusting exposure in the darkroom or color correction in Photoshop. Editors shouldn’t take liberties to add Instagram-like filters that might suggest the photo was taken at a different time of day, or go even further by removing or adding elements in a scene.
With this spirit in mind, we came up with this policy for editing 3D images:
Captured objects may be edited for technical and visual clarity if the edits are as faithfully representative as possible.
So what does this mean when it’s put into practice?
Since photogrammetry requires lots of photos, we had a ton of visual information to reference as we edited the chewed-up dishes, like this one of Vietnamese soup, for visual clarity. We looked closely at the photographs and used a digital sculptor’s toolkit — “digital clay” in the 3D sculpting software ZBrush — to fill in the missing parts of the shape.
Then, to get the color and texture right, we used a combination of the clone stamp tool and hard and soft brushes in Photoshop to copy the surface from a photograph.
With those two techniques working together (within a limited time frame), the final image is as faithfully representative as possible.
In an effort to be transparent, we run an editor’s note in our app explaining the edits and defining this type of object as a “photo-realistic model.”
The glass bowl
Another test of our ethics code came up when we wanted to feature a model of the dessert dish bouza in the episode taking viewers to Little Arabia in Anaheim, Calif.
In this case, the area’s iconic stretchy “ice cream” was served in a glass bowl, and the model of the dish had more than a few small holes:
To fix this, we would have to build the bowl from scratch referencing the original scan that had information on the circumference and photographs of the surface’s texture. Then, we put the photogrammetric model of the bouza inside the re-creation of the glass bowl.
While getting as close to the original object as possible was still the goal, we think this goes beyond simply editing for technical and visual clarity. We also know it’s important to give ourselves limits to how and when to use a realistic re-creation, so we included an editor’s note at the beginning of the episode.
