The news about Resident Evil’s use of photogrammetry is just the latest buzz in the building hum of excitement for the topic. Photogrammetry as a science dates back to more than 100 years ago. Once used mostly for surveying, the process has since been adapted by filmmakers, and, more recently, by video game developers. To put it simply, photogrammetry software takes in many photos from many angles of the same object/scene and builds a fully textured 3D model. A quick google search would be more informative towards the actual science and history of of the process than I. As far as I’m concerned, it’s magic.
Aside from Resident Evil, it can be found in the newest Star Wars Battlefront, Battlefield 1, The Vanishing of Ethan Carter, L.A. Noire, and CD Projekt Red’s up and coming game Cyberpunk 2077. Let’s talk about L.A. Noire for a second.
L.A. Noire utilized a complex method of photogrammetry in order to create its uncannily realistic facial animations and models. When the game was at its height of promotion, they were hammering the idea home by explaining that the actors in their game had to wear makeup.
L.A. Noire did well critically, with most praise going to the story, the realistic depiction of 20s LA, and the technology behind the faces. But the game was famously so expensive to make that the poor sales ended up putting Team Bondi out of business. Fast forward to today, and I just replicated the theory of their technology (sans animation), at home with my cell phone and a trial run of Agisoft Photoscan.
Now, there are obvious problems with the model I made. There are pieces of empty space around certain edges of the bananas, and areas where the textures seem to melt or warp. Since the software works by finding common patterns in multiple images, there must be a blanket of even lighting from all angles around the object being scanned or artifacts similar to those found on mine will manifest. For example, letting the camera’s shadow fall across the object could confuse the software and cause a hole in the final model. Similarly, reflective surfaces cause problems because they lack consistency from one shot to the other. So, to summarize, my bananas have problems because of my shoddy camera work.
But that is amazing. If that model is the result of an amateur giving 10%, imagine what that means for the concepts driving the software and how far photogrammetry has come as a science.
The point is that a method similar to the 3D scanning that put Team Bondi dangerously over budget was just done by me in my underwear in less than an hour. No, it wasn’t perfect, it’s a static banana for Pete’s sake, but that’s just my fault. The technology itself is compelling enough to warrant the iteration it is receiving.
First, we painted moments. It took time, practice, and talent. Then cameras were invented. They took time too, though, and training to work. People weren’t running out of their doors to buy their own camera. But then something happened. Cameras got better, simpler to use, cheaper. Gone was the prerequisite knowledge, the days of 15 minute captures. Instead, people started to smile and say cheese. The camera re-branded itself a consumer product. Now we all walk around with a camera in our pockets. All for the sake of saving a snapshot of a moment. I think that same compulsion can move photogrammetry out of the exclusive hands of professionals, and into each of ours as well.
There are three areas of improvement I can think of that will need to happen before the “photogrammetry mode” appears in our camera apps. The first two are easy: cell phones need to get more processing power, and they need better cameras. Because those trends are already firmly in place, it is left up to developers to continue pushing out new improved versions of photogrammetry software, making it more user friendly and more adept at filtering out distracting lighting/reflections. A tough goal, I’m sure, but I believe in them. Somehow those crazy coding bastards always do it.
You can get a glimpse of what a world of consumer-oriented photogrammetry looks like by plopping on a VR headset and loading up Valve’s Destinations app on Steam. Valve has put up some of their own (very well done (and in 1:1 scale)) examples that include an English church and Mars. Yes, that Mars. What is great about Valve’s offerings is that they utilize Destinations‘ open source scripting to create floating text boxes in specific locations which explain what you’re looking at, their process for creating it, and how any mistakes were made. These function very much like the plaques at a museum, but with the added benefit of being micro lessons in photogrammetry best practices. Also, I virtually stood next to the Mars rover. It’s a lot bigger than I expected.
Destinations also does a great job of opening it’s toolbox up to anyone. People all over the world are already beginning to scan their bedrooms and put them up for free. There’s a rooftop in India to explore, or an impressive man-cave in Vancouver. Someone scanned a pile of popcorn and scaled it up to gigantic proportions. Someone else gave similar treatment to a typewriter. These people are capturing perspectives of life in 2016, all over the world, and you can walk around inside them all.
Perhaps in ten years a wedding album will be a memory to step into. Tomorrow’s social media may be filled with little portals to past events. Future Yelp reviews could allow someone to size up the restaurant’s booths before leaving the house. Memes themselves might soon go 360 degrees. Until then, I’ll just be over here, trying to get my cat to sit still long enough to scan her into my PC.