Experimenting with Light Fields

We’ve always believed in the power of virtual reality to take you places. That’s why we created Expeditions, to transport people around the world to hundreds of amazing, hard-to-reach or impossible-to-visit places. It’s why we launched Jump, which lets professional creators film beautiful scenes in stereoscopic 360 VR video, and it’s why we’re introducing VR180, a new format for anyone—even those unfamiliar with VR technology—to capture life’s special moments.

But to create the most realistic sense of presence, what we show in VR needs to be as close as possible to what you’d see if you were really there. When you’re actually in a place, the world reacts to you as you move your head around: light bounces off surfaces in different ways and you see things from different perspectives. To help create this more realistic sense of presence in VR, we’ve been experimenting with Light fields.

Light fields are a set of advanced capture, stitching, and rendering algorithms. Much more work needs to be done, but they create still captures that give you an extremely high-quality sense of presence by producing motion parallax and extremely realistic textures and lighting. To demonstrate the potential of this technology, we’re releasing “Welcome to Light Fields,” a free app available on Steam VR for HTC Vive, Oculus Rift, and Windows Mixed Reality headsets. Let’s take a look at how it works.

Capturing and processing a light field

With light fields, nearby objects seem near to you—as you move your head, they appear to shift a lot. Far-away objects shift less and light reflects off objects differently, so you get a strong cue that you’re in a 3D space. And when viewed through a VR headset that supports positional tracking, light fields can enable some truly amazing VR experiences based on footage captured in the real world.

This is possible because a light field records all the different rays of light coming into a volume of space. To record them, we modified a GoPro Odyssey Jump camera, bending it into a vertical arc of 16 cameras mounted on a rotating platform.

Left: A time lapse video of recording a spherical light field on the flight deck of Space Shuttle Discovery.
Right: Light field rendering allows us to synthesize new views of the scene anywhere within the spherical volume by sampling and interpolating the rays of light recorded by the cameras on the rig.

It takes about a minute for the camera rig to swing around and record about a thousand outward-facing viewpoints on a 70cm sphere. This gives us a two-foot wide diameter volume of light rays, which determines the size of the headspace that users have to lean around in to explore the scenes once they are processed. To render views for the headset, rays of light are sampled from the camera positions on the surface of the sphere to construct novel views as seen from inside the sphere to match how the user moves their head. They’re aligned and compressed in a custom dataset file that’s read by special rendering software we’ve implemented as a plug-in for the Unity game engine.

Light Fields.PNG

Recording the World with Light Fields

We chose a few special places to try out our light field-recording camera rig. We love the varnished teak and mahogany interiors at the Gamble House in Pasadena, the fragments of glossy ceramic and shiny mirrors adorning the Mosaic Tile House in Venice, and the sun-filled stained glass window at St. Stephen’s Church in Granada Hills. Best of all, the Smithsonian Institute’s Air and Space Museum and 3D Digitization Office gave us access to NASA’s Space Shuttle Discovery, providing an astronaut’s view inside the orbiter’s flight deck which has never been open to the public.  And we closed with recording a variety of light fields of people, experimenting with how eye contact can be made to work in a 6-degrees-of-freedom experience.

Try “Welcome to Light Fields”

VR video is a promising technology for exploring the world, and while they are still an experiment, light fields show a new level of how convincing virtual reality experiences can be. We hope you enjoy our “Welcome to Light Fields” experience, available now on Steam VR. Take the seven-minute Guided Tour to learn more about the technology and the locations, and then take your time exploring the spaces in the Gallery. This is only the beginning, and lots more needs to be done, but we’re excited about this step toward more realistic capture for VR.

Leave a Reply

Your email address will not be published. Required fields are marked *