Category Archives: VR/AR

Walk through walls in this eye-popping 3D reconstruction & visualization

Wow—check out this amazing fly-through from Oddviz:

Orphanages are dense and harmonious living spaces housing hundreds of children under same roof simultaneously. Abandoned Jewish Orphanage Building in Ortaköy (OHR-tah-keuy) Istanbul (also known as El Orfelinato) has been home for thousand lives during its century old history. It holds the memory of the past in worn stairs and layers of paint.

El Orfelinato means ‘The Orphanage’ in Spanish. The name has been used by Sephardi Jews (Jews from Spain) community in Istanbul for decades. Sephardi Jews have a 500 year history in Istanbul since they were forced to migrate with mass conversions and executions by Catholic Monarchs in Iberia in 15th century.

oddviz sheds light upon the visual and spatial memory of El Orfelinato, documenting it as it is with photogrammetry and presenting it in doll house view.

And if that’s up your alley, check out their similar Hotel:


[Vimeo] [Via]

What might be next for Facebook 3D photos?

Facebook’s 3D photos (generated from portrait-mode images) have quickly proven to be my favorite feature added to that platform in years. Hover or drag over this example:

My crazy three! 😝😍 #007 #HappyHalloween

Posted by John Nack on Wednesday, October 31, 2018

The academic research they’ve shared, however, promises to go farther, enabling VR-friendly panoramas with parallax. The promise is basically “Take 30 seconds to shoot a series of images, then allow another 30 seconds for processing.” The first portion might well be automated, enabling the user to simply pan slowly across a scene.


This teaser vid shows how scenes are preserved in 3D, enabling post-capture effects like submerging them in water:

Will we see this ship in FB, and if so when? Your guess is as good as mine, but I find the progress exciting.


Costume-Aware Fill? Disney shows off neat AR clothing tech

Pretty cool stuff, though at the moment it seems to require using a pre-captured background:

When overlaying a digital costume onto a body using pose matching, several parts of the person’s cloth or skin remain visible due to differences in shape and proportions. In this paper, we present a practical solution to these artifacts which requires minimal costume parameterization work, and a straightforward inpainting approach.


[YouTube] [Via Steve Toh]

AR: A virtual desktop on your actual desktop?

Here’s a pretty darn clever idea for navigating among apps by treating your phone as a magic window into physical space.

You use the phone’s spatial awareness to ‘pin’ applications in a certain point in space, much like placing your notebook in one corner of your desk, and your calendar at another… You can create a literal landscape of apps that you can switch between by simply switching the location of your phone.



BumpTop is back… now in AR!

I wish I could find the joke video someone made when Google acquired the beloved email client Sparrow back in 2012. The vid presented itself as a cheerful tutorial on “How to prepare for the Sparrow Google migration” that basically went like this: 

  1. Open your Applications folder.
  2. Locate the Sparrow app icon.
  3. Now, carefully just drag it to the trash.
  4. Empty the trash.
  5. And voila, now you’re ready for Google’s ownership of this great app!

I’ve thought of it countless times over the years, and I think of it now recalling BumpTop, a 3D desktop GUI that I first wrote about 13 years ago (!):

Google acquired the tech & team back in 2010, and then… nothing, as far as I know. (All this predated my time at the company, so I have zero inside info.) Empty the trash, and voila!

But evidently not content to let things die, the BumpTop team has reemerged with Spatial, a new approach to team collaboration done via virtual & augmented reality. Check it out:

Will Spatial have more lasting impact than BumpTop? I have no idea! But I look forward to trying it out, and I’m sure glad that teams like this are busy trying to make the world more delightful.


[YouTube 1 & 2]

Pixel 3 Playground is here!

I’m so pleased to reveal what we’ve been working on for quite some time—the new Playground augmented reality mode on Google Pixel devices!

Playground brings you more powerful AR experiences and uses AI to recommend content for expressing yourself in the moment. You can make your photos and videos come to life with Playmoji—characters that react to each other and to you—and tell a richer story by adding fun captions or animated stickers.

Playground makes real-time suggestions to recommend content based on the scene you’re in. Are you walking your dog? Cooking in the kitchen? Gardening in the backyard? Playground uses advanced computer vision and machine learning to recommend relevant Playmoji, stickers and captions to populate the scene.

My team contributed tech that enables selfie stickers (using realtime segmentation to let characters stand behind you), reactive stickers (those that respond to humans in the frame), object tracking (so that you can attach stickers to moving elements like pets & hands), and glue that helps the pieces communicate. Happily, too, we’re just getting warmed up.

Oh, and stay tuned for Gambeezy!