Category Archives: AR/VR

AR: Google & Fiat-Chrysler bring super detailed models to your driveway

Psst—wanna see a multi-gig, super detailed 3D model appear in your driveway almost instantaneously?

I’m delighted to say that our work in cloud-rendered streaming 3D is paying off via this year’s virtual CES show. Per the Google Cloud blog:

As part of Fiat Chrysler’s Virtual Showroom CES event, you can experience the new innovative 2021 Jeep Wrangler 4xe by scanning a QR code with your phone. You can then see an Augmented Reality (AR) model of the Wrangler right in front of you—conveniently in your own driveway or in any open space. Check out what the car looks like from any angle, in different colors, and even step inside to see the interior with incredible details.

A bit on how it works:

The Cloud AR tech uses a combination of edge computing and AR technology to offload the computing power needed to display large 3D files, rendered by Unreal Engine, and stream them down to AR-enabled devices using Google’s Scene Viewer. Using powerful rendering servers with gaming-console-grade GPUs, memory, and processors located geographically near the user, we’re able to deliver a powerful but low friction, low latency experience.

This rendering hardware allows us to load models with tens of millions of triangles and textures up to 4k, allowing the content we serve to be orders of magnitude larger than what’s served on mobile devices (i.e., on-device rendered assets).

And to try it out:

Scan the QR code below, or check out the FCA CES website. Depending on your OS, device, and network strength, you will see either a photorealistic, cloud-streamed AR model or an on-device 3D car model, both of which can then be placed in your physical environment.

AR_Wrangler-4xe.jpg

AR: Baby Yoda comes to Google Search

Putting the “AR” in “Galaxy far, far away…” 😌

Just don’t expect to get a Baby Yoda Hellraiser Pinhead version. 😬

Also, remember that if you have a Pixel 5G or a compatible Android 5G device, you can install The Mandalorian” AR Experience.

#ThisIsTheWay

AR: Come try new cars & makeup in Google Search

I’m delighted to be closing out 2020 on a pair of high notes, welcoming the arrival of my two biggest efforts from the last year+.

First, Google Search now supports 150+ new cars that you can view in 3D and AR (via iPhone or Android device), including in beautiful cloud-rendered quality (provided you have a good connection & up-to-date Android). As we initially previewed in October:

Bring the showroom to you with AR

You can easily check out what the car looks like in different colors, zoom in to see intricate details like buttons on the dashboard, view it against beautiful backdrops and even see it in your driveway. We’re experimenting with this feature in the U.S. and working with top auto brands, such as Volvo and Porsche, to bring these experiences to you soon.

Second, you can try on AR beauty products right through Search:

Now, when you search for a lipstick or eyeshadow product, like L’Oreal’s Infallible Paints Metallic Eyeshadow, you can see what it looks like on a range of skin tones and compare shades and textures to help you find the right products.

To help you find the perfect match, you can now also virtually try makeup products right from the Google app.

New tech promises super fast, high quality background removal

Google researchers Ira Kemelmacher-Shlizerman, Brian Curless, and Steve Seitz have been working with University of Washington folks on tech that promises “30fps in 4K resolution, and 60fps for HD on a modern GPU.”

Our technique is based on background matting, where an additional frame of the background is captured and used in recovering the alpha matte and the foreground layer.

Check it out:


See the PDF, abstract, and GitHub repo for details.

Google Photos rolls out Cinematic Photos & more

Nearly 20 years ago, on one of my first customer visits as a Photoshop PM, I got to watch artists use PS + After Effects to extract people from photo backgrounds, then animate the results. The resulting film—The Kid Stays In The Picture—lent its name to the distinctive effect (see previous).

Now I’m delighted that Google Photos is rolling out similar output to its billion+ users, without requiring any effort or tools:

We use machine learning to predict an image’s depth and produce a 3D representation of the scene—even if the original image doesn’t include depth information from the camera. Then we animate a virtual camera for a smooth panning effect—just like out of the movies.

Photos is also rolling out new collages, like this:

And they’re introducing new themes in the stories-style Memories section up top as well:

Now you’ll see Memories surface photos of the most important people in your life…  And starting soon, you’ll also see Memories about your favorite things—like sunsets—and activities—like baking or hiking—based on the photos you upload.

Enjoy!

AR: Google Maps can point you towards your friends

I love these simple, practical uses of augmented reality. The Maps team writes,

Last month, we launched Live View in Location Sharing for Pixel users, and we’ll soon expand this to all Android and iOS users around the globe. When a friend has chosen to share their location with you, you can easily tap on their icon and then on Live View to see where and how far away they are–with overlaid arrows and directions that help you  know where to go.

Live View in Location Sharing will soon expand to all Android and iOS users globally on ARCore and ARKit supported phones.

They’re also working hard to leverage visual data & provide better localization and annotation.

With the help of machine learning and our understanding of the world’s topography, we’re able to take the elevation of a place into account so we can more accurately display the location of the destination pin in Live View. Below, you can see how Lombard Street—a steep, winding street in San Francisco—previously appeared far off into the distance. Now, you can quickly see that Lombard Street is much closer and the pin is aligned with where the street begins at the bottom of the hill.

50 new AR animals arrive in Google search

“If your dog woke up 10 times its current size, it would lick you; if your cat woke up 10 times bigger, it would eat you,” or so I’ve heard.

In any case, building on the world’s viral (in every sense) adoption of AR animals already in search, my team has added a bunch more:

https://twitter.com/Google/status/1337506620612485120?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1337506620612485120%7Ctwgr%5E%7Ctwcon%5Es1_&ref_url=https%3A%2F%2Fwww.theverge.com%2F2020%2F12%2F12%2F22171601%2Fgoogle-adds-augmented-reality-ar-animals-search-3d

The Verge writes,

When Google started putting 3D animals in Search last year it only had a few standard animals available like a tiger, a lion, a wolf, and a dog. It added more creatures in March, including alligators, ducks, and hedgehogs. In August, Google made prehistoric creatures and historical artifacts available in AR via its Arts and Culture app— and who among us wouldn’t love to check out the ancient crustacean Cambropachycope up close and personal?

Meanwhile my man Seamus abides. 🐕😌

Google gives apps simultaneous on-device face, hand and pose prediction

From sign language to sports training to AR effects, tracking the human body unlocks some amazing possibilities, and my Google Research teammates are delivering great new tools:

We are excited to announce MediaPipe Holistic, […] a new pipeline with optimized poseface and hand components that each run in real-time, with minimum memory transfer between their inference backends, and added support for interchangeability of the three components, depending on the quality/speed tradeoffs.

When including all three components, MediaPipe Holistic provides a unified topology for a groundbreaking 540+ keypoints (33 pose, 21 per-hand and 468 facial landmarks) and achieves near real-time performance on mobile devices. MediaPipe Holistic is being released as part of MediaPipe and is available on-device for mobile (Android, iOS) and desktop. We are also introducing MediaPipe’s new ready-to-use APIs for research (Python) and web (JavaScript) to ease access to the technology.

Check out the rest of the post for details, and let us know what you create!