Researchers at Facebook have devised a way to extract playable characters from regular videos, creating animated sprites you can control with a joystick or keyboard. Check it out in action:

[YouTube] [Via Luca Prasso]

Five new characters—War Machine, Thor, Black Widow, Rocket and Captain Marvel—are ready to rock in the Marvel Studios’ Avengers: Endgame pack.

[YouTube] [Via]

I love the execution here. I’ve long wanted to automatically put people into LovePop-style 3D cards, but no one listens & I can just say these things here because lol nothing matters. 😌

VR Scout writes,

The first augmented scene converts the bottle into a model of the actual Jack Daniels distillery located in Lynchburg, Tennessee. The second experience takes you step-by-step through their whiskey distillation process, while the third tells the story of Jack Daniels himself.

[YouTube] [Via Asad Ullah Naweed]

Really damn impressive.

Snapchat users in NYC will be able to experience the Landmarker lens starting Sunday, April 14 — the premiere date for “Game of Thrones” Season 8 — and the week thereafter, to see the fantasy creature encase the Flatiron in ice… To access the “GOT” Landmaker Lens, users in the vicinity of the Flatiron Building must tap the Snapchat camera and select the “Game of Thrones” Lens.

Check out the super cool creative journey of Jonathan Yeo, an accomplished painter who took the leap into 3D via Google’s Tilt Brush VR painting app:

https://twitter.com/jessedamiani/status/1115649449353674752?s=11

How might our virtual assistants notice hidden connections, then bring us together with people together around our needs & abilities? This charming & touching vid from Denmark’s TV2 gives a glimpse of how that might feel.

[YouTube] [Via Ben Jones]

This thing looks quite impressive:

Mill Mascot combines real-time game engine technology with motion sensors, so characters can be puppeteered through hand and facial gestures. This gives directors, creatives and artists the ability to ‘direct’ or ‘perform’ CG character animation live and make creative decisions on the fly.

The system by-passes the lengthy timelines involved with traditional animation and pre-rendered content, reducing animation and rendering time down to 42 milliseconds per frame.

[Vimeo]

Cheeky & on-brand, in more ways that one:

Mobile users can overlay digital images of a blazing inferno that burns away competitors’ ads to reveal a mobile coupon for a free whopper sandwich from the chain’s nearest location.

[YouTube]

Kind of a cool idea—even if I can’t embed the damn video.

NatGeo’s Lost Treasures of the Maya Snake Kings makes some really cool use of augmented reality to visualize ruins revealed via LIDAR scans. Check it out:

[YouTube]

Congrats to Paul Debevec, Xueming Yu Wan-Chun Alex Ma, and their former colleague Timothy Hawkins for the recognition of their groundbreaking Light Stage work! 

[YouTube]

Now they’re working with my extended team:

“We try to bring our knowledge and background to try to make better Google products,” Ma says. “We’re working on improving the realism of VR and AR experiences.”

I go full SNL Sue thinking about what might be possible.

Oh, and they worked on Ready Player One (nominated for Best Visual Effects this year) and won for Blade Runner 2049 last year:

Just prior to heading to Google, they worked on “Blade Runner 2049,” which took home the Oscar for Best Visual Effects last year and brought back the character Rachael from the original “Blade Runner” movie. The new Rachael was constructed with facial features from the original actress, Sean Young, and another actress, Loren Peta, to make the character appear to be the same age she was in the first film.

Check out their work in action:

[YouTube 1 & 2]

Heh—I’m charmed that Hugh Jackman has invited our Childish Gambino sticker to his workouts. 😛⽖⽖💪

View this post on Instagram

New guy. @dogpound #childishgambino #BR #Matty

A post shared by Hugh Jackman (@thehughjackman) on Feb 15, 2019 at 4:59am PST

[Via Bilawal Sidhu]

A number of our partner teams have been working on both the foundation for browser-based ML & on cool models that can run there efficiently:

We are excited to announce the release of BodyPix, an open-source machine learning model which allows for person and body-part segmentation in the browser with TensorFlow.js. With default settings, it estimates and renders person and body-part segmentation at 25 fps on a 2018 15-inch MacBook Pro, and 21 fps on an iPhone X. […]

This might all make more sense if you try a live demo here.

Check out this post for more details.

I’m so pleased to say that my team’s face-tracking tech (which you may have seen powering AR effects in YouTube Stories and elsewhere) is now available for developers to build upon:

ARCore’s new Augmented Faces API (available on the front-facing camera) offers a high quality, 468-point 3D mesh that lets users attach fun effects to their faces. From animated masks, glasses, and virtual hats to skin retouching, the mesh provides coordinates and region specific anchors that make it possible to add these delightful effects.

“Why do you keep looking at King Midas’s wife?” my son Finn asked as I was making this GIF the other day. :-p

Check out details & grab the SDKs:

• Native
• Unity
• Unreal
• iOS
• Sceneform

We can’t wait to see what folks build with this tech, and we’ll share more details soon!

I’m really pleased to see that augmented reality navigation has gone into testing with Google Maps users:

On the Google AI Blog, the team gives some insights into the cool tech at work:

We’re experimenting with a way to solve this problem using a technique we call global localization, which combines Visual Positioning Service (VPS), Street View, and machine learning to more accurately identify position and orientation. […]

VPS determines the location of a device based on imagery rather than GPS signals. VPS first creates a map by taking a series of images which have a known location and analyzing them for key visual features, such as the outline of buildings or bridges, to create a large scale and fast searchable index of those visual features. To localize the device, VPS compares the features in imagery from the phone to those in the VPS index. However, the accuracy of localization through VPS is greatly affected by the quality of the both the imagery and the location associated with it. And that poses another question—where does one find an extensive source of high-quality global imagery?

Read on for the full story.

This is America… augmented by Childish Gambino on Pixel:

The Childish Gambino Playmoji pack features unique moves that map to three different songs: “Redbone,” “Summertime Magic,” and “This is America.” Pixel users can start playing with them today using the camera on their Pixel, Pixel XL, Pixel 2, Pixel 2 XL, Pixel 3 and Pixel 3 XL.

And with some help from my team:

He even reacts to your facial expressions in real time thanks to machine learning—try smiling or frowning in selfie mode and see how he responds.

Enjoy!

I love assistive superpowers like this work from Caltech:

VR Scout shares numerous details:

[T]he team used the Microsoft HoloLens’s capability to create a digital mesh over a “scene” of the real-world. Using unique software called Cognitive Augmented Reality Assistant (CARA), they were able to convert information into audio messages, giving each object a “voice” that you would hear while wearing the headset. […]

If the object is at the left, the voice will come from the left side of the AR headset, while any object on the right will speak out to you from the right side of the headset. The pitch of the voice will change depending on how far you are from the object.

Meanwhile Huawei is using AI to help visually impaired users “hear” facial expressions:

Facing Emotions taps the Mate 20 Pro’s back cameras to scan the faces of conversation partners, identifying facial features like eyes, nose, brows, and mouth, and their positions in relation to each other. An offline, on-device machine learning algorithm interprets the detected emotions as sounds, which the app plays on the handset’s loudspeaker.

[YouTube] [Via Helen Papagiannis]

“Lantern-jawed freak!” Yes, now you too can resemble what Phil Hartman as Ed McMahon called Jay Leno, thanks to face-fun app Meograph (App Store):

[YouTube]

Holy crap—Google Earth Studio is like flying a drone… that goes to outer space:

Prediction: The Micronaxx & I are gonna crush some school projects with this. 🙂 Meanwhile we’ll try to look through mom’s window at Adobe, below.

[YouTube]

Heh—turn your iPhone + a rolled-up paper tube into a light saber:

Alternately, try using a tattoo to control Sonic the Hedgehog:

Here's a direct from the screen version of my AR tat pic.twitter.com/CuIYaDZT6m

— Olo Sabandija (@olomerol) November 19, 2018

[YouTube]

Wow—check out this amazing fly-through from Oddviz:

Orphanages are dense and harmonious living spaces housing hundreds of children under same roof simultaneously. Abandoned Jewish Orphanage Building in Ortaköy (OHR-tah-keuy) Istanbul (also known as El Orfelinato) has been home for thousand lives during its century old history. It holds the memory of the past in worn stairs and layers of paint.

El Orfelinato means ‘The Orphanage’ in Spanish. The name has been used by Sephardi Jews (Jews from Spain) community in Istanbul for decades. Sephardi Jews have a 500 year history in Istanbul since they were forced to migrate with mass conversions and executions by Catholic Monarchs in Iberia in 15th century.

oddviz sheds light upon the visual and spatial memory of El Orfelinato, documenting it as it is with photogrammetry and presenting it in doll house view.

And if that’s up your alley, check out their similar Hotel:

[Vimeo] [Via]

I love this mind-bending, AR-style video manipulation from Russian artist Vladimir Tomin:

[Vimeo] [YouTube] [Via]

Prior to joining Google, my teammate Cristian Sminchisescu & his crew developed some neat techniques (see paper) for estimating two human bodies as 3D objects, then transferring clothing from one to another. Check it out:

[YouTube]

Pretty cool stuff, though at the moment it seems to require using a pre-captured background:

When overlaying a digital costume onto a body using pose matching, several parts of the person’s cloth or skin remain visible due to differences in shape and proportions. In this paper, we present a practical solution to these artifacts which requires minimal costume parameterization work, and a straightforward inpainting approach.

[YouTube] [Via Steve Toh]

I wish I could find the joke video someone made when Google acquired the beloved email client Sparrow back in 2012. The vid presented itself as a cheerful tutorial on “How to prepare for the Sparrow Google migration” that basically went like this: 

1. Open your Applications folder.
2. Locate the Sparrow app icon.
3. Now, carefully just drag it to the trash.
4. Empty the trash.
5. And voila, now you’re ready for Google’s ownership of this great app!

I’ve thought of it countless times over the years, and I think of it now recalling BumpTop, a 3D desktop GUI that I first wrote about 13 years ago (!):

Google acquired the tech & team back in 2010, and then… nothing, as far as I know. (All this predated my time at the company, so I have zero inside info.) Empty the trash, and voila!

But evidently not content to let things die, the BumpTop team has reemerged with Spatial, a new approach to team collaboration done via virtual & augmented reality. Check it out:

Will Spatial have more lasting impact than BumpTop? I have no idea! But I look forward to trying it out, and I’m sure glad that teams like this are busy trying to make the world more delightful.

[YouTube 1 & 2]

I’m so pleased to reveal what we’ve been working on for quite some time—the new Playground augmented reality mode on Google Pixel devices!

Playground brings you more powerful AR experiences and uses AI to recommend content for expressing yourself in the moment. You can make your photos and videos come to life with Playmoji—characters that react to each other and to you—and tell a richer story by adding fun captions or animated stickers.

Playground makes real-time suggestions to recommend content based on the scene you’re in. Are you walking your dog? Cooking in the kitchen? Gardening in the backyard? Playground uses advanced computer vision and machine learning to recommend relevant Playmoji, stickers and captions to populate the scene.

My team contributed tech that enables selfie stickers (using realtime segmentation to let characters stand behind you), reactive stickers (those that respond to humans in the frame), object tracking (so that you can attach stickers to moving elements like pets & hands), and glue that helps the pieces communicate. Happily, too, we’re just getting warmed up.

Oh, and stay tuned for Gambeezy!

[YouTube]

“I’m Crushing Your Head,” meet augmented reality (starting around 40s or so):

[YouTube]

Nifty, even if it doesn’t include the actual images produced on-device. More details.

Sounds pretty cool:

Today we’re announcing that VR Creator Lab is coming to London. Participants will receive between $30,000 and $40,000 USD in funding towards their VR project, attend a three day “boot camp” September 18-20, 2018, and receive three months of training from leading VR instructors and filmmakers.

Applications are open through 5pm British Summer Time on August 6, 2018. YouTube creators with a minimum of 10,000 subscribers and independent filmmakers are eligible.

[YouTube]

Wow: You can fly through some amazing goals thanks to the Times graphics staff using 3D illustration package Mental Canvas to convert single images into 3D videos. Check it out (click here if the vid below doesn’t load):

Somehow I’d never heard of Mental Canvas previously. Looks rather amazing: 

[Via Mark Dochtermann & Chris Bregler]

Can low-res YouTube footage be used to generate a 3D model of a ballgame—one that can then be visualized from different angles & mixed into the environment in front of you? Kinda, yeah!

Per TechCrunch,

The “Soccer On Your Tabletop” system takes as its input a video of a match and watches it carefully, tracking each player and their movements individually. The images of the players are then mapped onto 3D models “extracted from soccer video games,” and placed on a 3D representation of the field. Basically they cross FIFA 18 with real life and produce a sort of miniature hybrid.

[YouTube]

Sounds handy for storytellers embracing new perspectives:

VR180 Creator currently offers two features for VR videos. “Convert for Publishing” takes raw fisheye footage from VR180 cameras like the Lenovo Mirage Camera and converts it into a standardized equirect projection. This can be edited with the video editing software creators already use, like Adobe Premiere and Final Cut Pro. “Prepare for Publishing” re-injects the VR180 metadata after editing so that the footage is viewable on YouTube or Google Photos in 2D or VR.

You can learn more about how to use VR180 Creator here and you can download it here.

[Via]

Hmm—I’m intrigued by the filmmaking-for-kids possibilities here, but deeply ambivalent about introducing screen time into one of the great (and threatened) pure-imagination oases in my kids’ lives:

LEGO + AR + Apple at WWDC! pic.twitter.com/TXlx0pyTz4

— CNET (@CNET) June 4, 2018

Up to four friends can play in the same set on four different iOS devices, and notably all of the virtual aspects of the LEGO AR app will be connected to physical LEGO sets. “We can save our entire world back into our physical set, and pick up where we left off later,” Sanders said.