A number of our partner teams have been working on both the foundation for browser-based ML & on cool models that can run there efficiently:
We are excited to announce the release of BodyPix, an open-source machine learning model which allows for person and body-part segmentation in the browser with TensorFlow.js. With default settings, it estimates and renders person and body-part segmentation at 25 fps on a 2018 15-inch MacBook Pro, and 21 fps on an iPhone X. […]
This might all make more sense if you try a live demo here.
ARCore’s new Augmented Faces API (available on the front-facing camera) offers a high quality, 468-point 3D mesh that lets users attach fun effects to their faces. From animated masks, glasses, and virtual hats to skin retouching, the mesh provides coordinates and region specific anchors that make it possible to add these delightful effects.
“Why do you keep looking at King Midas’s wife?” my son Finn asked as I was making this GIF the other day. :-p
We’re experimenting with a way to solve this problem using a technique we call global localization, which combines Visual Positioning Service (VPS), Street View, and machine learning to more accurately identify position and orientation. […]
VPS determines the location of a device based on imagery rather than GPS signals. VPS first creates a map by taking a series of images which have a known location and analyzing them for key visual features, such as the outline of buildings or bridges, to create a large scale and fast searchable index of those visual features. To localize the device, VPS compares the features in imagery from the phone to those in the VPS index. However, the accuracy of localization through VPS is greatly affected by the quality of the both the imagery and the location associated with it. And that poses another question—where does one find an extensive source of high-quality global imagery?
The Childish Gambino Playmoji pack features unique moves that map to three different songs: “Redbone,” “Summertime Magic,” and “This is America.” Pixel users can start playing with them today using the camera on their Pixel, Pixel XL, Pixel 2, Pixel 2 XL, Pixel 3 and Pixel 3 XL.
And with some help from my team:
He even reacts to your facial expressions in real time thanks to machine learning—try smiling or frowning in selfie mode and see how he responds.
Wearing a Magic Leap One headset connected to a Wacom Intuos Pro pen tablet, designers can use the separate three-button Pro Pen 3D stylus to control their content on a platform called Spacebridge, which streams 3D data into a spatial computing environment. The program allows multiple people in a room to interact with the content, with the ability to view, scale, move, and sketch in the same environment.
Check out the rest of the Verge article for details. I very much look forward to seeing how this develops.
[T]he team used the Microsoft HoloLens’s capability to create a digital mesh over a “scene” of the real-world. Using unique software called Cognitive Augmented Reality Assistant (CARA), they were able to convert information into audio messages, giving each object a “voice” that you would hear while wearing the headset. […]
If the object is at the left, the voice will come from the left side of the AR headset, while any object on the right will speak out to you from the right side of the headset. The pitch of the voice will change depending on how far you are from the object.
Facing Emotions taps the Mate 20 Pro’s back cameras to scan the faces of conversation partners, identifying facial features like eyes, nose, brows, and mouth, and their positions in relation to each other. An offline, on-device machine learning algorithm interprets the detected emotions as sounds, which the app plays on the handset’s loudspeaker.
This new camera feature, dubbed “Live Stickers,” allows users to produce multiple animated stickers of themselves or friends, and place them in “live” environments before sharing them with the world through social media platforms.
Is it useful? I’m not sure, but I’d welcome your thoughts. You can grab the app for iOS and take it for a spin.