“Why doesn’t it recognize The Finger?!” asks my indignant, mischievous 10-year-old Henry, who with his brother has offered to donate a rich set of training data. 🙃
Juvenile amusement notwithstanding, I’m delighted that my teammates have released a badass hand-tracking model, especially handy (oh boy) for use with MediaPipe (see previous), our open-source pipeline for building ML projects.
Today we are announcing the release of a new approach to hand perception, which we previewed CVPR 2019 in June, implemented in MediaPipe—an open source cross platform framework for building pipelines to process perceptual data of different modalities, such as video and audio. This approach provides high-fidelity hand and finger tracking by employing machine learning (ML) to infer 21 3D keypoints of a hand from just a single frame. Whereas current state-of-the-art approaches rely primarily on powerful desktop environments for inference, our method achieves real-time performance on a mobile phone, and even scales to multiple hands. We hope that providing this hand perception functionality to the wider research and development community will result in an emergence of creative use cases, stimulating new applications and new research avenues.
I’ve been collaborating with these folks for a few months & am incredibly excited about this feature:
With a beta feature called Live View, you can use augmented reality (AR) to better see which way to walk. Arrows and directions are placed in the real world to guide your way. We’ve tested Live View with the Local Guides and Pixel community over the past few months, and are now expanding the beta to Android and iOS devices that support ARCore and ARKit starting this week.
Like the Dos Equis guy, “I don’t always use augmented reality—but when I do, I navigate in Google Maps.” We’ll look back at these first little steps (no pun intended) as foundational to a pretty amazing new world.
In case you’ve ever wondered about the math behind placing, say, virtual spiders on my kid works, wonder no more: my teammates have published lots o’ details.
One of the key challenges in enabling AR features is proper anchoring of the virtual content to the real world, a process referred to as tracking. In this paper, we present a system for motion tracking, which is capable of robustly tracking planar targets and performing relative-scale 6DoF tracking without calibration. Our system runs in real-time on mobile phones and has been deployed in multiple major products on hundreds of millions of devices.
I’ve gotta say, they look pretty gnarly in 3D (below). I wonder whether these creepy photogrammetry(?)-produced results are net-appealing to customers. I have the same question about AR clothing try-on: even if we make it magically super accurate, do I really want to see my imperfect self rocking some blazer or watch, or would I rather see a photo of Daniel Craig doing it & just buy the dream that I’ll look similar?
Fortunately, I found the visual appearance much more pleasing when rendered in AR on my phone vs. when rendered in 3D on my Mac, at least unless I zoomed in excessively.
Inside the NYT, readers will find a full page ad in the Main News section and quarter page ads both in Arts and Business sections of the paper with a CTA encouraging readers to scan the ads with Google Lens, where they might find that things are stranger than they seem. 🙃
Tangentially related: this is bonkers:
This is amazing—Stranger Things 3's Starcourt Mall wasn't a sound stage. It was all built inside an actual dying mall in Georgia. And the set designers made more than simple storefronts—they made FULL INTERIORS, even for stores that were never seen on-screen… pic.twitter.com/v5RahFLPeR
EasyJet has launched a brand new hand luggage app that enables customers to check their bag size before they leave for the airport. The technology offers 3D augmented reality and shows if the baggage will fit the cabin bag dimensions correctly.