Musicians zoom in & comment on paintings through Google’s Art Zoom

Ever wondered what Feist thinks about Bruegel the Elder? Well wonder no more, my friend! She & other musicians have recorded their thoughts on the details of famous paintings. To wit:

More than 10,000 artworks from 208 partners worldwide have been captured with Art Camera and digitized in ultra-high resolution, from the fluffy fabric from which Vivienne Westwood tailored the Keith Haring “Witches” dress, to the almost photographic View of Delft by Vermeer. You can see these works in intricate detail simply by browsing on the Google Arts & Culture app. Explore Art Zoom online at g.co/ArtZoom, or download our free app for iOS or Android.

[YouTube]

AR: Cloaking device… engaged!

Zach Lieberman has been on a tear lately with realtime body-segmentation experiments (see his whole recent feed), and now he’ll ghost ya for real:

It’s crazy to think that this stuff works in realtime on a telephone, when just 7 years ago here’s how Content-Aware Fill looked when applied to video:

ML developers: Come check out MediaPipe

The glue my team developed to connect & coordinate machine learning, computer vision, and other processes is now available for developers:

The main use case for MediaPipe is rapid prototyping of applied machine learning pipelines with inference models and other reusable components. MediaPipe also facilitates the deployment of machine learning technology into demos and applications on a wide variety of different hardware platforms (e.g., Android, iOS, workstations).

If you’ve tried any of the Google AR examples I’ve posted in the last year+ (Playground, Motion Stills, YouTube Stories or ads, etc.), you’ve already used MediaPipe, and  now you can use it to remove some drudgery when creating your own apps.

Here’s a whole site full of examples, documentation, a technical white paper, graph visualizer, and more. If you take it for a spin, let us know how it goes!

NewImage

Check out Fresco, Adobe’s new tablet drawing app

People have been trying to combine the power of vector & raster drawing/editing for decades. (Anybody else remember Creature House Expression, published by Fractal & then acquired by Microsoft? Congrats on also being old! 🙃) It’s a tough line to walk, and the forthcoming Adobe Fresco app is far from Adobe’s first bite at the apple (I remember you, Fireworks).

Back in 2010, I transitioned off of Photoshop proper & laid out a plan by which different mobile apps/modules (painting, drawing, photo library) would come together to populate a share, object-centric canvas. Rather than build the monolithic (and now forgotten) Photoshop Touch that we eventually shipped, I’d advocated for letting Adobe Ideas form the drawing module, Lightroom Mobile form the library, and a new Photoshop-derived painting/bitmap editor form the imaging module. We could do the whole thing on a new imaging stack optimized around mobile GPUs.

Obviously that went about as well as conceptually related 90’s-era attempts at OpenDoc et al.—not because it’s hard to combine disparate code modules (though it is!), but because it’s really hard to herd cats across teams, and I am not Steve Fucking Jobs.

Sadly, I’ve learned, org charts do matter, insofar as they represent alignment of incentives & rewards—or lack thereof. “If you want to walk fast, walk alone; if you want to walk far, walk together.” And everyone prefers “innovate” vs. “integrate,” and then for bonus points they can stay busy for years paying down the resulting technical debt. “…Profit!”

But who knows—maybe this time crossing the streams will work. Or, see you again in 5-10 years the next time I write this post. 😌

[YouTube]

Introducing AR makeup on YouTube

I’m so pleased to be able to talk about the augmented reality try-on feature we’ve integrated with YouTube, leveraging the face-tracking ML tech we recently made available for iOS & Android:

Today, we’re introducing AR Beauty Try-On, which lets viewers virtually try on makeup while following along with YouTube creators to get tips, product reviews, and more. Thanks to machine learning and AR technology, it offers realistic, virtual product samples that work on a full range of skin tones. Currently in alpha, AR Beauty Try-On is available through FameBit by YouTube, Google’s in-house branded content platform.

M·A·C Cosmetics is the first brand to partner with FameBit to launch an AR Beauty Try-On campaign. Using this new format, brands like M·A·C will be able to tap into YouTube’s vibrant creator community, deploy influencer campaigns to YouTube’s 2 billion monthly active users, and measure their results in real time.

As I noted the other day with AR in Google Lens, big things have small beginnings. Stay tuned!

Google makes… a collaborative game level builder?

Hey, I’m as surprised as you probably are. 🙃 And yet here we are:

What if creating games could be as easy and fun as playing them? What if you could enter a virtual world with your friends and build a game together in real time? Our team within Area 120, Google’s workshop for experimental projects, took on this challenge. Our prototype is called Game Builder, and it is free on Steam for PC and Mac.

I’m looking forward to taking it for a spin!

Predicting someone’s gestures from just their words

Whoa—what an eerie, funky thing to undertake: using a computer to anticipate a specific person’s idiosyncratic (yet predictable!) hand gestures based just on a recording of their speech:

Now they just need to automatically add accordions.

As for the training data:

Our speakers come from a diverse set of backgrounds: television show hosts, university lecturers and televangelists. They span at least three religions and discuss a large range of topics from commentary on current affairs through the philosophy of death, chemistry and the history of rock music, to readings in the Bible and the Qur’an.

As always, “This is the strangest life I’ve ever known…”

[YouTube]