I used to enjoy busting out that humblebrag on Google Photos (and before that, at Adobe) where I got to work with John Schlag. Troubles in the VFX industry yielded a windfall of imaging talent for Google (which occupies the former LA office of Rhythm & Hues, FWIW), and we had a real murderer’s row of talent from DreamWorks, PDI, Sony, and other shops. (There’s so much potential yet to realize… but I digress.)
Are Irish eyes smiling? I should ask my teammate Avneesh to scan me & find out:
Google Research presents a machine learning based approach to infer select facial action units and expressions entirely by analyzing a small part of the face while the user is engaged in a virtual reality experience. Specifically, images of the user’s eyes captured from an infrared (IR) gaze-tracking camera within a VR headset are sufficient to infer at least a subset of facial expressions without the use of any external cameras or additional sensors.
Photographer Jesse Watson captured 2452 images with his wide-angle Nikon D810 setup, culling & cropping the results into a 6K time-lapse using After Effects and Premiere Pro. Check out the beautiful results (and other geeky details here):
You know how Google Assistant can say “Hey, [stateyourname], you should probably leave for the airport by 5 to make it in time for your 7 o’clock flight?” I want it to also say, “You know, it’s Mother’s Day on Sunday. Would you like this photo book to show up on your mom’s doorstep then together with some nice flowers?” Take my money, robot; make me into a better son!
Clearly such work involves a lot of moving parts & hard-to-define qualities (e.g. whether the memories evoked by an image are happy or sad may change greatly depending on things entirely outside the pixels). On the visual quality front, however, my teammates are making interesting progress. As Engadget writes,
If Google has its way, AI may serve as an art critic. It just detailed work on a Neural Image Assessment (NIMA) system that uses a deep convolutional neural network to rate photos based on what it believes you’d like, both technically and aesthetically. It trains on a set of images based on a histogram of ratings (such as from photo contests) that give a sense of the overall quality of a picture in different areas, not just a mean score or a simple high/low rating.
Dwayne Collins’ daughter was diagnosed with a rare eye condition, and none of the doctors they saw could help in any meaningful way. He turned to training videos on YouTube to learn techniques for building prosthetic eyes. It worked—and now he owns his own prosthetics clinic.