Erik Härkönen recently interned at Adobe, collaborating with several of my teammates on interesting emerging tech. This was all a pleasant surprise to me, as I’d independently stumbled across this fun vid in which he encapsulates some exciting things AI is learning to do with photos:
Monthly Archives: May 2021
In “Beautiful Numbers,” Stefan Sagmeister sees reasons for hope
I have to admit it’s getting better, it’s getting better all the time…
Well, not everything, clearly— but it’s nice to be reminded about human progress and our place on its arc. I also enjoyed the well-executed little animations of Stefan’s ancestors.
BMW art cars go AI
Generative artist Nathan Shipley has been doing some amazing work with GANs, and he recently collaborated with BMW to use projection mapping to turn a new car into a dynamic work of art:
I’ve long admired the Art Cars series, with a particular soft spot for Jenny Holzer’s masterfully disconcerting PROTECT ME FROM WHAT I WANT:
Here’s a great overview of the project’s decades of heritage, including a dive into how Andy Warhol adorned what may be the most valuable car in the world—painting on it at lightning speed:
AI art: GANimated flowers
Years ago my friend Matthew Richmond (Chopping Block founder, now at Adobe) would speak admiringly of “math-rock kids” who could tinker with code to expand the bounds of the creative world. That phrase came to mind seeing this lovely little exploration from Derrick Schultz:
Here it is in high res:
Product development by way of mullets
Current status: attempting to show teammates how we can offer both “Business Up Front” & “Party In The Back.” 🙃
Mo’ betta witchcraft: NVIDIA turns 2D images into 3D
From LiDAR scanners in millions of pockets to AIs that can now generate 3D from 2D, the magic’s getting deep:
NVIDIA Research is revving up a new deep learning engine that creates 3D object models from standard 2D images — and can bring iconic cars like the Knight Rider’s AI-powered KITT to life — in NVIDIA Omniverse.
A single photo of a car, for example, could be turned into a 3D model that can drive around a virtual scene, complete with realistic headlights, tail lights and blinkers.
AR: Snapchat + Lego introduce collaborative lenses 🔥
Awesome! Here’s hoping this kind of multiplayer experience brings more heat & adoption around world-facing AR (vs. just more puppy-dog selfies):
Apply for the Adobe Stock Artist Development Fund
I’m really happy to see Adobe putting skin in the game to increase diversity & inclusion in stock imagery:
Introducing the Artist Development Fund, a new $500,000 creative commission program from Adobe Stock. As an expression of our commitment to inclusion we’re looking for artists who self-identify with and expertly depict diverse communities within their work.
Here’s how it works:
The fund also ensures artists are compensated for their work. We will be awarding funding of $12,500 each to a total of 40 global artists on a rolling basis during 2021. Artist Development Fund recipients will also gain unique opportunities, including having their work and stories featured across Adobe social and editorial channels to help promote accurate and inclusive cultural representation within the creative industry.
Holy cow: Bovine Lego abduction
Heh—I find this fan-proposed set (which is currently collecting votes) udderly charming:
We don’t need to subjugate humanity, we only need 10,000 votes to make LEGO consider turning it into a set. So vote today and avoid unpleasant probing incidents tonight!
Using AI to enhance film dubbing
I’ve always had a soft spot for incredibly crappy film dubbing—especially this Bill Murray SNL classic that I hadn’t seen in 30 years but remember like it was yesterday…
…and not to mention Police Academy (“Damn you, wanna fight? Fight me!!“):
Not every example is so charming, however and now a company called Flawless plans to use neural networks to fit actors’ mouth movements to dialogue:
“The extraction of facial data” — a time-consuming computational process — “runs parallel with the production itself.” The technology strips actors’ faces off, converting their visages into a 3D model, according to Lynes. “This creates millions of 3D models, which the AI uses as reference points,” he says.
“And then, using an existing foreign-language recording of the dialogue, it studies the actor and generates a new 3D model per frame,” he adds. Finally, the imagery is converted back to 2D. Digital effects artists can then manually fix anything that seems off.
Google reveals Project Starline, a next-gen 3D video chat booth
I’m thrilled that a bunch of Google friends (including Dan Goldman, who was instrumental in bringing Content-Aware Fill to Photoshop) have gotten to reveal Project Starline, their effort to deliver breakthrough 3D perception & display to bring people closer together:
Imagine looking through a sort of magic window, and through that window, you see another person, life-size and in three dimensions. You can talk naturally, gesture and make eye contact.
To make this experience possible, we are applying research in computer vision, machine learning, spatial audio and real-time compression. We’ve also developed a breakthrough light field display system that creates a sense of volume and depth that can be experienced without the need for additional glasses or headsets.
Check out this quick tour, even if it’s hard to use regular video to convey the experience of using the tech:
I hope that Dan & co. will be able to provide some peeks behind the scenes, including at how they captured video for testing and demos. (Trust me, it’s all way weirder & more fascinating than you’d think!)
Body Movin’: Adobe Character Animator introduces body tracking (beta)
You’ll scream, you’ll cry, promises designer Dave Werner—and maybe not due just to “my questionable dance moves.”
Live-perform 2D character animation using your body. Powered by Adobe Sensei, Body Tracker automatically detects human body movement using a web cam and applies it to your character in real time to create animation. For example, you can track your arms, torso, and legs automatically. View the full release notes.
Check out the demo below & the site for full details.
A beautiful collage mural celebrates Dallas’s Deep Ellum
I love this piece from artist Tristan Eaton, celebrating Dallas’s historic Deep Ellum neighborhood:
Syncopated AI nightmare fuel
I’ve obviously been talking a ton about the crazy-powerful, sometimes eerie StyleGAN2 technology. Here’s a case of generative artist Mario Klingemann wiring visuals to characteristics of music:
Watch it at 1/4 speed if you really want to freak yourself out.
Beats-to-visuals gives me an excuse to dig up & reshare Michel Gondry’s brilliant old Chemical Brothers video that associated elements like bridges, posts, and train cars with the various instruments at play:
Back to Mario: he’s also been making weirdly bleak image descriptions using CLIP (the same model we’ve explored using to generate faces via text). I congratulated him on making a robot sound like Werner Herzog. 🙃
Say it -> Select it: Runway ML promises semantic video segmentation
I find myself recalling something that Twitter founder Evan Williams wrote about “value moving up the stack“:
As industries evolve, core infrastructure gets built and commoditized, and differentiation moves up the hierarchy of needs from basic functionality to non-basic functionality, to design, and even to fashion.
For example, there was a time when chief buying concerns included how well a watch might tell time and how durable a pair of jeans was.
Now apps like FaceTune deliver what used to be Photoshop-only levels of power to millions of people, and Runway ML promises to let you just type words to select & track objects in video—using just a Web browser. 👀
ML + MIDI = trippy facial fun
“Hijacking Brains: The Why I’m Here Story” 😌
As I wrote many years ago, it was the chance to work with alpha geeks that drew me to Adobe:
When I first encountered the LiveMotion team, I heard that engineer Chris Prosser had built himself a car MP3 player (this was a couple of years before the iPod). Evidently he’d disassembled an old Pentium 90, stuck it in his trunk, connected it to the glovebox with some Ethernet cable, added a little LCD track readout, and written a Java Telnet app for synching the machine with his laptop. Okay, I thought, I don’t want to do that, but I’d like to hijack the brains of someone who could.
Now my new teammate Cameron Smith has spent a weekend wiring MIDI hardware to StyleGAN to control facial synthesis & modification:
A painterly, AI-hallucinated music video
The Avalanches used ArtBreeder (see previous) to make this trippy, ever-morphing music video.
This provides me a periodic reminder that I’ve never seen What Dreams May Come & should someday fix that.
VFX & photography: Fireside chat tonight with Paul Debevec
If you liked yesterday’s news about Total Relighting, or pretty much anything else related to HDR capture over the last 20 years, you might dig this SIGGRAPH LA session, happening tonight at 7pm Pacific:
Paul Debevec is one of the most recognized researchers in the field of CG today. LA ACM SIGGRAPH’s “fireside chat” with Paul and Carolyn Giardina, of the Hollywood Reporter, will allow us a glimpse at the person behind all the innovative scientific work. This event promises to be one of our most popularas Paul always draws a crowd and is constantly in demand to speak at conferences around the world.
“Total Relighting” promises to teleport(rait) you into new vistas
This stuff makes my head spin around—and not just because the demo depicts heads spinning around!
You might remember the portrait relighting features that launched on Google Pixel devices last year, leveraging some earlier research. Now a number of my former Google colleagues have created a new method for figuring out how a portrait is lit, then imposing new light sources in order to help it blend into new environments. Check it out:
“No One Is Coming. It’s Up To Us.”
“Everyone sweeps the floor around here.”
As I’ve noted many times, that core ethos from Adobe’s founders has really stuck with me over the years. In a similar, if superficially darker, vein, I keep meditating on the phrase “No One Is Coming, It’s Up To Us,” which appears in a sticker I put on the back of my car:
It’s reeeeealy easy to sit around and complain that we don’t have enough XYZ support (design cycles, eng bodies, etc.), and it’s all true/fair—but F that ‘cause it doesn’t move the ball. I keep thinking of DMX, with regard to myself & other comfortable folks:
I put in work, and it’s all for the kids (uh)
But these cats done forgot what work is (uh-huh)
Some brief & bracing wisdom:
Happy Monday. Go get some.
A thoughtful conversation about race
I know it’s not a subject that draws folks to this blog, but I wanted to share a really interesting talk I got to attend recently at Google. Broadcaster & former NFL player Emmanuel Acho hosts “Uncomfortable Conversations With A Black Man,” and I was glad that he shared his time and perspective with us. If you stick around to the end, I pop in with a question. The conversation is also available in podcast form.
This episode is with Emmanuel Acho, who discusses his book and YouTube Channel series of the same name: “Uncomfortable Conversations with a Black Man”, which offers conversations about race in an effort to drive open dialogue.
Emmanuel is a Fox Sports analyst and co-host of “Speak for Yourself”. After earning his undergraduate degree in sports management in 2012, Emmanuel was drafted by the Cleveland Browns. He was then traded to the Philadelphia Eagles in 2013, where he spent most of his career. While in the NFL, Emmanuel spent off seasons at the University of Texas to earn his master’s degree in Sports Psychology. Emmanuel left the football field and picked up the microphone to begin his broadcast career. He served as the youngest national football analyst and was named a 2019 Forbes 30 Under 30 Selection. Due to the success of his web series, with over 70 million views across social media platforms, he wrote the book “Uncomfortable Conversations with a Black Man”, and it became an instant New York Times Best Seller.
Interesting, interactive mash-ups powered by AI
Check out how StyleMapGAN (paper, PDF, code) enables combinations of human & animal faces, vehicles, buildings, and more. Unlike simple copy-paste-blend, this technique permits interactive morphing between source & target pixels:
From the authors, a bit about what’s going on here:
Generative adversarial networks (GANs) synthesize realistic images from random latent vectors. Although manipulating the latent vectors controls the synthesized outputs, editing real images with GANs suffers from i) time-consuming optimization for projecting real images to the latent vectors, ii) or inaccurate embedding through an encoder. We propose StyleMapGAN: the intermediate latent space has spatial dimensions, and a spatially variant modulation replaces AdaIN. It makes the embedding through an encoder more accurate than existing optimization-based methods while maintaining the properties of GANs. Experimental results demonstrate that our method significantly outperforms state-of-the-art models in various image manipulation tasks such as local editing and image interpolation. Last but not least, conventional editing methods on GANs are still valid on our StyleMapGAN. Source code is available at https://github.com/naver-ai/StyleMapGAN.
Using VFX to protect the vulnerable
One of the candidates for this year’s visual effects Oscars was the documentary Welcome to Chechnya, where the faces of people who had to remain anonymous were digitally altered to make them unrecognizable without hiding their facial expressions. Here’s a look at how it was done:
[Via Florian Kainz]
VFX: A little homebrew X-Wing fun
Okay, I’m a day late for May the Fourth, but in the spirit of yesterday’s DIY filmmaking fun, here’s a neat use of an Insta360 device to create in-camera visual effects. (Besides the cam, one only needs giant scale models of Big Ben, etc.—as of course we all have. 🙃 But even “flying” around one’s back yard would be fun.)
“Ikorodu Bois,” scrappy filmmakers from Nigeria
Joyful, homespun storytelling FTW:
The Ikorodu Bois from Lagos, Nigeria make movies because they love movies. Here’s to them and all the movie fans out there.
This all good-natured expression makes me want to see more films “sweded,” a la Be Kind, Rewind:
A little fun with Bullet Time
During our epic Illinois-to-California run down Route 66 in March, my son Henry and I had fun capturing all kinds of images, including via my Insta360 One X2 camera. Here are a couple of “bullet time” slow-mo vids I thought were kind of fun. The first comes from the Round Barn in Arcadia, OK…
…and the second from the Wigwam Motel in Holbrook, AZ (see photos):
It’s a bummer that the optical quality here suffers from having the company’s cheap-o lens guards applied. (Without the guards, one errant swipe of the selfie stick can result in permanent scratches to the lens, necessitating shipment back to China for repairs.) They say they’re working on more premium glass ones, for which they’ll likely get yet more of my dough. ¯\_(ツ)_/¯
What a difference four years makes in iPhone cameras
“People tend to overestimate what can be done in one year and to underestimate what can be done in five or ten years,” as the old saying goes. Similarly, it can be hard to notice one’s own kid’s progress until confronted with an example of that kid from a few years back.
My son Henry has recently taken a shine to photography & has been shooting with my iPhone 7 Plus. While passing through Albuquerque a few weeks back, we ended up shooting side by side—him with the 7, and me with an iPhone 12 Pro Max (four years newer). We share a camera roll, and as I scrolled through I was really struck seeing the output of the two devices placed side by side.
I don’t hold up any of these photos (all unedited besides cropping) as art, but it’s fun to compare them & to appreciate just how far mobile photography has advanced in a few short years. See gallery for more.