I meant to share this one last month, but there’s just no keeping up with the pace of progress!
My initial results are on the uncanny side, but more skillful practitioners like Paul Trillo have been putting the tech to impressive use:
Happy Thanksgiving! Pass the tasty inpainting.
Among the many, many things for which I can give thanks this year, I want to express my still-gobsmacked appreciation of the academic & developer communities that have brought us this year’s revolution in generative imaging. One of those developers is our friend & Adobe veteran Christian Cantrell, and he continues to integrate new tech from his new company (Stability AI) into Photoshop at a breakneck pace. Here’s the latest:
Here he provides a quick comparison between results from the previous Stable Diffusion inpainting model (top) & the latest one:
In any event, wherever you are & however you celebrate (or don’t), I hope you’re well. Thanks for reading, and I wish all the best for the coming year!
Dalí meets DALL•E! 👨🏻🎨🤖
Among the great pleasures of this year’s revolutions in AI imaging has been the chance to discover & connect with myriad amazing artists & technologists. I’ve admired the work of Nathan Shipley, so I was delighted to connect him with my self-described “grand-mentee” Joanne Jang, PM for DALL•E. Nathan & his team collaborated with the Dalí Museum & OpenAI to launch Dream Tapestry, a collaborative realtime art-making experience.
The Dream Tapestry allows visitors to create original, realistic Dream Paintings from a text description. Then, it stitches a visitor’s Dream Painting together with five other visitors’ paintings, filling in the spaces between them to generate one collective Dream Tapestry. The result is an ever-growing series of entirely original Dream Tapestries, exhibited on the walls of the museum.
Check it out:
My Heritage introduces “AI Time Machine”
Another day, another special-purpose variant of AI image generation.
A couple of years ago, MyHeritage struck a chord with the world via Deep Nostalgia, an online app that could animate the faces of one’s long-lost ancestors. In reality it could animate just about any face in a photo, but I give them tons of credit for framing the tech in a really emotionally resonant way. It offered not a random capability, but rather a magical window into one’s roots.
Now the company is licensing tech from Astria, which itself builds on Stable Diffusion & Google Research’s DreamBooth paper. Check it out:
Interestingly (perhaps only to me), it’s been hard for MyHeritage to sustain the kind of buzz generated by Deep Nostalgia. They later introduced the much more ambitious DeepStory, which lets you literally put words in your ancestors’ mouths. That seems not to have bent the overall needle in awareness, at least in the way that the earlier offering did. Let’s see how portrait generation fares.
Charming one’s mom with AI 🐶
Speaking of Bilawal, and in the vein of the PetPortrait.ai service I mentioned last week, here’s a fun little video in which he’s trained an AI model to create images of his mom’s dog. “Oreo lookin’ FESTIVE in that sweater, yo!” 🥰 I can only imagine that this kind of thing will become mainstream quickly.
Neural JNack has entered the chat… 🤖
Last year my friend Bilawal Singh Sidhu, a PM driving 3D experiences for Google Maps/Earth, created an amazing 3D render (also available in galactic core form) of me sitting atop the Trona Pinnacles. At that time he used “traditional” photogrammetry techniques (kind of a funny thing to say about an emerging field that remains new to the world), and this year he tried processing the same footage (comprised of a couple simple orbits from my drone) using new Neural Radiance Field (“NeRF”) tech:
For comparison, here’s the 3D model generated via the photogrammetry approach:
The file is big enough that I’ve had some trouble loading it on my iPhone. If that affects you as well, check out this quick screen recording:
Feedback, please: AI-powered ideation & collaboration?
A new (to me, at least) group called Kive has just introduced AI Canvas.
Here’s a quick demo:
To my eye it’s similar to Prompt.ist, introduced a couple of weeks ago by Facet:
I’m curious: Have you checked out these tools, and do you intend to put them to use in your creative processes? I have some thoughts that I can share soon, but in the meantime it’d be great to hear yours.
An illustrated wedding album made with DreamBooth
I’m not sure whom to credit with this impressive work (found here), nor how exactly they made it, but—like the bespoke pet portraits site I shared yesterday—I expect to see an explosion in such purpose-oriented applications of AI imaging:
PetPortrait.ai promises bespoke images of animals
We’re at just the start of what I expect to be an explosion of hyper-specific offerings powered by AI.
For $24, PetPortrait.ai offers “40 high resolution, beautiful, one-of-a-kind portraits of your pets in a variety of styles.” They say it takes 4-6 hours and requires the following input:
- ~10 portrait photos of their face
- ~5 photos from different angles of their head and chest
- ~5 full-body photos
It’ll be interesting to see what kind of traction this gets. The service Turn Me Royal offers more human-made offerings in a similar vein, and we delighted our son by commissioning this doge-as-Venetian-doge portrait (via an artist on Etsy) a couple of years ago:
Podcast: “Why Figma is selling to Adobe for $20 billion, with CEO Dylan Field”
I had the chance to grab breakfast with Figma founder & CEO Dylan Field a couple of weeks ago, and I found him to be incredibly modest and down to earth. He reminded me of certain fellow Brown CS majors—the brilliant & gracious founding team of Adobe After Effects. I can’t wait for them all to meet someday soon.
In any case, I really enjoyed the hour-long interview Dylan did with Nilay Patel of The Verge. Here’s hoping that the Adobe deal goes through as planned & that we get to do great things together!
AI photography: A deeper dive into “Infinite Nature”
A few weeks ago I shared info on Google’s “Infinite Nature” tech for generating eye-popping fly-throughs from still images. Now that team has shared various interesting tech details on how it all works. And if reading all that isn’t your bag, hey, at least enjoy some beautiful results:
Midjourney can produce stunning type
At Adobe MAX a couple of weeks ago, the company offered a sneak peek of editable type in Adobe Express being rendered via a generative model:
That sort of approach could pair amazingly with this sort of Midjourney output:
I’m not working on such efforts & am not making an explicit link between the two—but broadly speaking, I find the intersection of such primitives/techniques to be really promising.
Try Adobe’s new “Animate from audio” tool
Check out explanation below, then start creating right here.
Use DALL•E + customized Stable Diffusion models together in Photoshop
Christian Cantrell’s back & killing it as usual with the new version of his free Photoshop plugin:
He notes, “Custom, fine-tuned models are absolutely game-changing, and in the future will almost certainly represent the majority of diffusion-based creativity.” 👀 Seems like a non-trivial statement coming from the new VP of product at Stability.ai.
Adobe 3D Design is looking for 2023 interns
The 3D and Immersive Design Team at Adobe is looking for a design intern who will help envision and build the future of Adobe’s 3D and MR creative tools.
With the Adobe Substance 3D Collection and Adobe Aero, we’re making big moves in 3D, but it is still early days! This is a huge opportunity space to shape the future of 3D and AR at Adobe. We believe that tools shape our world, and by building the tools that power 3D creativity we can have an outsized impact on our world.
Type: “United States of Letterpress”
As I may have mentioned, my 13yo son Henry—lover of all things vintage & (often) arcane—has recently taken a real interest in typography. We both enjoyed this short love letter to the craft of typesetting & printing:
DALL•E arrives in Photoshop via the Flying Dog panel
I haven’t yet gotten to try this integration, but I’m excited to see it arrive.
Adobe Mixamo + 3D scanning = AR magic
Check out this cool little workflow from Sergei Galkin:
It uses Mixamo specifically for auto-rigging:
Runway “Infinite Canvas” enables outpainting
I’ve tried it & it’s pretty slick. These guys are cooking with gas! (Also, how utterly insane would this have been to see even six months ago?! What a year, what a world.)
“Mundane Halloween” win: “Person whose skeleton is being estimated by machine learning” 
Happy day to all who celebrate. 😌
The whole thread is hilarious & well worth a look:
A fistful of generative imaging news
Man, I can’t keep up with this stuff—and that’s a great problem to have. Here are some interesting finds from just the last few days:
- Custom SD models are coming to Photoshop via Christian Cantrell’s plugin panel.
- Christian has trained a model on Rivians & says (ambitiously, but not without some justification) that “This is how all advertising and marketing collateral will be made sooner than most of the world realizes.”
- On a related note, here’s a thread (from an engineer at Shopify) on fine-tuning models to generate images of specific products (showing strengths/limitations).
- I see numerous custom models emerging that enable creation of art in the style of Spider-Man, Pixar, and more.
- Stability has rolled out new fine-tuned decoders. (See thread for what that means.)
- Here’s another multiplayer SD-powered creation experience.
- VR: Sketch Diffusion runs live in Gravity Sketch on Quest Pro headsets.
- Enabling text-based image editing via SD.
- Astria is another site promising custom image generation via uploaded pics.
VFX: Top Gun… with a cat
You can take Maverick out of the Tomcat, but you can’t take the tom cat out of Maverick. 😸
[Via CAPT Chris Peppel, USN]
98% unrelated, but possibly amusing:
“Prey” VFX breakdown
I always enjoy this kind of quick peek behind the scenes:
Adobe “Made In The Shade” sneak is 😎
OMG—interactive 3D shadow casting in 2D photos FTW! 🔥
In this sneak, we re-imagine what image editing would look like if we used Adobe Sensei-powered technologies to understand the 3D space of a scene – the geometry of a road and the car on the road, and the trees surrounding, the lighting coming from the sun and the sky, the interactions between all these objects leading to occlusions and shadows – from a single 2D photograph.
Check out AI backdrop generation, right in the Photoshop beta today
One of the sleeper features that debuted at Adobe MAX is the new Create Background, found under Neural Filters. (Note that you need to be running the current public beta release of Photoshop, available via the Creative Cloud app—y’know, that little “Cc” icon dealio you ignore in your menu bar. 🙃)
As this quick vid demonstrates, the filter can not only generate backgrounds based on text, it links to a Behance gallery containing images and popular prompts. You can use these visuals as inspiration, then use the prompts to produce artwork within the plugin:
Here’s the Behance browser:
Midjourney enables “remixing”
I’m really excited to learn more about this development, which I’ve been eagerly awaiting. More control + more speed will make generative imaging truly, broadly useful. I’d like to understand how it compares to techniques like prompt editing.
Quick Tour: The Flying Dog plugin puts Stable Diffusion in Photoshop
Here’s a nice three-minute overview:
Adobe Character Animator introduces Motion Library
Motion Library allows you to easily add premade animated motions like fighting, dancing, and running to your characters. Choose from a collection of over 350 motions and watch your puppets come to life in new and exciting ways!
New Lightroom features: A 1-minute tour, plus a glimpse of the future
The Lightroom team has rolled out a ton of new functionality, from smarter selections to adaptive presets to performance improvements. You should read up on the whole shebang—but for a top-level look, spend a minute with Ben Warde:
And looking a bit more to the future, here’s a glimpse at how generative imaging (in the style of DALL•E, Stable Diffusion, et al) might come into LR. Feedback & ideas welcome!
Stable Diffusion + Adobe Fonts = 🧙♂️🔥
Check out my teammates’ new explorations, demoed here on Adobe Express:
Can’t wait for generative AI + editable text in Adobe tools! 🤖🔥 pic.twitter.com/2kZi4rYM21
— John Nack (@jnack) October 19, 2022
Per the blog post:
Generative AI incorporated into Adobe Express will help less experienced creators achieve their unique goals. Rather than having to find a pre-made template to start a project with, Express users could generate a template through a prompt, and use Generative AI to add an object to the scene, or create a unique font based on their description. But they still will have full control — they can use all of the Adobe Express tools for editing images, changing colors, and adding fonts to create the flyer, poster, or social media post they imagine.
Turn images into usable Stable Diffusion prompts
LatentSpace.dev promises to turn your images into text prompts that can be used in Stable Diffusion to create new artwork. Watch it work:
It interpreted a pic of my old whip as being, among other things, a “5. 1975 pontiac firebird shooting brake wagon estate.” Not entirely bad! 😌
“Imagic”: Text-based editing of photos
It seems almost too good to be true, but Google Researchers & their university collaborators have unveiled a way to edit images using just text:
In this paper we demonstrate, for the very first time, the ability to apply complex (e.g., non-rigid) text-guided semantic edits to a single real image. For example, we can change the posture and composition of one or multiple objects inside an image, while preserving its original characteristics. Our method can make a standing dog sit down or jump, cause a bird to spread its wings, etc. — each within its single high-resolution natural image provided by the user.
Contrary to previous work, our proposed method requires only a single input image and a target text (the desired edit). It operates on real images, and does not require any additional inputs (such as image masks or additional views of the object).
I can’t wait to see it in action!
Stable Diffusion meets WebAR
Back at the start of my DALL•E journey, I wished aloud for a diffusion-powered mobile app:
Now, thanks to the openness of Stable Diffusion & WebAR, creators are bringing that vision closer to reality:
I can’t wait to see what’s next!
Blender + Stable Diffusion = 🪄
Easy placement/movement of 3D primitives -> realistic/illustrative rendering has long struck me as extremely promising. Using tech like StyleGAN to render from 3D can produce interesting results, but it’s been difficult to bring the level of quality & consistency up to what Adobe users demand.
Now with Stable Diffusion (and, one hopes, other diffusion models in the future) attached to Blender (and, one hopes, other object manipulation tools), the vision is getting closer to reality:
Wayback machine: When “AI” was “Adobe Illustrator”
Check out a fun historical find from Adobe evangelist Paul Trani:
The video below shipped on VHS with the very first version of Adobe Illustrator. Adobe CEO & Illustrator developer John Warnock demonstrated the new product in a single one-hour take. He was certainly qualified, being one of the four developers whose names were listed on the splash screen!
How lucky it was for the world that a brilliant graphics engineer (John) married a graphic designer (Marva Warnock) who could provide constant input as this groundbreaking app took shape.
If you’re interested in more of the app’s rich history, check out The Adobe Illustrator Story:
Check out NeRF Studio & some eye-popping results
The power & immersiveness of rendering 3D from images is growing at an extraordinary rate. NeRF Studio promises to make creation much more approachable:
The kind of results one can generate from just a series of photos or video frames is truly bonkers:
Here’s a tutorial on how to use it:
You can now use Stable Diffusion local inference with Photoshop
Check out Christian Cantrell’s latest work (still free!):
A new, free AI colorization tool
Check out Palette:
🌿 Eat your heart out, Homer Simpson 🌿
Here’s another beautiful, DALL•E-infused collaboration between VFX whiz Paul Trillo & Shyama Golden:
Zooming around the world through Google Street View
“My whole life has been one long ultraviolent hyperkinetic nightmare,” wrote Mark Leyner in “Et Tu, Babe?” That thought comes to mind when glimpsing this short film by Adam Chitayat, stitched together from thousands of Street View images (see Vimeo page for a list of locations).
I love the idea—indeed, back in 2014 I tried to get Google Photos to stitch together visual segues that could interconnect one’s photos—but the pacing here has my old man brain pulling the e-brake after just some short exposure. YMMV, so here ya go:
[Via]
“Infinite Nature”: Google’s perceptual view generation sends us flying
Easily my favorite thing at Google was getting to work with stone-cold geniuses like Noah Snavely (one of the minds behind Microsoft’s PhotoSynth) and Richard Tucker. Now they & their teammates have produced some jaw-dropping image synthesis tech:
And “hold onto your papers,” as here’s a look into how it all works:
Interior AI: Generative decor
Interior AI enables you to upload an image of your room, then restyle it in various idioms (Modern, Cyberpunk, Art Nouveau, and more).
Amazingly, it was whipped up in very short order:
Impressive stuff, though know that your results—like mine—may vary. 😅
Google Photos redesigns Memories
Nice work from my old crew:
With the update that starts rolling out today, you’ll see more videos — including the best snippets from your longer videos that Photos will automatically select and trim so you can relive the most meaningful moments. Even your still photos will feel more dynamic thanks to a subtle zoom that brings movement to your memories. And to bring it all together, next month we’ll start adding instrumental music to some Memories.
Happily, they’ve finally built a subset of the collage editor I spec’d out eight years ago (🧂🤷🏼).
Also,
Soon, you’ll begin to see full Cinematic Memories that transform multiple still photos into an end-to-end cinematic experience, taking you back to that moment in time. Cinematic Memories will also have music, making your photos feel a little more like a movie.
Snapchat: Even simple AR is effective AR
A quarter billion people engage with AR content every day, the company says.
And interestingly, one need not create a complex lens in order to have it pay off:
“The research found that simple AR can be just as performant as a sophisticated, custom Lens in driving both upper and lower-funnel metrics like brand awareness and purchase intent. Brands with the resources to execute a more sophisticated Lens will see additional benefits in mid-funnel brand metrics, including favorability and consideration.”
Tutorial: Using Stable Diffusion in Photoshop
Photographer Greg Benz has posted a detailed tutorial showing how to use Christian Cantrell’s Stable Diffusion-Photoshop plugin (now available for free via the Adobe Marketplace). Check it out: