I had a ball presenting Firefly during this past week’s Adobe Live session. I showed off the new Recolor Vectors feature, and my teammate Samantha showed how to put it to practical use (along with image generation) as part of a moodboarding exercise. I think you’d dig the whole session, if you’ve got time.
The highlight for me was the chance to give an early preview of the 3D-to-image creation module we have in development:
Use prompts to generate new color palettes, then apply them to SVG artwork, reshuffling colors & applying harmony rules as desired. Check it out: pic.twitter.com/1E30EZiQik
To start, we’re exploring a range of concepts, including:
Text to color enhancements: Change color schemes, time of day, or even the seasons in already-recorded videos, instantly altering the mood and setting to evoke a specific tone and feel. With a simple prompt like “Make this scene feel warm and inviting,” the time between imagination and final product can all but disappear.
Advanced music and sound effects: Creators can easily generate royalty-free custom sounds and music to reflect a certain feeling or scene for both temporary and final tracks.
Stunning fonts, text effects, graphics, and logos: With a few simple words and in a matter of minutes, creators can generate subtitles, logos and title cards and custom contextual animations.
Powerful script and B-roll capabilities: Creators can dramatically accelerate pre-production, production and post-production workflows using AI analysis of script to text to automatically create storyboards and previsualizations, as well as recommending b-roll clips for rough or final cuts.
Creative assistants and co-pilots: With personalized generative AI-powered “how-tos,” users can master new skills and accelerate processes from initial vision to creation and editing.
Meta Research has introduced Animated Drawings, “A Method for Automatically Animating Children’s Drawings of the Human Figure” (as their forthcoming paper is titled).
Today he joined us for a live stream on Discord (below), sharing details about his explorations so far. He also shared a Google Doc that contains details, including a number of links you can click in order to kick off the creation process. Enjoy, and please let me know what kinds of things you’d like to see us cover in future sessions.
Terry White vanquished a chronic photographic bummer—the blank or boring sky—by asking Firefly to generate a very specific asset (namely, an evening sky at the exact site of the shoot), then using Photoshop’s sky replacement feature to enhance the original. Check it out:
On Thursday I had the chance to talk with folks via a Discord livestream, demoing vector recoloring enhancements (not yet shipping, but getting close), talking about how we evaluate feature requests, showing some early thinking about saving presets, talking about “FM technology” (F’ing Magic), and more. Check it out if you’re interested:
I promise I don’t have this stupid look on my face all the time. 😅
Chris Georgenes has been sharing tons of great Firefly-enabled creations (see recent posts), and he’ll be presenting presented live via Behance at 10:30am Pacific today:
Wouldn’t it be amazing to make and composite things like this right in Photoshop? I can’t speak for that team, of course, but it’s easy to imagine ways that one might put the proverbial chocolate into the peanut butter.
A couple of weeks ago I got the chance to attend Runway’s inaugural AI Film Fest in San Francisco, from which the team has now posted the winners. Numerous entries are well worth a look, and I thought I’d highlight a couple of my favorites here (with perhaps more to come later).
“Checkpoint,” below, offers a concise & stylish intro to the emerging domain of AI-assisted storytelling. I particularly like the new-to-me phrase “cultural ratcheting”:
I also vibed out with the sheer propulsive, explosive energy of “Generation”:
And if you want a deeper dive into What This All Might Mean, check out a recording of the panel discussion that accompanied the debut session in New York:
O.G. animator Chris Georgenes has been making great stuff since the 90’s (anybody else remember Home Movies?), and now he’s embracing Adobe Firefly. He’s using it with both Adobe Animate…
NYC (4/20) (Terry White + Brooke Hopper presenting)
SF (4/26) (Paul Trani + Brooke Hopper presenting)
Here’s info for the London event:
——–
We are finally back in London! Join us for a VERY special creative community night.
Get to know the latest from Adobe creative tools, Adobe Express and Adobe Firefly. Learn why you should have Adobe Express on your list of tools to quickly create standout content for social media and beyond using beautiful templates from Adobe. We’ll show you how to leverage your designed assets from Photoshop in to your workflow.
We’re also presenting Adobe Firefly, a generative AI made for creators. With the beta version of the first Firefly model, you can use everyday language to generate extraordinary new content. Get ready to create unique posters, banners, social posts, and more with a simple text prompt. With Firefly, the plan is to do this and more — like uploading a mood board to generate totally original, customizable content.
Meet creators, artists, writers, and designers. Plus hang out with Chris Do and The Futur team! With sips, snacks, and a spotlight on inspiring projects — you won’t want to miss this.
I love these little buggers from longtime Adobean Lee Brimelow. We really need to make it easy to save and share cool prompt/preset combos like these. Stay tuned!
Quiet Nackblog = hard at work, trying to speed up progress. 😅
Some updates on the Adobe Firefly Beta:
* Super high demand / response * Adding 10's of thousands daily * Everyone who has applied should expect to get in over coming weeks * Community Live stream tomorrow (maybe a sneak or two) * In person eventshttps://t.co/C9oZ0Xzvbp
Check out this recording of evangelist Paul Trani’s 1-hour deep dive into Firefly, including examples of how to refine & extend its output in Photoshop:
I enjoyed hearing my colleagues & outside folks discussing the origin, vision, and road ahead for Adobe Firefly in this livestream…
Eric Snowden is the VP of Design at Adobe and is responsible for the product design teams for the Digital Media business, which include Creative Cloud…. Nishat Akhtar is a designer and creative leader with 15+ years of experience in designing and leading initiatives for global brands… Danielle Morimoto is a Design Manager for Adobe Express, based in San Francisco.
…and this Twitter space, featuring our group’s CTO Ely Greenfield, along with creator Karen X. Cheng (whose work I’ve featured here countless times), illustrator & brush creator Kyle T. Webster, and director of design Samantha Warren. Scrub ahead to about 2:45 to get to the conversation.
Made with genuine diabeetus! All right stop, collaborate and listen:
On one hand, you may be convinced we somehow assembled the original cast of The Matrix alongside the ghost of Wilford Brimley to record one of the greatest rap covers of all time. On the other hand, you may find it more believable that we’ve been experimenting with AI voice trainers and lip flap technology in a way that will eventually open up some new doors for how we make videos. You have to admit, either option kind of rules.
Hey, remember when we launched Adobe Firefly what feels like 63 years ago? 😅 OMG, what a week. I am so tired & busy trying to get folks access (thanks for your patience!), answer questions, and more that I’ve barely had time to catch up on all the great content folks are making. I’ll work on that soon, and in the meantime, here are three quick clips that caught my eye.
First, OG author Deke McClelland shows off type effects:
I really appreciate hearing Karen X. Cheng’s thoughts on the essential topics of consent, compensation, and more. We’ve been engaging in lots of very helpful conversations with creators, and there’s of course much more to sort through. As always, your perspective here is most welcome.
I’m so pleased—and so tired! 😅—to be introducing Adobe Firefly, the new generative imaging foundation that a passionate band of us have been working to bring to the world. Check out the high-level vision…
…as well as the part more directly in my wheelhouse: the interactive preview site & this overview of great stuff that’s waiting in the wings:
I’ll have a lot more to share soon. In the meantime, we’d love to hear what you think of what you see so far!
Starting today our community can test Midjourney V5. It has much higher image quality, more diverse outputs, wider stylistic range, support for seamless textures, wider aspect ratios, better image prompting, wider dynamic range and more. Let’s explore!
I’ve spent the last ~year talking about my brain being “DALL•E-pilled,” where I’ve started seeing just about everything (e.g. a weird truck) as some kind of AI manifestation. But that’s nothing compared to using generative imaging models to literally see your thoughts:
Researchers Yu Takagi and Shinji Nishimoto, from the Graduate School of Frontier Biosciences at Osaka University, recently wrote a paper outlining how it’s possible to reconstruct high res images (PDF) using latent diffusion models, by reading human brain activity gained from functional Magnetic Resonance Imaging (fMRI), “without the need for training or fine-tuning of complex deep generative models” (via Vice).
Check out this integration of sketch-to-image tech—and if you have ideas/requests on how you’d like to see capabilities like these get more deeply integrated into Adobe tools, lay ’em on me!
Also, it’s not in Photoshop, but as it made me think of the Photo Restoration Neural Filter in PS, check out this use of ControlNet to revive an old family photo:
I’m really excited to see what kinds of images, not to mention videos & textured 3D assets, people will now be able to generate via emerging techniques (depth2img, ControlNet, etc.):
In a demo video, Qualcomm shows version 1.5 of Stable Diffusion generating a 512 x 512 pixel image in under 15 seconds. Although Qualcomm doesn’t say what the phone is, it does say it’s powered by its flagship Snapdragon 8 Gen 2 chipset (which launched last November and has an AI-centric Hexagon processor). The company’s engineers also did all sorts of custom optimizations on the software side to get Stable Diffusion running optimally.
This new capability in Stable Diffusion (think image-to-image, but far more powerful) produces some real magic. Check out what I got with some simple line art:
My friend Bilawal Sidhu made a 3D scan of his parents’ home (y’know, as one does), and he recently used the new ControlNet functionality in Stable Diffusion to restyle it on the fly. Check out details in this post & in the vid below:
1992 Pink Floyd laser light show in Dubuque, IA—you are back. 😅
Through this AI DJ project, we have been exploring the future of DJ performance with AI. At first, we tried to make an AI-based music selection system as an AI DJ. In the second iteration, we utilized a few AI models on stage to generate real-time symbolic music (i.e., MIDI). In the performance, a human DJ (Tokui) controlled various parameters of the generative AI models and drum machines. This time, we aim to advance one step further and deploy AI models to generate audio on stage in near real-time. Everything you hear during the performance will be pure AI-generation (no synthesizer, no drum machine).
In this performance, Emergent Rhythm, the human DJ will become an AJ or “AI Jockey” instead of a Disk Jockey, and he is expected to tame and ride the AI-generated audio stream in real-time. The distinctive characteristics of AI-based audio generation and “morphing” will provide a unique and even otherworldly sonic experience for the audience.
Introducing the new DigitalFUTURES course of free AI tutorials.
Several of the top AI designers in the world are coming together to offer the world’s first free, comprehensive course in AI for designers. This course starts off at an introductory level and gets progressively more advanced. 18 Feb, Introductory Session 10.00 am EST, 4.00 pm CET, 11.00 pm China What is AI? What are Midjourney, DALL•E, Stable Diffusion, etc.? What is GPT3? What is ChatGPT? And how are they revolutionizing design?
Check out this craziness (which you can try online) from Google researchers, who write, “We introduce Piano Genie, an intelligent controller that maps 8-button input to a full 88-key piano in real time”:
Paul Trillo used Runway’s new Gen-1 experimental model to create a Cubist Simpsons intro:
“The Simpsons” but make it an experimental cubist stop motion. Felt right given long tradition of reanimating the Simpsons intro. Created with the spellbindingly addictive #Gen1 AI video generator from @runwayml — still early days but step in the future if #animation#ai#aiartpic.twitter.com/XZLgGpBLCw
Check out this new generative stylization model. I’m intrigued by the idea of using simple primitives (think dollhouse furniture) to guide synthesis & stylization (e.g. of the buildings shown briefly here).
Today, Generative AI takes its next big step forward.
Introducing Gen-1: a new AI model that uses language and images to generate new videos out of existing ones.
Last month Paul Trillo shared some wild visualizations he made by walking around Michelangelo’s David, then synthesizing 3D NeRF data. Now he’s upped the ante with captures from the Louvre:
NeRFs of the Louvre made from a handful of short iPhone videos shot during a location scout last month. Each shot reimagined over a month later. The impossibilities are endless. More to come…
I got my professional start at AGENCY.COM, a big dotcom-era startup co-founded by creative whirlwind Kyle Shannon. Kyle has been exploring AI imaging like mad, and recently he’s organized an AI Artists Salon that anyone is welcome to join in person (Denver) or online:
The AI Artists Salon is a collaborative group of creatively-minded people and we welcome anyone curious about the tsunami of inspiring generative technologies already rocking our our world. See Community Links & Resources.
On Tuesday evening I had the chance to present some ideas & progress that has inspired me—nothing confidential about Adobe work, of course, but hopefully illuminating nonetheless. If you’re interested, check it out (and pro tip: if you set playback to 1.5x speed or higher, I sound a lot sharper & funnier!).
Here’s an example made from a quick capture I did of my friend (nothing special, but amazing what one can get simply by walking in a circle while recording video):
As luck (?) would have it, the commercial dropped on the third anniversary of my former teammate Jon Barron & collaborators bringing NeRFs into existence:
Three years ago today, the project that eventually became NeRF started working (positional encoding was the missing piece that got us from "hmm" to "wow"). Here's a snippet of that email thread between Matt Tancik, @_pratul_, @BenMildenhall, and me. Happy birthday NeRF! pic.twitter.com/UtuQpWsOt4
Thank God for the vibrant developer community—esp. Adobe vet Christian Cantrell (who somehow finds time to rev his plugin while serving as VP of product for Stability.ai):
“HEY MAN, you ever drop acid?? No? Well I do, and it looks *just like this*!!” — an excitable Googler when someone wallpapered a big meeting room in giant DeepDream renderings
In a similar vein, have fun tripping balls with AI, courtesy of Remi Molettee:
The company has announced a new mode for their Canvas painting app that turns simple brushstrokes into 360 environment maps for use in 3D apps or Omniverse. Check out this quick preview:
My teammate CJ Gammon has released a handy new Chrome extension that lets you select any image, then use it as the seed for new image generation. Check it out:
In this beautiful work from Paul Trillo & co., AI extends—instead of replaces—human creativity & effort:
Here’s a peek behind the scenes:
This project would have never existed without the use of AI. A variety of tools were used from #dalle2 and #stablediffusion to generate the background assets Automatic1111 #img2img and @runwayml to process the video along with @AdobeAE to create the camera moves and transitions pic.twitter.com/FwqwWto966
1. Take reference photo (you can use any photo – e.g. your real house, it doesn’t have to be dollhouse furniture) 2. Set up Stable Diffusion Depth-to-Image (google “Install Stable Diffusion Depth to Image YouTube”) 3. Upload your photo and then type in your prompts to remix the image
We recommend starting with simple prompts, and then progressively adding extra adjectives to get the desired look and feel. Using this method, @justinlv generated hundreds of options, and then we went through and cherrypicked our favorites for this video
I’m not sure what to say about “The first rap fully written and sung by an AI with the voice of Snoop Dogg,” except that now I really want the ability to drop in collaborations by other well known voices—e.g. Christopher Walken.
Maybe someone can now lip-sync it with the faces of YoDogg & friends:
