Man, not a day goes by without the arrival of some new & mind-blowing magic—not a day!

We introduce Lumiere — a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion — a pivotal challenge in video synthesis. To this end, we introduce a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model. This is in contrast to existing video models which synthesize distant keyframes followed by temporal super-resolution — an approach that inherently makes global temporal consistency difficult to achieve. […]

We demonstrate state-of-the-art text-to-video generation results, and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation.

We’re only just beginning to discover the experiential possibilities around generative creation, so I’m excited to see this rare gig open up:

You will build new and innovative user interactions and interfaces geared towards our customers unique needs, test and refine those interfaces in collaboration with academic research, user researchers, designers, artists and product teams.

Check out the listing for the full details.

• Fun with famous IP:
  • Game of Thrones Mario Kart Edition
  • Midjourney happily replicates the Simpsons
  • Batman, Star Wars, and 300 with text
  • Star Wars action figures
  • Biden in space, Trump with AR, etc.
• Vectors: StarVector: Generating Scalable Vector Graphics Code from Images
• How to train a custom Stable Diffusion model to generate consistent characters via LensGo.ai, which lets you train 3 custom models for free every month.
• Insane 128x zoom-in on AI-generated meat.

It’s a little nerdy even for my blood, but some of my teammates swear by these techniques that enable connecting Photoshop to a hosted instance of Stable Diffusion, enabling one to guide the process via a Photoshop doc and/or custom-trained styles:

Hah—I can dig this finger-rich pin from Pictoplasma.

Cats-per-second (CPS) FTW! 😀

I mean, seriously, what even is all this?? I can’t explain; just please watch.

• 0:00 Intro
• 0:19 Multimodal Dialogue
• 1:32 Multilinguality 2:04
• Game Creation 2:31
• Visual Puzzles 3:17
• Making Connections
• 3:39 Image & Text Generation
• 4:06 Logic & Spatial Reasoning
• 4:55 Translating Visuals
• 5:27 Cultural Understanding

It’s ludicrous to think that these folks formed the company just six months ago, and even more ludicrous to see what the model can already do—from video synthesis, to image animation, to inpainting/outpainting:

Our vision for Pika is to enable everyone to be the director of their own stories and to bring out the creator in each of us. Today, we reached a milestone that brings us closer to our vision. We are thrilled to unveil Pika 1.0, a major product upgrade that includes a new AI model capable of generating and editing videos in diverse styles such as 3D animation, anime, cartoon and cinematic, and a new web experience that makes it easier to use. You can join the waitlist for Pika 1.0 at https://pika.art.

This tech—or something much like it—is going to be a very BFD. Imagine simply describing the change you’d like to see in your image—and then seeing it.

[Generative models] still face limitations when it comes to offering precise control. That’s why we’re introducing Emu Edit, a novel approach that aims to streamline various image manipulation tasks and bring enhanced capabilities and precision to image editing.

Emu Edit is capable of free-form editing through instructions, encompassing tasks such as local and global editing, removing and adding a background, color and geometry transformations, detection and segmentation, and more. […]

Emu Edit precisely follows instructions, ensuring that pixels in the input image unrelated to the instructions remain untouched. For instance, when adding the text “Aloha!” to a baseball cap, the cap itself should remain unchanged.

Read more here & here.

And for some conceptually related (but technically distinct) ideas, see previous: Iterative creation with ChatGPT.

I’m really digging the experience of (optionally) taking a photo, feeding it into ChatGPT, and then riffing my way towards an interesting visual outcome. Here’s a gallery in which you can see some of the journeys I’ve undertaken recently.

• Image->description->image quality is often pretty hit-or-miss. Even so, it’s such a compelling possibility that I keep wanting to try it (e.g. seeing a leaf on the ground, wanting to try turning it into a stingray).
• The system attempts to maintain various image properties (e.g. pose, color, style) while varying others (e.g. turning the attached vehicle from a box truck to a tanker while maintaining its general orientation plus specifics like featuring three Holstein cows).
• Overall text creation is vastly improved vs. previous models, though it can still derail. It’s striking that one can iteratively improve a particular line of text (e.g. “Make sure that the second line says ‘TRAIN’“).