All posts by jnack

A Brief History of the World (Models)

3D, AI/ML

On Friday I got to meet Dr. Fei-Fei Li, “the godmother of AI,” at the launch party for her new company, World Labs (see her launch blog post). We got to chat a bit about the paradox that as computer models for perceiving & representing the world grow massively more sophisticated, the interfaces for doing common things—e.g. moving a person in a photo—can get radically simpler & more intentional. I’ll have more to say about this soon.

Meanwhile, here’s her fascinating & wide-ranging conversation with Lenny Rachitsky. I’m always a sucker for a good Platonic allegory-of-the-cave reference. 🙂

From the YouTube summary:

(00:00) Introduction to Dr. Fei-Fei Li
(05:31) The evolution of AI
(09:37) The birth of ImageNet
(17:25) The rise of deep learning
(23:53) The future of AI and AGI
(29:51) Introduction to world models
(40:45) The bitter lesson in AI and robotics
(48:02) Introducing Marble, a revolutionary product
(51:00) Applications and use cases of Marble
(01:01:01) The founder’s journey and insights
(01:10:05) Human-centered AI at Stanford
(01:14:24) The role of AI in various professions
(01:18:16) Conclusion and final thoughts

And here’s Gemini’s solid summary of their discussion of world models:

  • The Motivation: While LLMs are inspiring, they lack the spatial intelligence and world understanding that humans use daily. This ability to reason about the physical world—understanding objects, movement, and situational awareness—is essential for tasks like first response or even just tidying a kitchen 32:23.
  • The Concept: A world model is described as the lynchpin connecting visual intelligence, robotics, and other forms of intelligence beyond language 33:32. It is a foundational model that allows an agent (human or robot) to:
    • Create worlds in their mind’s eye through prompting 35:01.
    • Interact with that world by browsing, walking, picking up objects, or changing things 35:12.
    • Reason within the world, such as a robot planning its path 35:31.
  • The Application: World models are considered the key missing piece for building effective embodied AI, especially robots 36:08. Beyond robotics, the technology is expected to unlock major advances in scientific discovery (like deducing 3D structures from 2D data) 37:48, games, and design 37:31.
  • The Product: Dr. Li co-founded World Labs to pursue this mission 34:25. Their first product, Marble, is a generative model that outputs genuinely 3D worlds which users can navigate and explore 49:11. Current use cases include virtual production/VFX, game development, and creating synthetic data for robotic simulation 53:05.

“How ChatGPT is fueling an existential crisis in education”

AI/ML

I thought this was a pretty interesting & thoughtful conversation. It’s interesting to think about ways to evaluate & reward process (hard work through challenges) and not just product (final projects, tests, etc.). AI obviously enables a lot of skipping the former in pursuit of the latter—but (shocker!) people then don’t build knowhow around solving problems, or even remember (much less feel pride in) the artifacts they produce.

The issues go a lot deeper, to the very philosophy of education itself. So we sat down and talked to a lot of teachers — you’ll hear many of their voices throughout this episode — and we kept hearing one cri du coeur again and again: What are we even doing here? What’s the point?

Links, courtesy of the Verge team:

  • A majority of high school students use gen AI for schoolwork | College Board
  • About a quarter of teens have used ChatGPT for schoolwork | Pew Research
  • Your brain on ChatGPT | MIT Media Lab
  • My students think it’s fine to cheat with AI. Maybe they’re on to something. | Vox
  • How children understand and learn from conversational AI | McGill University
  • File not Found | The Verge

Adobe Research debuts incredibly fast video synthesis

AI/ML

Check out MotionStream, “a streaming (real-time, long-duration) video generation system with motion controls, unlocking new possibilities for interactive content generation.” It’s said to run at 29fps on a single H100 GPU (!).

What I’m really wondering, though, it whether/when/how an interactive interface like this can come to Photoshop & other image-editing environments. I’m not yet sure how the dots connect, but could it be paired with something like this model?

Chocolate-coated glass shards

AI/ML

Oh man, this parody of the messaging around AI-justified (?) price increases is 100% pitch perfect. (“It’s the corporate music that sends me into a rage.”)

 

 
 
 
 
 
View this post on Instagram
 
 
 
 
 
 
 
 
 
 
 

 

A post shared by Mark Edwards (@someguymark)

“We Built The Matrix to Train You for What’s Coming”

3D, AI/ML

My friend Bilawal got to sit down with VFX pioneer John Gaeta to discuss “A new language of perception,” Bullet Time, groundbreaking photogrammetry, the coming Big Bang/golden age of storytelling, chasing “a feeling of limitlessness,” and much more.

In this conversation:

— How Matrix VFX techniques became the prototypes for AI filmmaking tools, game engines, and AR/VR systems
— How The Matrix team sourced PhD thesis films from university labs to invent new 3D capture techniques
— Why “universal capture” from Matrix 2 & 3 was the precursor to modern volumetric video and 3D avatars
— The Matrix 4 experiments with Unreal Engine that almost launched a transmedia universe based on The Animatrix
— Why dystopian sci-fi becomes infrastructure (and what that means for AI safety)
— Where John is building next: Escape.art and the future of interactive storytelling

Continue reading →

A cool new Photoshop feature (that’s still kinda dumb)

Typography

I’m pleased to see that as promised back in May, Photoshop has added a “Dynamic Text” toggle that automatically resizes the size of the letters in each line to produce a visually “packed” look:

Results can be really cool, but because the model has no knowledge of the meaning and importance of each word, they can sometimes look pretty dumb. Here’s my canonical example, which visually emphasizes exactly the wrong thing:

I continue to want to see the best of both worlds, with a layout engine taking into account the meaning & thus visual importance of words—like what my team shipped last year:

I’m absolutely confident that this can be done. I mean, just look at the kind of complex layouts I was knocking out in Ideogram a year ago.

The missing ingredient is just the link between image layouts & editability—provided either by bitmap->native conversion (often hard, but doable in some cases), or by in-place editing (e.g. change “Merry Christmas” to “Happy New Year” on a sign, then regenerate the image using the same style & dimensions)—or both.

Bonus points go to the app & model that enable generation with transparency (for easy compositing), or conversion to vectors—or, again, ¿porque no los dos? 🙂

Demo: Flux vs. Nano Banana inside Photoshop

AI/ML

I recently shared a really helpful video from Jesús Ramirez that showed practical uses for each model inside Photoshop (e.g. text editing via Flux). Now here’s a direct comparison from Colin Smith, highlighting these strengths:

  • Flux: Realistic, detailed; doesn’t produce unwanted shifts in regions that should stay unchanged. Tends to maintain more of the original image, such as hair or background elements.
  • Nano Banana: Smooth & pleasing (if sometimes a bit “Disney”); good at following complex prompts. May be better at removing objects.

These specific examples are great, but I continue to wish for more standardized evals that would help produce objective measures across models. I’m investigating the state of the art there. More to share soon, I hope!

Emu 3.5 looks seriously impressive

3D, AI/ML

Improvements to imaging continues its breakneck pace, as engines evolve from “simple” text-to-image (which we considered miraculous just three years ago—and which I still kinda do, TBH) to understanding time & space.

Now Emu (see project page, code) can create entire multi-page/image narratives, turn 2D images into 3D worlds, and more. Check it out:

Nodevember comes early: Runway Workflows

AI/ML, User Interface

“Nodes, nodes, nodes!” — my exasperated then-10yo coming home from learning Unreal at summer camp 🙂

Love ’em or hate ’em, these UI building blocks seem to be everywhere these days—including in Runway’s new Workflows environment:

Alloy promises PMs on-brand prototyping

AI/ML, User Interface

Hmm—consider me intrigued:

Alloy is AI Prototyping built for Product Management:
➤ Capture your product from the browser in one click
➤ Chat to build your feature ideas in minutes
➤ Share a link with teammates and customers
➤ 30+ integrations for PM teams: Linear, Notion, Jira Product Discovery, and more

Check out the brief demo:

Demo: Specific, practical uses of Flux + Nano Banana inside Photoshop

AI/ML

Twitter (yes, always “Twitter”) can be useful, but a ton of the AI-related posts there are often fairly superficial and/or impractical rehashes of eye candy that garners attention & not much else.

By contrast, Photoshop expert Jesús Ramirez has put together a really solid, nutrient-dense tour—complete with all his prompts—that I think you’ll find immediately useful. Dive on in, or jump directly to one of the topics linked below.

I particularly like this demo of using Flux to modify the text in an image:

Continue reading →

Flux hackathon provides perspective

AI/ML, Illustration

The team at BFL is celebrating some of the most interesting, creative uses of the Flux model. Having helped bring the Vanishing Point tool to Photoshop, and always having been interested in building more such tech, this one caught my eye:

Snapseed adds automatic object selection & editing

AI/ML, Photography

Back when I worked in Google Research, my teammates developed fast models divide images & video into segments (people, animals, sky, etc.). I’m delighted that they’ve now brought this tech to Snapseed:

The new Object Brush in Snapseed on iOS, accessible in the “Adjust” tool, now lets you edit objects intuitively. It allows you to simply draw a stroke on the object you want to edit and then adjust how you want it to look, separate from the rest of the image.

Check out the team blog post for lots of technical details on how the model was trained.

The underlying model powers a wide range of image editing and manipulation tasks and serves as a foundational technology for intuitive selective editing. It has also been shipped in the new Chromebook Plus 14 to power AI image editing in the Gallery app. Next, we plan to integrate it across more image and creative editing products at Google.

HBoooookay

Design, Typography

“A few weeks ago,” writes John Gruber, “designer James Barnard made this TikTok video about what seemed to be a few mistakes in HBO’s logo. He got a bunch of crap from commenters arguing that they weren’t mistakes at all. Then he heard from the designer of the original version of the logo, from the 1970s.”

Check out these surprisingly interesting three minutes of logo design history:

@barnardco “Who. Cares? Unfollowed” This is how a *lot* of people responded to my post about the mistake in the HBO logo. For those that didn’t see it, the H and the B of the logo don’t line up at the top of the official vector version from the website. Not only that, but the original designer @Gerard Huerta700 got in touch! Long story short, we’re all good, and Designerrrs™ community members can watch my interview with Gerard Huerta where we talk about this and his illustrious career! #hbo #typography #logodesign #logo #designtok  original sound – James Barnard

How to change your eval ways (baby)

AI/ML

As much as one can be said to enjoy thinking through the details of how to evaluate AI (and it actually can be kinda fun!), I enjoyed this in-depth guide from Hamel Husain & Shreya Shankar.

All year I’ve been focusing pretty intently on how to tease out the details of what makes image creation & editing models “good” (e.g. spelling, human realism, prompt alignment, detail preservation, and more). This talk pops up a level, focusing more on holistic analysis of end-to-end experiences. If you’re doing that kind of work, or even if you just want to better understand the kind of thing that’s super interesting to hiring managers now, I think you’ll find watching this to be time well spent.

Photoshop integrates Flux, Nano Banana

AI/ML

I’m so happy to see Adobe greatly accelerating the pace of 3p API integrations!

Show your work, AI edition

AI/ML

Microsoft VP Aparna Chennapragada, who recruited me to Microsoft after I reported to her at Google, recently wrote a thoughtful piece about building trust through transparency. Specifically around AI agents, we want less of this…

…and more of this:

I agree completely. Having some thoughtful back-and-forth makes me feel better understood & therefore more confident in my assistant’s work.

And feel here is a big deal. As Maya Angelou said, “People won’t remember what you said, or even what you did, but they’ll remember how you made them *feel*. Microsoft AI leader (and previously DeepMind cofounder) Mustafa Suleyman totally gets this.

Conversely, I just saw a founder advertising his product as “visual storytelling on autopilot.” I get the intent, but I find the phrasing oxymoronic: would any worthwhile “story” be generated by autopilot? Yuck.

When apps try to do too much with my sparse input, seeing the results makes me feel like Neven Mrgan did upon receiving AI-generated slop from a friend: “I was repelled, as if digital anthrax had poured out of the app.” I don’t even want to read such content, much less share it, much less be judged on it.

So yeah, apps: ditch autopilot & instead take the time to show interest & ask good questions. “Slow is smooth, and smooth is fast”—and a little thoughtfulness up front will save me time while increasing my pride of ownership.

Google Flow adds Nano Banana

AI/ML

In addition to adding support for vertical video & greater character consistency, the new Veo-powered storytelling tool now includes direct image creation & manipulation via tiny, tiny fruit:

Google introduces “Learn Your Way”

AI/ML

This paper seems really promising. From textbooks it promises to make:

— Mind maps if you think visually
— Audio lessons with simulated teacher conversations
— Interactive timelines
— Quizzes that change based on where you’re struggling

More details:

Vibe Coding at Google: Prototyping the all-new AI Studio

AI/ML, User Interface

Check out these interesting insights from the former head of design at ElevenLabs, who recently joined Google to help build their AI Studio:

In today’s episode Ammaar Reshi shows exactly how he uses AI to prototype ideas for the new Google AI Studio. He shares his Figma files and two example prototypes (including how he vibe-coded his own version of AI Studio in a couple of days). We also go deep into:

— 4 lessons for vibe-coding like a pro
— When to rely on mockups vs. AI prototypes
— Ammaar’s step-by-step process for prompting
— How Ammaar thinks about the fidelity of his prototypes
— a lot more

BFL’s Flux hackathon kicks off

AI/ML

Prizes include $5,000, an NVIDIA RTX 5090 GPU, and $3K in FAL credits. Check out the site for more info.

“Ruining” art with Nano Banana

AI/ML, Illustration

But, y’know, in a fun & cheeky way. 🙂 Check out this little iterative experiment from Ethan Mollick:

As a longtime Bosch enthusiast, I’m partial to this one:

Reminds me of the time in 2023 (i.e. 10,000 AI years ago) that I forced DALL•E to keep making images look more & more “cheugy”:

The Phantom Superbad

AI/ML

I never want to get used to just how transformative the latest crop of AI-powered tools has become! Check out just one of the latest examples:

Nano Banana is coming to Photoshop—officially!

AI/ML, Miscellaneous

“Yes, And”: It’s the golden rule of improv comedy, and it’s the title of the paper I wrote & circulated throughout Adobe as soon as DALL•E dropped 3+ years ago: yes, we should make our own great models, and of course we should integrate the best of what the rest of the world is making! I mean, duh, why wouldn’t we??

This stuff can take time, of course (oh, so much time), but here we are: Adobe has announced that Google’s Nano Banana editing model will be coming to a Photoshop beta build near you in the immediate future.

Side note: it’s funny that in order to really upgrade Photoshop, one of the key minds behind Firefly simply needed to quit the company, move to Google, build Nano Banana, and then license it back to Adobe. Funny ol’ world…

Beautiful new AI mograph explorations

After Effects, AI/ML

Check out this new work from Alex Patrascu. As generative video tools continue to improve in power & precision, what’ll be the role of traditional apps like After Effects? ¯\_(ツ)_/¯

AI Lego Redux

AI/ML

Back when DALL•E 3 launched (not even two years ago, though in AI time it feels like a million), I used it to delight friends by rendering them & their signature vehicles in Lego form.

Now that Google’s Nano Banana model has dropped, I felt like revisiting the challenge, comparing results to the original plus ones from ChatGPT 4o.

As you can see in the results, 4o increases realism relative to DALL•E, but it loses a lot of expressiveness & soul. Nano Banana manages to deliver the best of both worlds.

Nano Banana comes to Photoshop

AI/ML

Rob de Winter is back at it, mixing in Google’s new model alongside Flux Kontext.

Rob notes,

From my experiments so far:
• Gemini shines at easy conversational prompting, character consistency, color accuracy, understanding reference images
• Flux Kontext wins at relighting, blending, and atmosphere consistency

Barber, gimme the “Kling-Nano Banana…”

AI/ML

And yes, I do feel like I’m having a stroke when I type our actual phrases like that. 🙂 But putting that aside, check out the hairstyling magic that can come from pairing Google’s latest image-editing model with an image-to-video system: