Category Archives: Miscellaneous

AI can remap speakers’ faces to put words in their mouths

There’s just no way this ends badly. No possible way.

Given audio of President Barack Obama, we synthesize a high quality video of him speaking with accurate lip sync, composited into a target video clip. Trained on many hours of his weekly address footage, a recurrent neural network learns the mapping from raw audio features to mouth shapes. Given the mouth shape at each time instant, we synthesize high quality mouth texture, and composite it with proper 3D pose matching to change what he appears to be saying in a target video to match the input audio track.



Netflix introduces “interactive storytelling”

As an ex-child old enough to remember Captain Power and “interactive TV” in the 80’s, I’m intrigued that Netflix is introducing branching, Choose Your Own Adventure-style storytelling:

So do kids really want this kind of more lean-forward experience? No one seems sure:

We’ve done extensive research and talked to lots of kids and parents, collecting qualitative data to better understand if this is something viewers will like. While we’ve gotten positive feedback (for example, parents like the fact their child has the ability to make decisions and take a seat in the director’s chair, if you will), we’re eager to learn how our members will engage with the experience. Which choices or storylines will be the most popular? Will the mean bears or the friendly bears be more popular? Are members more compelled to rewatch and uncover all of the different storylines?

They say that old age is a second childhood, so this dovetails nicely with my (otherwise super proper, Midwestern Catholic) dad now yelling profanity at Trump on MSNBC. 😝

On a similar note, the Choose Your Own Adventures books haven’t been updated in nearly 20 years (!), but they’re being reissued with maps of the hidden structures in each book:

Nick Montfort, a poet and professor at the Massachusetts Institute of Technology who studies interactive fiction, has a habit of asking people what they know about “Choose Your Own Adventure” books. “They often say, ‘You have two choices after every page,’” he says. “That’s not true. Sometimes you have one choice. Sometimes you have more than two. When you show the maps, you can see that these books don’t look exactly the same.”


[YouTube] [Via]

Affinity Photo on iPad: Will anyone care?

Shivering in the former East Germany back in 2011, I tried—and painfully failed—to get Adobe to invest in a fresh, modern, from-the-ground-up imaging pipeline for Photoshop. It would run beautifully on mobile hardware & scale up to desktops. Instead our mobile play (Photoshop Touch) remained built on top of a mid-90’s platform for banner ads (Flash!), which was almost whimsically insane. Soon after I left the team.

Years later, I think (or at least hope) Adobe might be taking this plunge across their imaging tools. The more interesting question is, will anyone care?

On Monday Serif introduced Affinity Photo for iPad, offering desktop-level photo editing for $20.

Developed without compromise, Affinity Photo for iPad is the first full blown, truly professional photo editing tool to make its way onto the Apple tablet. Built from exactly the same back-end as our award-winning desktop version, and fully optimised to harness the full power of the iPad’s hardware and touch capabilities. Affinity Photo for iPad offers an incredibly fast, powerful and immersive experience whether you are at home, in the studio or on the move.

Here’s a 30-second tour:

I’m eager to try it, and I wish these folks well—but I’m skeptical about it finding a large audience. Working on Photoshop Touch, I struggled to discern an audience that wants “real” power/complexity on mobile devices. Building a desktop-style editor was like building a great home office… on the beach. That is, it’d be sticking a serious, productivity-oriented tool into a lean-back consumption context.

I saw that when people wanted to work, they’d simply put away the tablet and take out their “real” computer (which no one had confiscated, and which retained a larger screen, more memory, etc.). In powerful mobile apps like Snapseed & VSCO, the vast majority of use happens on phones, not tablets. Outside of sketching/painting apps (which cater to a skills set that few people really have), I’ve yet to see serious creative tools take root on tablets.

Anyway, we shall see!


Drinks & food worth knowing in San Jose

With the help of a few friends, I’ve gathered some links to places worth checking out during WWDC and beyond. Add comments with good stuff I’ve missed! [Update: Thanks to Brendan McKenna for making a Foursquare list version of this post.]


  • Haberdasher (subterranean speakeasy) [Update: They were closed when I went last night; call ahead if interested]
  • San Pedro Square (food, music, outdoor seating)
  • Mezcal (Oaxacan cooking, killer moles & mezcal)
  • Hapa’s Brewery (about 5 min by car from downtown; drinks, art, & cornholing)
  • Santana Row (yes, it’s effectively a goddamn mall, but there are ~20 restaurants, many of them really good; 15 min by car)


Haberdasher (speakeasy; reservations recommended if you want your own table; very small snacks, no meals)
43 W. San Salvador San José, CA 95113

5 Points – Craft Cocktail Bar (good drinks, small snacks)
169 w Santa Clara St

AFK Gamer Lounge – (sort of a sports bar but with video games on the TVs instead of sports)
163 W Santa Clara St

Paper Plane – (craft bar, decent food and dessert)
2 S. First St. San Jose, CA, 95113

Original Gravity – (craft beer, sausage, and duck-fat fries)
66 S 1st Street

ISO Beers (tons of beers on tap, nice outdoor patio)
75 E Santa Clara St

Cafe Stritch (live jazz, food and drinks)
374 S 1st st

Drinks (non-alcoholic)
Boba Bar (good boba, and pretty good lunch)
310 S 3rd St, San Jose, CA 95112

Social Policy (coffee shop with fancier pastries and breakfasts)
200 S 1st St, San Jose, CA 95113

San Pedro Square (lots of great fancy-pants little restaurants & options for drinks under two big roofs, with lots of outdoor seating & often live music)
87 N San Pedro

Poor House (New Orleans-inspired food like po’ boys, near the Shark Tank aka SAP Center)
91 S Autumn St

Mezcal (Oaxacan cooking, great moles & mezcal)

The Loft (two levels, solid food/chow)
90 S 2nd St

SOFA Market (like a mini San Pedro Square, the best ice cream bars in downtown)
387 S 1st street

The Farmers Union (Nice sit-down restaurant near San Pedro Square)
151 W Santa Clara St

La Victoria Taqueria (worthwhile arterial destruction with a splash of orange sauce)
131 W Santa Clara St, San Jose, CA 95113

Hom Korean Kitchen (good for lunch)
76 E Santa Clara St, San Jose, CA 95113

Original Joe’s (a downtown institution; Italian food)
301 S 1st St, San Jose, CA 95113


[Special thanks to Dev Davis & Phil Fernandez]


Flo rider: Intelligent, automatic video creation?

The dream of the 90’s is alive in Portland… and perhaps in Flo, a new video-creation app that promises to use AI & voice commands to synthesize great movies from unwashed source material:

Just request a story by location, time period, tags or all of the above and Flo will respond to you like your very own video making assistant, creating the video story of your choice—e.g. ‘Make me a video story of my cat’ or ‘Make a video of my weekend trip to the beach’.

I’m off to try it out, but color me skeptical: My Emmy-winning colleague Bill Hensler, who used to head up video engineering at Adobe, said he’d been pitched similar tech since the early 90’s and always said, “Sure, just show me a system that can match a shot of a guy entering a room with another shot of the same thing from a different angle—then we’ll talk.” As far as I know, we’re still waiting.

No word yet on what happens if you invite Flo to kiss your grits. (Also, saying this makes me Very Old.)