Let’s start Monday with some moments of Zen. 😌
All posts by jnack
Behold the “Icelandverse”: Stone-cold trolling from up north
This is delightful (“and not super weird!” 😛).
Best Inventions of 2021: Adobe Super Resolution
Congrats to Eric Chan & the whole crew for making Time’s list:
Most of the photos we take these days look great on the small screen of a phone. But blow them up, and the flaws are unmistakable. So how do you clean up your snaps to make them poster-worthy? Adobe’s new Super Resolution feature, part of its Lightroom and Photoshop software, uses machine learning to boost an image’s resolution up to four times its original pixel count. It works by looking at its database of photos similar to the one it’s upscaling, analyzing millions of pairs of high- and low-resolution photos (including their raw image data) to fill in the missing data. The result? Massive printed smartphone photos worthy of a primo spot on your living-room wall. —Jesse Will
[Via Barry Young}
Beeple visits The Tonight Show
Perhaps a bit shockingly, I’ve somehow been only glancingly familiar with the artist’s career, and I really enjoyed this segment—including a quick visual tour spanning his first daily creation through the 2-minute piece he made before going to the hospital for the birth of his child (!), to the work he sold on Tuesday for nearly $30 million (!!).
ProsePainter enables painting via descriptions
Type the name of something (e.g. “beautiful flowers”), then use a brush to specify where you want it applied. Here, just watch this demo:
The project is open source, complements of the creators of ArtBreeder.
Google “Pet Portraits” find doppelgängers in art
Today we are introducing Pet Portraits, a way for your dog, cat, fish, bird, reptile, horse, or rabbit to discover their very own art doubles among tens of thousands of works from partner institutions around the world. Your animal companion could be matched with ancient Egyptian figurines, vibrant Mexican street art, serene Chinese watercolors, and more. Just open the rainbow camera tab in the free Google Arts & Culture app for Android and iOS to get started and find out if your pet’s look-alikes are as fun as some of our favorite animal companions and their matches.
Check out my man Seamus:
Niantic introduces the Lightship AR dev platform
Hmm—I want to get excited here, but as I’ve previously detailed, I’m finding it tough.
Pokemon Go remains the one-hit wonder of the location-based content/gaming space. That being true 5+ years after its launch, during which time Niantic has launched & killed Harry Potter Wizard Unite; Microsoft has done the same with Minecraft Earth; and Google has (AFAIK) followed suit with their location-based gaming API, I’m not sure that we’ll turn a corner until real AR glasses arrive.
Still & all, here it is:
The Niantic Lightship Augmented Reality Developer Kit, or ARDK, is now available for all AR developers around the world at Lightship.dev. To celebrate the launch, we’re sharing a glimpse of the earliest AR applications and demo experiences from global brand partners and developer studios from across the world.
We’re also announcing the formation of Niantic Ventures to invest in and partner with companies building the future of AR. With an initial $20 million fund, Niantic Ventures will invest in companies building applications that share our vision for the Real-World Metaverse and contribute to the global ecosystem we are building. To learn more about Niantic Ventures, go to Lightship.dev.
It’s cool that “The Multiplayer API is free for apps with fewer than 50,000 monthly active users,” and even above that number, it’s free to everyone for the first six months.
FaceStudio enables feature-by-feature editing via GANs
In traditional graphics work, vectorizing a bitmap image produces a bunch of points & lines that the computer then renders as pixels, producing something that approximates the original. Generally there’s a trade-off between editability (relatively few points, requiring a lot of visual simplification, but easy to see & manipulate) and fidelity (tons of points, high fidelity, but heavy & hard to edit).
Importing images into a generative adversarial network (GAN) works in a similar way: pixels are converted into vectors which are then re-rendered as pixels—and guess what, it’s a generally lossy process where fidelity & editability often conflict. When the importer tries to come up with a reasonable set of vectors that fit the entire face, it’s easy to end up with weird-looking results. Additionally, changing one attribute (e.g. eyebrows) may cause changes to others (e.g. hairline). I saw a case once where making someone look another direction caused them to grow a goatee (!).
My teammates’ FaceStudio effort proposes to address this problem by sidestepping the challenge of fitting the entire face, instead letting you broadly select a region and edit just that. Check it out:
Two fascinating minutes of Lego engineering
Enjoy. 😌
Trippy Lego kinetic sculpture
For about two and a half minutes you’re gonna say, “Dude, this is the most boring content you’ve ever posted; thanks for wasting my time!” And then you’ll see why I posted it. 🙃
[Via]
NVIDIA promises 10x lower bandwidth use by synthesizing head movements
Leaving aside the eye-popping, sometimes disconcerting applications of GANs for facial synthesis & editing, what if the core tech could be used just to generate high-quality results even with poor bandwidth? That’s one possible application of NVIDIA’s recent endeavors:
Birds with human arms, y’all
Sure, face swapping and pose manipulation on humans is cool and all, but our industry’s next challenge must be beak swapping and wing manipulation. 😅🐦
[Via]
George Orwell on your self-delusional product metrics
Okay, not directly, but generally dead-on:
We are all capable of believing things which we know to be untrue, and then, when we are finally proved wrong, impudently twisting the facts so as to show that we were right. Intellectually, it is possible to carry on this process for an indefinite time: the only check on it is that sooner or later a false belief bumps up against solid reality, usually on a battlefield.” – George Orwell, 1946.
“…or at reorg time.” — JNack
“The Face of Art” promises geometry-aware style transfer
By analyzing various artists’ distinctive treatment of facial geometry, researchers in Israel devised a way to render images with both their painterly styles (brush strokes, texture, palette, etc.) and shape. Here’s a great six-minute overview:
Russell Brown shows creating Smart Objects & importing raw in Photoshop for iPad
90 seconds well spent with the sensei:
And here’s how Camera Raw can feed into SO’s:
MAX Sneak: Project In-Between synthesizes animation between similar photos
Turning bursts of what would have been outtakes into compelling little animations: that’s the promise of Project In-Between.
Design: Electrified highways?
My little brother is a trucker, and although I can’t imagine a solution like this working for the rural routes he drives, it’ll be interesting to see how it might work for long-haul highways. Check out the idea (not cheap, but potentially highly impactful):
Mental Canvas enables 3D drawing
10 years ago we put a totally gratuitous (but fun!) 3D view of the layers stack into Photoshop Touch. You couldn’t actually edit in that mode, but people loved seeing their 2D layers with 3D parallax.
More recently apps are endeavoring to turn 2D photos into 3D canvases via depth analysis (see recent Adobe research), object segmentation, etc. That is, of course, an extension of what we had in mind when adding 3D to Photoshop back in 2007 (!)—but depth capture & extrapolation weren’t widely available, and it proved too difficult to shoehorn everything into the PS editing model.
Now Mental Canvas promises to enable some truly deep expressivity:
I do wonder how many people could put it to good use. (Drawing well is hard; drawing well in 3D…?) I Want To Believe… It’ll be cool to see where this goes.
MAX Sneak: Smarter vectorization through “Make It Pop”
Semantic segmentation + tracing FTW!
By using machine learning to understand the scene, Project Make it Pop makes it easy to create and customize an illustration by distinguishing between the background and the foreground as well as recognizing connected shapes and structures.
And you’ve gotta stick around for the whole thing, or just jump to around 2:52 where I literally started saying “WTF…?”
MAX Sneak: Morpheus edits faces in video
What if Photoshop’s breakthrough Smart Portrait, which debuted at MAX last year, could work over time?
One may think this is an easy task as all that is needed is to apply Smart Portrait for every frame in the video. Not only is this tedious, but also visually unappealing due to lack of temporal consistency.
In Project Morpheus, we are building a powerful video face editing technology that can modify someone’s appearance in an automated manner, with smooth and consistent results.
Check it out:
Check out all the sneaks at Adobe MAX
I plan to highlight several of the individual technologies & try to add whatever interesting context I can. In the meantime, if you want the whole shebang, have at it!
Come try Photoshop Web!
I kinda can’t believe it, but the team has gotten the old gal (plus Illustrator) running right in Web browsers!
VP of design Eric Snowden writes,
Extending Illustrator and Photoshop to the web (beta) will help you share creative work from the Illustrator and Photoshop desktop and iPad apps for commenting. Your collaborators can open and view your work in the browser and provide feedback. You’ll also be able to make basic edits without having to download or launch the apps.
Creative Cloud Spaces (beta) are a shared place that brings content and context together, where everyone on your team can access and organize files, libraries, and links in a centralized location.
Creative Cloud Canvas (beta) is a new surface where you and your team can display and visualize creative work to review with collaborators and explore ideas together, all in real-time and in the browser.
From the FAQ:
Adobe extends Photoshop to the web for sharing, reviewing, and light editing of Photoshop cloud documents (.psdc). Collaborators can open and view your work in the browser, provide feedback, and make basic edits without downloading the app.
Photoshop on the web beta features are now available for testing and feedback. For help, please visit the Adobe Photoshop beta community.
So, what do you think?
Adobe MAX starts now!
The keynote begins at 9am Pacific today, and Kenan Thompson hosts sneaks at 9am tomorrow. Come check it out!
The list of speakers (from Tilda Swinton & Bryan Cranston to Aaron Draplin & Casey Neistat) is wild, and there are 400+ sessions planned across myriad disciplines & themes.
Inside the 50-megapixel Pixel 6 camera
“Folded optics” & computational zoom FTW! The ability to apply segmentation and selective blur (e.g. to the background behind a moving cyclist) strikes me as especially smart.
On a random personal note, it’s funny to see demo files for features like Magic Eraser and think, “Hey, I know that guy!” much like I did with Content-Aware Fill eleven (!) years ago. And it’s fun that some of the big brains I got to work with at Google have independently come over to collaborate at Adobe. It’s a small, weird world.
Photography: A closer look at the Ronin 4D
I know I posted about it just the other day, but the design of this system is legit interesting. I thought it was especially cool that one can remove the side grips, attach them to the monitor, and control the whole rig from literally miles away (!).
Jetson ONE 🔥🚁
For anyone who’s ever flown a drone but felt insufficiently self-conscious & at risk, let the good times fly!
The Jetson ONE measures 2,845 mm long, 2,400 mm wide, 1,030 mm high, and weighs 86 kg, and is capable of flying a pilot weighing up to 95 kg. It is also collapsible to 900 mm wide when not in use.
Includes LIDAR & a parachute for a cool $92k.
DJI Ronin 4D looks amazing
Built-in gimbal, 8K rez, LIDAR rangefinder for low-light focusing—let’s go!
It commands a pro price tag, too. Per The Verge:
The 6K version costs $7,199, the 8K version is $11,499, and both come with a decent kit: the gimbal, camera, LIDAR range finder, a monitor and hand grips / top handle, a carrying case, and a battery (the 8K camera also comes with a 1TB SSD). In the realm of production cameras and stabilization systems, that’s actually on the lower end (DJI’s cinema-focused Ronin 2 stabilizer costs over $8,000 without any camera attached, and Sony’s FX9 6K camera costs $11,000 for just the body), but if you were hoping to use the LIDAR focus system to absolutely nail focus in your vlogs, you may want to rethink that.
“Dogs & cats morphing together—mass hysteria!”
It’s that thing where you wake up, see some exciting research, tab over to Slack to share it with your team—and then notice that the work is from your teammates. 😝
Check out StyleAlign from my teammate Eli Shechtman & collaborators. Among other things, they’ve discovered interesting, useful correspondences in ML models for very different kinds of objects:
We find that the child model’s latent spaces are semantically aligned with those of the parent, inheriting incredibly rich semantics, even for distant data domains such as human faces and churches. Second, equipped with this better understanding, we leverage aligned models to solve a diverse set of tasks. In addition to image translation, we demonstrate fully automatic cross-domain image morphing
Here’s a little taste of what it enables:
And to save you the trouble of looking up the afore-referenced Ghostbusters line, here ya go. 👻
StyleNeRF promises greater control of 3D facial editing
The visualizations for StyleNeRF tech are more than a little trippy, but the fundamental idea—that generative adversarial networks (GANs) can enable 3D control over 2D faces and other objects—is exciting. Here’s an oddly soundtracked peek:
And here’s a look at the realtime editing experience:
Google enables Pixel -> Snap in two taps
I was so excited to build an AR stack for Google Lens, aiming to bring realtime magic to billions of phones’ default camera. Sadly, after AR Playground went out the door three years ago & the world shrugged, Google lost interest.
At least they’re letting others like Snap grab the mic.
Dubbed “Quick Tap to Snap,” the new feature will enable users to tap the back of the device twice to open the Snapchat camera directly from the lock screen. Users will have to authenticate before sending photos or videos to a friend or their personal Stories page.
Snapchat’s Pixel service will also include extra augmented-reality lenses and integrate some Google features, like live translation in the chat feature, according to the company.
I wish Apple would offer similar access to third-party camera apps like Halide Camera, etc. Its absence has entirely killed my use of those apps, no matter how nice they may be.
Plus Code addresses make the world more navigable
Finding my grandmother’s home in Ireland was one of the weirder adventures I’ve experienced. Directions were literally “Go to the post office and ask for directions.” This worked in 1984, but we visited again in 2007, the P.O. was defunct, so we literally had to ask some random neighbor on the road—who of course knew the way!
Much of the world similarly operates without the kind of street names & addresses most of us take for granted, and Google and others are working to enable Plus Code addresses to help people get around. Check out how it works:
Google writes,
Previously, creating addresses for an entire town or village could take years. Address Maker shortens this time to as little as a few weeks — helping under-addressed communities get on the map quickly, while also reducing costs. Address Maker allows organizations to easily assign addresses and add missing roads, all while making sure they work seamlessly in Google Maps and Maps APIs. Governments and NGOs in The Gambia, Kenya, India, South Africa and the U.S. are already using Address Maker, with more partners on the way. If you’re part of a local government or NGO and think Address Maker could help your community, reach out to us here g.co/maps/addressmaker.
Aquatic Maneuvers In The Dark
Photo rig: The MotoCrane Radical
Many, many years ago, en route home from Legoland, we spied a crazy-looking photography rig atop a car on the freeway, so naturally the boys had to recreate it in Lego when we got home:
Design details of the Blackbird
I know it’s a little OT for this blog, but as I’m always fascinated with clever little design solutions, I really enjoyed this detailed look at the iconic SR-71 Blackbird. I had no idea about things like it having a little periscope, or that its turn radius is so great that pivoting 180º at speed would necessitate covering the distance between Dayton, Ohio & and Chicago (!). Enjoy:
AI: Cats n’ Cages
Things the internet loves:
Nicolas Cage
Cats
Mashups
Let’s do this:
Elsewhere, I told my son that I finally agree with his strong view that the live-action Lion King (which I haven’t seen) does look pretty effed up. 🙃
Demo: Camera Raw is coming to Photoshop for iPad
Nine years ago, Google spent a tremendous amount of money buying Nik Software, in part to get a mobile raw converter—which, as they were repeatedly told, didn’t actually exist. (“Still, a man hears what he wants to hear and disregards the rest…”)
If all that hadn’t happened, I likely never would have gone there, and had the acquisition not been so ill-advised & ill-fitting, I probably wouldn’t have come back to Adobe. Ah, life’s rich pageant… ¯\_(ツ)_/¯
Anyway, back in 2021, take ‘er away, Ryan Dumlao:
“How To Animate Your Head” in Character Animator
Let’s say you dig AR but want to, y’know, actually create instead of just painting by numbers (just yielding whatever some filter maker deigns to provide). In that case, my friend, you’ll want to check out this guidance from animator/designer/musician/Renaissance man Dave Werner.
0:00 Intro
1:27 Character Animator Setup
7:38 After Effects Motion Tracking
14:14 After Effects Color Matching
17:35 Outro (w/ surprise cameo)
Online seminar tomorrow: Russell Brown discusses his latest photographic explorations
I had a ball schlepping all around Death Valley & freezing my butt off while working with Russell back in January, and this seminar sounds fun:
Oct 12, 2021; 7:00 – 8:30pm Eastern
Russell Preston Brown is the senior creative director at Adobe, as well as an Emmy Award-winning instructor. His ability to bring together the world of design and software development is a perfect match for Adobe products. In Russell’s 32 years of creative experience at Adobe, he has contributed to the evolution of Adobe Photoshop with feature enhancements, advanced scripts and development. He has helped the world’s leading photographers, publishers, art directors and artists to master the software tools that have made Adobe’s applications the standard by which all others are measured.
Behind the scenes on “Empire Strikes Back”
Tauntauns & wampas & Sno-Cats, oh my!
I’d never seen any of this footage & I really enjoyed it:
Strike a pose with Adobe AI
My colleagues Jingwan, Jimei, Zhixin, and Eli have devised new tech for re-posing bodies & applying virtual clothing:
Our work enables applications of posed-guided synthesis and virtual try-on. Thanks to spatial modulation, our result preserves the texture details of the source image better than prior work.
Check out some results (below), see the details of how it works, and stay tuned for more.
Enjoy a little realtime nightmare fuel
Hard to believe that it’s been almost seven years since my team shipped Halloweenify face painting at Google, and hard to believe how far things have come since then. For this Halloween you can use GANs to apply & animate all kinds of fun style transfers, like this:
I dunno, but it’s got me feeling kinda Zucked up…
Snapchat embraces deepfakes
They’re using using deepfakes for scripted micro-storytelling:
The new 10-episode Snap original series “The Me and You Show” taps into Snapchat’s Cameos — a feature that uses a kind of deepfake technology to insert someone’s face into a scene. Using Cameos, the show makes you the lead actor in comedy skits alongside one of your best friends by uploading a couple of selfies. […]
The Cameos feature is based on tech developed by AI Factory, a startup developing image and video recognition, analysis and processing technology that Snap acquired in 2019. […]
According to Snap, more than 44 million Snapchat users engage with Cameos on a weekly and more than 16 million share Cameos with their friends.
I dunno—to my eye the results look like a less charming version of the old JibJab templates that were hot 20 years ago, but I’m 30 years older than the Snapchat core demographic, so what do I know?
Hilariously bad logo redesigns
New facial-puppeteering tech from the team behind Deep Nostalgia
The creative alchemists at D-ID have introduced “Speaking Portrait.” Per PetaPixel,
These can be made with any still photo and will animate the head while other parts stay static and can’t have replaced backgrounds. Still, the result below shows how movements and facial expressions performed by the real person are seamlessly added to a still photograph. The human can act as a sort of puppeteer of the still photo image.
What do you think?
AI-enhanced masking is coming to Lightroom, Camera Raw
I used to relish just how much Lightroom kicked Aperture’s butt when it came to making selective adjustments (for which, if I remember correctly, Aperture needed to rely on generating a “destructive” bitmap rendition to send to an outside editor). There’s no point in my mentioning this, I just like to live in the glorious past. 😉
But here’s some glorious future: Lightroom (both new & classic) plus Camera Raw are getting all kinds of AI-enhanced smart masking in the near future. Check out the team’s post for details, or just watch these 90 seconds:
[Via]
Come help me design The Future!
I’m incredibly excited to say that my team has just opened a really rare role to design AI-first experiences. From the job listing:
Together, we are working to inspire and empower the next generation of creatives. You will play an integral part, designing and prototyping exciting new product experiences that take full advantage of the latest AI technology from Adobe research. We’ll work iteratively to design, prototype, and test novel creative experiences, develop a deep understanding of user needs and craft new AI-first creative tools that empower users in entirely new and unimagined ways.
Your challenge is to help us pioneer AI-first creation experiences by creating novel experiences that are intuitive, empowering and first of kind.
By necessity that’s a little vague, but trust me, this stuff is wild (check out some of what I’ve been posting in the AI/ML category here), and I need a badass fellow explorer. I really want a partner who’s excited to have a full seat at the table alongside product & eng (i.e. you’re in the opposite of a service relationship where we just chuck things over the wall and say “make this pretty!”), and who’s excited to rapidly visualize a lot of ideas that we’ll test together.
We are at a fascinating inflection point, where computers learn to see more like people & can thus deliver new expressive superpowers. There will be many dead ends & many challenging ethical questions that need your careful consideration—but as Larry Page might say, it’s all “uncomfortably exciting.” 🔥
If you might be the partner we need, please get in touch via the form above, and feel free to share this opportunity with anyone who might be a great fit. Thanks!
Colbert imagines NFL player intros as shot by famous movie directors
Heh—enjoy.
“Float,” a beautiful short film
It’s odd to say “no spoilers” about a story that unfolds in less than three minutes, but I don’t want to say anything that would interfere with your experience. Just do yourself a favor and watch.
The fact of this all having been shot entirely on iPhone is perhaps the least interesting part about it, but that’s not to say it’s unremarkable: seeing images of my own young kids pop up, shot on iPhones 10+ years ago, the difference is staggering—and yet taken wholly for granted. Heck, even the difference made in four years is night & day.
A ballsy title sequence for The Morning Show
Literally! I love this kind of minimal yet visually rich work.
Google taps (heh) Project Jacquard to improve accessibility
It’s always cool to see people using tech to help make the world more accessible to everyone:
This research inspired us to use Jacquard technology to create a soft, interactive patch or sleeve that allows people to access digital, health and security services with simple gestures. This woven technology can be worn or positioned on a variety of surfaces and locations, adjusting to the needs of each individual.
We teamed up with Garrison Redd, a Para powerlifter and advocate in the disability community, to test this new idea.