Greetings from the galactic core, to which my friend Bilawal has dispatched me by editing the 3D model he made from drone-selfie footage that I recorded last year:
In case you’re having trouble loading it on your device, you can check out a recording of this non-starlit version:
As someone remarked on Twitter, “This looks better than any Marvel movie since the first Iron Man.” 🙃
I’m no 3D artist (had I but world enough and time…), but I sure love their work & anything that makes it faster and easier. Perhaps my most obscure point of pride from my Photoshop years is that we added per-layer timestamps into PSD files, so that Pixar could more efficiently render content by noticing which layers had actually been modified.
Anyway, now that Adobe has made a much bigger bet on 3D tooling, it’s great to see new support for Substance Painter coming to Unreal Engine:
The Substance 3D plugin (BETA) enables the use of Substance materials directly in Unreal Engine 5 and Unreal Engine 4. Whether you are working on games, visualization and or deploying across mobile, desktop, or XR, Substance delivers a unique experience with optimized features for enhanced productivity.
Work faster, be more productive: Substance parameters allow for real-time material changes and texture updates.
Substance 3D for Unreal Engine 5 contains the plugin for Substance Engine.
Access over 1000 high-quality tweakable and export-ready 4K materials with presets on the Substance 3D Asset library. You can explore community-contributed assets in the community assets library.
The Substance Assets platform is a vast library containing high-quality PBR-ready Substance materials and is accessible directly in Unreal through the Substance plugin. These customizable Substance files can easily be adapted to a wide range of projects.
As a fan of extremely dumb, simple gags, I hope that you too will enjoy these six seconds:
More elaborate, also fun:
Exciting news!
Once the deal closes, BRIO XR will be joining an unparalleled community of engineers and product experts at Adobe – visionaries who are pushing the boundaries of what’s possible in 3D and immersive creation. Our BRIO XR team will contribute to Adobe’s Creative Cloud 3D authoring and experience design teams. Simply put, Adobe is the place to be, and in fact, it’s a place I’ve long set my sights on joining.
Speaking of Substance, here’s a fun speed run through the kinds of immersive worlds artists are helping create using it:
Pretty slick!
[Adobe] announced a tool that allows consumers to point their phone at a product image on an ecommerce site—and then see the item rendered three-dimensionally in their living space. Adobe says the true-to-life size precision—and the ability to pull multiple products into the same view—set its AR service apart from others on the market. […]
Chang Xiao, the Adobe research scientist who created the tool, said many of the AR services currently on the market provide only rough estimations of the size of the product. Adobe is able to encode dimensions information in its invisible marker code embedded in the photos, which its computer vision algorithms can translate into more precisely sized projections.
Slot cars & CGI FTW:
And here’s a peek behind the scenes:
Last year I enjoyed creating a 3D dronie during my desert trip with Russell Brown, flying around the Pinnacles outside of Trona:
This year I just returned (hours ago!) from another trip with Russell, this time being joined by his son Davis (who coincidentally is my team’s new UI designer!). On Monday we visited the weird & wonderful International Car Forest of the Last Church, where Davis used his drone plus Metashape to create this 3D model:
And yes, technically neither of these locations is in Death Valley, where drone flying is prohibited. Close enough! ¯\_(ツ)_/¯
Neat work from my old teammates: search for “periodic table” to rotate elements in 3D thanks to the <model-viewer> component:
Hmm—I always want to believe in tools like this, but I remain skeptical. Back at Google I played with Blocks, which promised to make 3D creation fun, but which in my experience combined the inherent complexity of that art with the imprecision and arm fatigue of waving controllers in space. But who knows—maybe Shapes is different?
I’m intrigued but not quite sure how to feel about this. Precisely tracking groups of fast-moving human bodies & producing lifelike 3D copies in realtime is obviously a stunning technical coup—but is watching the results something people will prefer to high-def video of the real individuals & all their expressive nuances? I have no idea, but I’d like to know more.
Earlier this week I was amazed to see the 3D scan that Polycam founder Chris Heinrich was able to achieve by flying around LA & capturing ~100 photos of a neighborhood, then generating 3D results via the new Web version of Polycam:
I must try to replicate this myself!
You can take the results for a (literal) spin here, though note that they didn’t load properly on my iPhone.
As you may have seen in Google Earth & elsewhere, scanning & replicating amorphous organic shapes like trees remains really challenging:
It’s therefore all the more amazing to see the incredible results these artists exacting artists are able to deliver when creating free-to-use (!) assets for Unreal Engine:
[Via Michael Klynstra]
Check out this cool educational collaboration:
Discover the experience for yourself with these QR Codes by downloading the Aero app. We recommend running the experience for iOS on 8S and above, or on Android, Private Beta, US only, a list of Android can be found here on HelpX. (FYI, the experience may take a few seconds to load as it is a more sophisticated AR project.)
Here’s a fun, <60s year-end gift from 3D artist Tomas Kral, made with the help of Adobe Substance 3D:
SYNTHESIZE ALL THE CATS!!
This new witchcraft “synthesizes not only high-resolution, multi-view-consistent images in real time, but also produces high-quality 3D geometry.” Plus it makes a literally dizzying array of gatos!
Unsupervised generation of high-quality multi-view-consistent images and 3D shapes using only collections of single-view 2D photographs has been a long-standing challenge. Existing 3D GANs are either compute-intensive or make approximations that are not 3D-consistent; the former limits quality and resolution of the generated images and the latter adversely affects multi-view consistency and shape quality. In this work, we improve the computational efficiency and image quality of 3D GANs without overly relying on these approximations. For this purpose, we introduce an expressive hybrid explicit-implicit network architecture that, together with other design choices, synthesizes not only high-resolution multi-view-consistent images in real time but also produces high-quality 3D geometry. By decoupling feature generation and neural rendering, our framework is able to leverage state-of-the-art 2D CNN generators, such as StyleGAN2, and inherit their efficiency and expressiveness. We demonstrate state-of-the-art 3D-aware synthesis with FFHQ and AFHQ Cats, among other experiments.
Can computers see simple, polygonal worlds & hallucinate realistic scenes, complete with fog, snow, and reflections? So good of you to ask…
And check out this work made with Neural Filters, which you can play with in Photoshop right now:
The imagineers (are they still called that?) promise a new way to create photorealistic full-head portrait renders from captured data without the need for artist intervention.
Our method begins with traditional face rendering, where the skin is rendered with the desired appearance, expression, viewpoint, and illumination. These skin renders are then projected into the latent space of a pre-trained neural network that can generate arbitrary photo-real face images (StyleGAN2).
The result is a sequence of realistic face images that match the identity and appearance of the 3D character at the skin level, but is completed naturally with synthesized hair, eyes, inner mouth and surroundings.
This is amazingly well done…
and the behind-the-scenes tour is a delight:
Check out the latest demo of voice, character motion, and conversation all apparently synthesized in real time.
Now I want to see how it could handle powering a vampire doll, complete with accent. 🧛🏻♀️
10 years ago we put a totally gratuitous (but fun!) 3D view of the layers stack into Photoshop Touch. You couldn’t actually edit in that mode, but people loved seeing their 2D layers with 3D parallax.
More recently apps are endeavoring to turn 2D photos into 3D canvases via depth analysis (see recent Adobe research), object segmentation, etc. That is, of course, an extension of what we had in mind when adding 3D to Photoshop back in 2007 (!)—but depth capture & extrapolation weren’t widely available, and it proved too difficult to shoehorn everything into the PS editing model.
Now Mental Canvas promises to enable some truly deep expressivity:
I do wonder how many people could put it to good use. (Drawing well is hard; drawing well in 3D…?) I Want To Believe… It’ll be cool to see where this goes.
I plan to highlight several of the individual technologies & try to add whatever interesting context I can. In the meantime, if you want the whole shebang, have at it!
I hadn’t heard of Disney’s Gallery: The Mandalorian, but evidently it revealed more details about the Luke Skywalker scene. In response, according to Screen Rant,
VFX team Corridor Crew took the time to share their thoughts on the show’s process. From what they determined, Hamill was merely on set to provide some reference points for the creative team and the stand-in actor, Max Lloyd-Jones. The Mandalorian used deepfake technology to pull together Hamill’s likeness, and they combed through countless hours of Star Wars footage to find the best expressions.
I found the 6-minute segment pretty entertaining & enlightening. Check it out:
I keep meaning to pour one out for my nearly-dead homie, Photoshop 3D (post to follow, maybe). We launched it back in 2007 thinking that widespread depth capture was right around the corner. But “Being early is the same as being wrong,” as Marc Andreessen says, and we were off by a decade (before iPhones started putting depth maps into images).
Now, though, the world is evolving further, and researchers are enabling apps to perceive depth even in traditional 2D images—no special capture required. Check out what my colleagues have been doing together with university collaborators:
[Via]
By now you’ve probably seen this big gato bounding around:
I’ve been wondering how it was done (e.g. was it something from Snap, using the landmarker tech that’s enabled things like Game of Thrones dragons to scale the Flatiron Building?). Fortunately the Verge provides some insights:
In short, what’s going on is that an animation of the virtual panther, which was made in Unreal Engine, is being rendered within a live feed of the real world. That means camera operators have to track and follow the animations of the panther in real time as it moves around the stadium, like camera operators would with an actual living animal. To give the panther virtual objects to climb on and interact with, the stadium is also modeled virtually but is invisible.
This tech isn’t baked into an app, meaning you won’t be pointing your phone’s camera in the stadium to get another angle on the panther if you’re attending a game. The animations are intended to air live. In Sunday’s case, the video was broadcast live on the big screens at the stadium.
I look forward to the day when this post is quaint, given how frequently we’re all able to glimpse things like this via AR glasses. I give it 5 years, or maybe closer to 10—but let’s see.
What a great (and efficient!) love letter to the incredible craftsmanship—including tons of practical effects and miniatures—that went into this epic production:
Last year I was delighted to help launch ultra-detailed 3D vehicles & environments, rendered in the cloud, right in Google Search:
Although we didn’t get to do so on my watch, I was looking forward to leveraging Unreal’s amazing Quixel library of photo-scanned 3D environmental assets. Here’s a look at how they’re made:
This fruit of collaborative creation process, all keyed off of a single scene file, is something to be hold, especially when viewed on a phone (where it approximates scrolling through a magical world):
For Dynamic Machines, I challenged 3D artists to guide a chrome ball from point A to point B in the most creative way possible. Nearly 2,000 artists entered, and in this video, the Top 100 renders are featured from an incredible community of 3D artists!
Last summer my former teammates got all kinds of clever in working around Covid restrictions—and the constraints of physics and 3D capture—to digitize top Olympic athletes performing their signature moves. I wish they’d share the behind-the-scenes footage, as it’s legit fascinating. (Also great: seeing Donald Glover, covered in mocap ping pong balls for the making of Pixel Childish Gambino AR content, sneaking up behind my colleague like some weird-ass phantom. 😝)
Anyway, after so much delay and uncertainty, I’m happy to see those efforts now paying off in the form of 3D/AR search results. Check it out:
One of my favorite flexes while working on Google Photos was to say, “Hey, you remember the liquid-metal guy in Terminator 2? You know who wrote that? This guy,” while pointing to my ex-Adobe teammate John Schlag. I’d continue to go down the list—e.g. “You know who won an Oscar for rigging at DreamWorks? This guy [points at Alex Powell].” I did this largely to illustrate how insane it was to have such a murderer’s row of talent working on whatever small-bore project Photos had in mind. (Sorry, it was a very creatively disappointing time.)
Anyway, John S., along with Michael Natkin (who went on to spend a decade+ making After Effects rock), contributed to this great oral history of the making of Terminator 2. It’s loaded with insights & behind-the-scenes media I’d never seen before. Enjoy!
Back in the 90’s I pleaded with Macromedia to enable a “Flash Interchange Format” that would allow me to combine multiple apps in making great animated content. They paid this no attention, and that’s part of why I joined Adobe & started working on things like integrating After Effects with LiveMotion—a code path that helps connect AE with other apps even two+ decades (!) later.
Point is, I’ve always loved aligning tools in ways that help creators combine apps & reach an audience. While at Google I worked with Adobe folks on 3D data exchange, and now I’m happy to see that Adobe is joining the new Open 3D Foundation, meant to “accelerate developer collaboration on 3D engine development for AAA-games and high-fidelity simulations.”
Amazon… is contributing an updated version of the Amazon Lumberyard game engine as the Open 3D Engine (O3DE), under the permissive Apache 2.0 license. The Open 3D Engine enables developers and content creators to build 3D experiences unencumbered by commercial terms
As for Adobe’s role,
“Adobe is proud to champion the Open 3D Foundation as a founding member. Open source technologies are critical to advance sustainability across 3D industries and beyond. We believe collaborative and agnostic toolsets are the key to not only more healthy and innovative ecosystems but also to furthering the democratization of 3D on a global scale.” — Sebastien Deguy, VP of 3D & Immersive at Adobe.
This Adobe Research collaboration with Stanford & Brown Universities aims to make sense of people moving in space, despite having just 2D video as an input:
We introduce HuMoR: a 3D Human Motion Model for Robust Estimation of temporal pose and shape. Though substantial progress has been made in estimating 3D human motion and shape from dynamic observations, recovering plausible pose sequences in the presence of noise and occlusions remains a challenge. For this purpose, we propose an expressive generative model in the form of a conditional variational autoencoder, which learns a distribution of the change in pose at each step of a motion sequence. Furthermore, we introduce a flexible optimization-based approach that leverages HuMoR as a motion prior to robustly estimate plausible pose and shape from ambiguous observations. Through extensive evaluations, we demonstrate that our model generalizes to diverse motions and body shapes after training on a large motion capture dataset, and enables motion reconstruction from multiple input modalities including 3D keypoints and RGB(-D) videos.
From LiDAR scanners in millions of pockets to AIs that can now generate 3D from 2D, the magic’s getting deep:
NVIDIA Research is revving up a new deep learning engine that creates 3D object models from standard 2D images — and can bring iconic cars like the Knight Rider’s AI-powered KITT to life — in NVIDIA Omniverse.
A single photo of a car, for example, could be turned into a 3D model that can drive around a virtual scene, complete with realistic headlights, tail lights and blinkers.
I’m thrilled that a bunch of Google friends (including Dan Goldman, who was instrumental in bringing Content-Aware Fill to Photoshop) have gotten to reveal Project Starline, their effort to deliver breakthrough 3D perception & display to bring people closer together:
Imagine looking through a sort of magic window, and through that window, you see another person, life-size and in three dimensions. You can talk naturally, gesture and make eye contact.
To make this experience possible, we are applying research in computer vision, machine learning, spatial audio and real-time compression. We’ve also developed a breakthrough light field display system that creates a sense of volume and depth that can be experienced without the need for additional glasses or headsets.
Check out this quick tour, even if it’s hard to use regular video to convey the experience of using the tech:
I hope that Dan & co. will be able to provide some peeks behind the scenes, including at how they captured video for testing and demos. (Trust me, it’s all way weirder & more fascinating than you’d think!)
As I’m on a kick sharing recent work from Ira Kemelmacher-Shlizerman & team, here’s another banger:
Given an “in-the-wild” video, we train a deep network with the video frames to produce an animatable human representation.
This can be rendered from any camera view in any body pose, enabling applications such as motion re-targeting and bullet-time rendering without the need for rigged 3D meshes.
I look forward (?) to the not-so-distant day when a 3D-extracted Trevor Lawrence hucks a touchdown to Cleatus the Fox Sports Robot. Grand slam!!
It’s really cool to see the Goog leveraging its immense corpus of not just 2D or 3D, but actually 4D (time-based), data to depict our planetary home.
In the biggest update to Google Earth since 2017, you can now see our planet in an entirely new dimension — time. With Timelapse in Google Earth, 24 million satellite photos from the past 37 years have been compiled into an interactive 4D experience. Now anyone can watch time unfold and witness nearly four decades of planetary change. […]
We’ve also uploaded more than 800 Timelapse videos in both 2D and 3D for public use. You can select any video you want as a ready-to-use MP4 video or sit back and watch the videos on YouTube.
Try the interactive version, and see the video overview below.
See humanity’s impact on the Earth through a global time-lapse video of the planet since 1984.
The Epic team behind the hyper-realistic, Web-hosted MetaHuman Creator—which is now available for early access—rolled out the tongue-in-cheek “MetaPet Creator” for April Fool’s. Artist Jelena Jovanovic offers a peek behind the scenes.
Elsewhere I put my pal Seamus (who’s presently sawing logs on the couch next to me) through NVIDIA’s somewhat wacky GANimal prototype app, attempting to mutate him into various breeds—with semi-Brundlefly results. 👀
I was sorry to see the announcement that Google’s Poly 3D repository is going away, but I’m happy to see the great folks at Sketchfab stepping up to help creators easily migrate their content:
Poly-to-Sketchfab will help members of the Poly community easily transfer their models to Sketchfab before Poly closes its doors this summer. We’re happy to welcome the Poly community to Sketchfab and look forward to exploring their 3D creations.
Our Poly-to-Sketchfab app connects to both your Poly and Sketchfab accounts, presents you with a list of models that can be transferred, and then copies the models that you select from Poly to Sketchfab.
“This progress is absolute insanity,” says the narrator, and I’d readily agree. Watch how this tech auto-inflates a 2D sketch into 3D & applies rigging.
Topi Kauppinen creates a beautifully uncanny effect, turning 2D stills from The Shining into 3D:
Inspired by the awesome work of photogrammetry expert Azad Balabanian, I used my drone at the Trona Pinnacles to capture some video loops as I sat atop one of the structures. My VFX-expert friend & fellow Google PM Bilawal Singh Sidhu used it to whip up this fun, interactive 3D portrait:
The file is big enough that I’ve had some trouble loading it on my iPhone. If that affects you as well, check out this quick screen recording:
The facial fidelity isn’t on par with the crazy little 3D prints of my head I got made 15 (!) years ago—but for for footage coming from an automated flying robot, I’ll take it. 🤘😛
Imagine loading multi-gigabyte 3D models nearly instantaneously into your mobile device, then placing them into your driveway and stepping inside. That’s what we’ve now enabled via Google Search on Android:
Take it for a spin via the models listed below, and please let us know what you think!
Volvo: Volvo XC40, Volvo XC40 Recharge, Volvo XC60, Volvo XC90
Porsche: Porsche 911, Porsche Cayenne, Porsche Macan, Porsche Panamera, Porsche Taycan
Fiat Chrysler: Jeep Wrangler 4xE
Awesome Dad-flex: Telling your tiny, Cars-loving kids that you know Guido the forklift. 😌
Granted, it was a little confusing to explain that I knew the voice of the cartoon forklift & that he was actually a brainy Italian guy who worked at Pixar—but it worked. In any case, now Guido Quaroni—who spent 20 years at Pixar & who was always a fantastic host during Adobe customer visits—has now joined the Big Red A:
“I’ve been a customer of Adobe’s software for a number of years, and I always admired Adobe’s commitment to provide top of the line tools to creatives,” said Quaroni. “When I heard about Adobe’s renewed interest in entering into the 3D market, given how much more pervasive the consumption of 3D content is becoming, I knew it was something I wanted to be a part of. I’m excited to be joining the Adobe team to help accelerate and grow their 3D offerings for creatives worldwide.”
I remain proud to have delivered, at Guido’s urging, perhaps the most arcane feature request ever: he asked for per-layer timestamps in Photoshop so that Pixar’s rendering pipeline could discern which layers had actually been changed by artists, thereby saving a lot of rendering time. We got this done, and somehow it gives me roughly as much pleasure as having delivered a photo editor that’s used by hundreds of millions of people every month. 😌
Anyway, here’s to great things for Guido, Adobe, and 3D creators everywhere!
Psst—wanna see a multi-gig, super detailed 3D model appear in your driveway almost instantaneously?
I’m delighted to say that our work in cloud-rendered streaming 3D is paying off via this year’s virtual CES show. Per the Google Cloud blog:
As part of Fiat Chrysler’s Virtual Showroom CES event, you can experience the new innovative 2021 Jeep Wrangler 4xe by scanning a QR code with your phone. You can then see an Augmented Reality (AR) model of the Wrangler right in front of you—conveniently in your own driveway or in any open space. Check out what the car looks like from any angle, in different colors, and even step inside to see the interior with incredible details.
A bit on how it works:
The Cloud AR tech uses a combination of edge computing and AR technology to offload the computing power needed to display large 3D files, rendered by Unreal Engine, and stream them down to AR-enabled devices using Google’s Scene Viewer. Using powerful rendering servers with gaming-console-grade GPUs, memory, and processors located geographically near the user, we’re able to deliver a powerful but low friction, low latency experience.
This rendering hardware allows us to load models with tens of millions of triangles and textures up to 4k, allowing the content we serve to be orders of magnitude larger than what’s served on mobile devices (i.e., on-device rendered assets).
And to try it out:
Scan the QR code below, or check out the FCA CES website. Depending on your OS, device, and network strength, you will see either a photorealistic, cloud-streamed AR model or an on-device 3D car model, both of which can then be placed in your physical environment.
David Salesin led Adobe Research for the better part of a decade, and now that he’s at Google, he & others have been collaborating with university researchers to enable fast, fun character animation:
…or at least sufficiently indistinguishable from it.
Apple is adding a feature to help blind users detect the presence & distance of people:
Here’s how people detection works in iOS 14.2 beta – the voiceover support is a tiny bit buggy but still super cool https://t.co/vCyX2wYfx3 pic.twitter.com/e8V4zMeC5C
— Matthew Panzarino (@panzer) October 31, 2020
I’m super interested in seeing how it may help regular people generate 3D models, though it remains to be seen what they’ll then do with them:
Recording memories will never be the same.
Captured in 5 seconds with #iPhone12Pro lidar
Processed in 20 seconds with https://t.co/md5CBwvTYnDirect export to @Sketchfab: https://t.co/sZNMJHoJGi
Happy birthday Olivia! pic.twitter.com/SG9JqZBaQe
— alban denoyel (@albn) October 31, 2020
Here’s the capture flow in 3D Scanner App:
Probably needless to say, 3D model creation remains hard AF for most people, and as such it’s a huge chokepoint in the adoption of 3D & AR viewing experiences.
Fortunately we may be on the cusp of some breakthroughs. Apple is about to popularize LIDAR on phones, and with it we’ll see interesting photogrammetry apps like Polycam:
Got the #iPhone12Pro yesterday and started playing with the new lidar 3D scanner. Works great! Here is one of my first captures, made with @PolyCamAI: https://t.co/hDtg911fHO via @Sketchfab pic.twitter.com/cIHbqwgdiF
— alban denoyel (@albn) October 24, 2020
Meanwhile Adobe is working to enable 3D scanning using devices without fancy sensors. Check out Project Scantastic:
They’re also working to improve the digitization of materials—something that could facilitate the (presently slow, expensive) digitization of apparel:
I’m delighted to share that my team’s work to add 3D & AR automotive results to Google Search—streaming in cinematic quality via cloud rendering—has now been announced! Check out the demo starting around 36:30:
Here’s how we put it on via the Google Keyword blog:
Bring the showroom to you with AR
You can easily check out what the car looks like in different colors, zoom in to see intricate details like buttons on the dashboard, view it against beautiful backdrops and even see it in your driveway. We’re experimenting with this feature in the U.S. and working with top auto brands, such as Volvo and Porsche, to bring these experiences to you soon.
Cloud streaming enables us to take file size out of the equation, so we can serve up super detailed visuals from models that are hundreds of megabytes in size:
Right now the feature is in testing in the US, so there’s a chance you can experience it via Android right now (with iOS planned soon). We hope to make it available widely soon, and I can’t wait to hear what you think!
