Category Archives: 3D

LiDAR is magic

3D

…or at least sufficiently indistinguishable from it.

Apple is adding a feature to help blind users detect the presence & distance of people:

I’m super interested in seeing how it may help regular people generate 3D models, though it remains to be seen what they’ll then do with them:

Here’s the capture flow in 3D Scanner App:

New Adobe tech promises 3D & materials scanning

3D, Adobe MAX, AR/VR

Probably needless to say, 3D model creation remains hard AF for most people, and as such it’s a huge chokepoint in the adoption of 3D & AR viewing experiences.

Fortunately we may be on the cusp of some breakthroughs. Apple is about to popularize LIDAR on phones, and with it we’ll see interesting photogrammetry apps like Polycam:

Meanwhile Adobe is working to enable 3D scanning using devices without fancy sensors. Check out Project Scantastic:

They’re also working to improve the digitization of materials—something that could facilitate the (presently slow, expensive) digitization of apparel:

Come see the AR feature I’ve been working on all year!

3D, AR/VR

I’m delighted to share that my team’s work to add 3D & AR automotive results to Google Search—streaming in cinematic quality via cloud rendering—has now been announced! Check out the demo starting around 36:30:

Here’s how we put it on via the Google Keyword blog:

Bring the showroom to you with AR

You can easily check out what the car looks like in different colors, zoom in to see intricate details like buttons on the dashboard, view it against beautiful backdrops and even see it in your driveway. We’re experimenting with this feature in the U.S. and working with top auto brands, such as Volvo and Porsche, to bring these experiences to you soon.

Cloud streaming enables us to take file size out of the equation, so we can serve up super detailed visuals from models that are hundreds of megabytes in size:

Right now the feature is in testing in the US, so there’s a chance you can experience it via Android right now (with iOS planned soon). We hope to make it available widely soon, and I can’t wait to hear what you think!

Put semi-horrifying ancient creatures in your room through Google AR

3D, AR/VR

“I know it’s not the ‘woke’ thing to say, but I hope the world is enslaved by an ancient soulless sentience.” — my Lovecraft-loving weirdo friend

Heh—they’re not all creepy, to be sure, but come browse fun stuff on Google Arts & Culture, view the models in 3D, and if you’re on your phone, place them in your space via AR.

DesignTaxi writes,

Some of the creatures include the Aegirocassis, a sea creature that existed 480 million years ago; a creepy-looking ancient crustacean; and a digital remodel of the whale skeleton, which is currently in view in the National History Museum’s Hintze Hall.

Exceedingly tangentially: who doesn’t love a good coelacanth reference?

Google AR search gets super buggy!

3D, AR/VR

…In the best possible way, of course.

My mom loves to remind me about how she sweltered, hugely pregnant with me, through a muggy Illinois summer while listening to cicadas drone on & on. Now I want to bring a taste of the 70’s back to her via Google’s latest AR content.

You can now search for all these little (and not-so-little) guys via your Android or iPhone and see them in your room:

Here’s a list of new models:

  • Rhinoceros beetle
  • Hercules beetle
  • Atlas beetle
  • Stag beetle
  • Giant stag
  • Miyama stag beetle
  • Shining ball scarab beetle
  • Jewel beetle
  • Ladybug
  • Firefly
  • Rosalia batesi
  • Swallowtail butterfly
  • Morpho butterfly
  • Atlas moth
  • Mantis
  • Grasshopper
  • Dragonfly
  • Hornet
  • Robust cicada
  • Brown cicada
  • Periodical cicada
  • Walker’s cicada
  • Evening cicada.

Fun with Natzke’s virtual Legos

3D, AR/VR

Artist/technologist Erik Natzke has kept me inspired for the better part of 20 years. His work played a key role in sending me down a multi-year rabbit hole trying to get Flash (and later HTML) to be a live layer type within Photoshop and other Adobe apps. The creative possibilities were tremendous, and though I’ll always be sad we couldn’t make it happen, I’m glad we tried & grateful for the inspiration.

Anyway, since going independent following a multi-year stint at Adobe, Erik has been sharing delightful AR explorations—recently featuring virtual Legos interacting with realtime depth maps of a scene. He’s been sharing so much so quickly lately that I can’t keep up and would encourage you to follow his Twitter & Instagram feeds, but meanwhile here are some fun tastes:

Now, how soon until we can create the Fell In Love With A Girl video in realtime? 😌🤘

Google makes 3D ads available to all customers

3D

Now, as throughout my career, I’m trying to remove barriers to creative people making the world more fun & beautiful. Along those lines, I’m pleased that one of the teams with whom I’ve been collaborating has now made Swirl (3D display ads) available to all Display & Video 360 customers globally.

Here’s an example of how they look (apologies for the visual degradation from the GIFfing; I’ll see whether I can embed the interactive original):

People seem to dig ’em:

Nissan saw an engagement rate that was 8X higher than rich media benchmarks for the automotive vertical.

For Adidas, Swirl ads drove a 4x higher engagement rate than rich media benchmarks and had an average viewable time of 11 seconds,The 3D creatives also drove a return on ad spend (ROAS) of ~2.8 for the Colombia market.

For Belvedere The Swirl ads drove 6.5x higher brand favorability and 4.9x higher purchase intent vs. category norms.

To get started creating a Swirl ad, you can upload 3D assets to Google Web Designer and use the new Swirl templates. Brands and agencies can also edit, configure, and publish models using Google’s 3D platform, Poly.

“NeRF” promises amazing 3D capture

3D, AI/ML, AR/VR, NeRFs & Splats, Photography

“This is certainly the coolest thing I’ve ever worked on, and it might be one of the coolest things I’ve ever seen.”

My Google Research colleague Jon Barron routinely makes amazing stuff, so when he gets a little breathless about a project, you know it’s something special. I’ll pass the mic to him to explain their new work around capturing multiple photos, then synthesizing a 3D model:

I’ve been collaborating with Berkeley for the last few months and we seem to have cracked neural rendering. You just train a boring (non-convolutional) neural network with five inputs (xyz position and viewing angle) and four outputs (RGB+alpha), combine it with the fundamentals of volume rendering, and get an absurdly simple algorithm that beats the state of the art in neural rendering / view synthesis by *miles*.

You can change the camera angle, change the lighting, insert objects, extract depth maps — pretty much anything you would do with a CGI model, and the renderings are basically photorealistic. It’s so simple that you can implement the entire algorithm in a few dozen lines of TensorFlow.

Check it out in action:

[YouTube]

4D printing: 3D shapes that transform

3D

Interesting stuff:

Gizmodo writes,

In essence, the ‘fourth dimension’ represents change—4D printing simply refers to printers that create objects that can transform over time and, in some cases, self-assemble. […]
Kinematics is a system for 4D printing that creates complex, foldable forms composed of articulated modules. The system provides a way to turn any three-dimensional shape into a flexible structure using 3D printing.

[Vimeo]

Some rather brilliant imaging tech from Microsoft

3D, Image Science

The company’s Photosynth technology has been public since 2006, and while it’s been cool (placing photos into 3D space), I haven’t seen it gain traction in its original form or as a free panorama maker. That could now change.

The new version stitches photos into smooth fly-throughs. Per TechCrunch:

[U]sers upload a set of photos to Microsoft’s cloud service then the technology begins to looking for points (“features”) in the successive photos that appear to the be same object. It then determines where each photo was taken from, where in 3D space each of these objects were, and how the camera was oriented. Next, it generates the 3D shapes on a per-photo basis. And finally, the technology calculates a smooth path – like a Steadicam – through the locations for each photo, and then slices the images into multi-resolution pyramids for efficiency.

Check this out:

Once you’ve clicked it, try hitting “C” to reveal & interact with the 3D camera path. Here’s an example from photographer David Brashears, who captured Mt. Everest during one of the highest-elevation helicopter flights ever attempted:

So, will we see this become more common? It’s the first presentation I’ve seen that makes me want to don a wearable, lifelogging camera on vacation.

Rendering real-world 3D content in real time

3D, User Interface

MIT is making pure magic:

Check out some below-the-table stills on Colossal.

inFORM is a Dynamic Shape Display that can render 3D content physically, so users can interact with digital information in a tangible way. inFORM can also interact with the physical world around it, for example moving objects on the table’s surface. Remote participants in a video conference can be displayed physically, allowing for a strong sense of presence and the ability to interact physically at a distance. inFORM is a step toward our vision of Radical Atoms. 

[Vimeo]

Seene: a "3D instagram"?

3D, Instagram, Photography

It’s easier seen (heh) than described, so just check it out. The Verge writes,

With less than 30 seconds of setup after installing the app, you can record and manipulate an object in real-time, and in 3D. It’s like iOS 7 parallax gone wild…

Even with poorly done Seenes, the app’s 3D effect is breathtaking since it uses the iPhone’s accelerometer to alter the perspective of the image accordingly when you move your hand. On the web, moving your mouse on an image alters its perspective.

[Via Tomas Krcha]

World War II From Space

3D, After Effects, Illustration

“A stunning 90-minute documentary visualizing key events from World War II from the vantage point of space,” World War II From Space just won an Emmy for Outstanding Graphic Design and Art Direction. Featuring 300 animations and 79 VFX shots, it made heavy use of an Adobe workflow (script writing in Adobe Story, 3D integration with After Effects & Cinema 4D, editing in Premiere Pro). Check out an in-depth interview on how the team made it happen.

WWII 0

WWII 2

I can’t wait for the sequel, World War II In Space. [Vimeo]

Projection mapping + robots

3D

Oh God that’s cool. Just stick with it for a minute:

According to The Creators Project,

The team has pioneered the use of robots that are normally used for manufacturing vehicles, and instead developed software to interface the robots in 3D system Maya.[…] Designed to control camera movement, activate lights, and shift set pieces with ever-repeatable motion, this precision allows the virtual and physical worlds to unite and CG elements to match in real time.

Project creator Gmunk offers more behind-the-scenes perspective on his site. [Vimeo] [Via Nicholas Felton]

An artist's trippy 3D painting technique

3D, Photography

I’ve heard Photoshop layers referred to as “sheets of acetate,” but Juan Miguel Palacios’s work takes that literally:

A recent transplant to Brooklyn’s Bushwick neighborhood, Spanish artist Juan Miguel Palacios is making a splash in the New York art scene with a series of work focusing on the human form. Palacios uses a unique technique which involves layering a series of paintings on top of each other to create stunning three-dimensional pieces. We spent some time with Palacios in his studio to discuss his influences, his background and the complexities of his creative process.

[Vimeo] [Via Bryan O’Neil Hughes]

An augmented-reality coloring book

3D

ColAR (iOS/Android) is “the coloring book of the future.” As TechCrunch explains,

First, you go to colAR’s web site and print out your coloring page of choice (the free app usually comes with one option included and a few others available for in-app purchase, but their full catalog is free until July 28th)… Once you’re done [coloring], pop open the app, and hit the “Play” button to bring up a camera view. Hold your drawing up to the camera, and bam! It takes your work of art and wraps it around an animated 3D model.


If only one could draw a Centrifugal Bumblepuppy in this thing, it would be perfect. (Or my head would explode. Or both.)

Bruce Lee gets unearthed through CGI

3D

For real?

Creator Joseph Kahn writes,

My good friend BBH Creative Executive Johnny Tan and I first talked about doing this concept a year ago. We shot it in Hong Kong, and then we worked with vfx company The Mill in London to create a completely CGI Bruce Lee over nine months. EVERY shot of his head and every detail in there is completely cgi. We got Shannon Lee, Bruce Lee’s daughter, to come aboard and we really picked her brain to make sure that everything was accurate from look to soul. We wanted to be as respectful to the man and legend as we could.

Be like water, my friend, so that someday you may bend when they dig up & puppet you, too. [Vimeo] [Via Fran Roig]

Map Illustrator artwork to 3D via LiveSurface Context

3D, Illustration

You might already know LiveSurface, a stock-image library that featured preset grids optimized to work with Photoshop’s Vanishing Point feature. Now the crew behind it has announced the beta of LiveSurface Context, a unique 2.5D app with a built-in artwork store.
Founder Joshua Distler writes,

The app makes design exploration & visualization (for both designer and client) much faster and more fluid by acting as a kind of next-generation WYSIWYG tool. Designers can work naturally inside Illustrator and visualize their concepts rendered photographically with a click. With it you can:

  • Work inside Illustrator and preview ideas rendered in photographic realism with just a click.
  • Simulate a variety of inks and materials (such as foil, emboss, fluorescent) by simply choosing swatches in Illustrator.
  • Download surfaces by drag and drop; surfaces are automatically re-rendered at hi-res when the download completes.
  • Resize and/or rotate Plus Surfaces with a few clicks.
  • Output very hi-res renderings in the background, without interruption to workflow.

The app drew a nice write-up in Fast Company. Here’s a quick demo of browsing for photographic templates, then applying artwork:

More info is in 9to5Mac’s write-up.

Adobe & Maxon announce an alliance

3D, After Effects

“Today,” writes After Effects PM Steve Forde, “Adobe announced it is entering into a strategic alliance with MAXON, the makers of CINEMA 4D.” He goes on to hint at future integration:

“Do what you know, and be the best at it.” Hand in hand with this idea means that you DON’T do a whole lot of stuff you don’t know. With this relationship announcement you have two companies who focus on being the very best at what they do…

I wish could go into more detail right now – but stay tuned. This area is about to get very exciting.

See also the Maxon announcement which says, “As part of the alliance, both companies are expected to collaborate and engineer a pipeline between Adobe® After Effects® software and CINEMA 4D to give users a seamless 2D/3D foundation.”