Hmm—I’m intrigued by the filmmaking-for-kids possibilities here, but deeply ambivalent about introducing screen time into one of the great (and threatened) pure-imagination oases in my kids’ lives:

LEGO + AR + Apple at WWDC! pic.twitter.com/TXlx0pyTz4

— CNET (@CNET) June 4, 2018

Up to four friends can play in the same set on four different iOS devices, and notably all of the virtual aspects of the LEGO AR app will be connected to physical LEGO sets. “We can save our entire world back into our physical set, and pick up where we left off later,” Sanders said.

Hmm—interesting, if embryonic: 

Use simple sentences to add objects and give them behaviors. Say ‘I need some sheep’ to add sheep into your world. Then give the sheep something to do by saying ‘Sheep eat grass’ or ‘Sheep breed’.

Everything you add becomes part of a working system. By layering multiple objects and behaviors, you can keep increasing the complexity of your creation.

Everything old is new again: Anyone remember The Subservient Chicken? You could ask it to perform more than 300 commands, many of which live on Wikipedia, because the internet is magic. Anyway, driving things via voice for its own sake is generally cool but stupid, but I know someone will do it well.

[Vimeo] [Via Mike Rotondo]

The open-source Lantern project promises to transform any surface into AR using Raspberry Pi, a laser projector, and Android Things:

Rather than insisting that every object in our home and office be ‘smart’, Lantern imagines a future where projections are used to present ambient information, and relevant UI within everyday objects. Point it at a clock to show your appointments, or point to speaker to display the currently playing song. Unlike a screen, when Lantern’s projections are no longer needed, they simply fade away.

[YouTube] [Via]

Want to “photograph an imagined cyberpunk world in-camera without any post-production”? ARwall promises you can through “a visual effects compositing technology that combines new mixed-reality screens along with the oldest (and maybe best) trick in the filmmaking book.”

This piece delves into the inception, development process, and shows the production test footage of the first ever real-time, in-camera, real-light, real-lens, perspective-adapting, mixed-reality, rear-screen, compositing technique.

[Vimeo] [Via Luca Prasso]

Check out this neat technique in action:

Science notes,

The researchers tested the method with a variety of body shapes, clothing, and backgrounds and found that it had an average accuracy within 5 millimeters, they will report in June at the Computer Vision and Pattern Recognition conference in Salt Lake City. The system can also reproduce the folding and wrinkles of fabric, but it struggles with skirts and long hair. With a model of you, the researchers can change your weight, clothing, and pose—and even make you perform a perfect pirouette. No practice necessary.

[YouTube]

Google’s Arts & Culture team partnered with CyArk to travel to over 25 sites across 18 countries, using drone imagery and 3D laser scanners to capture intricate portraits of each place. You can explore the story and 3D model of each historic location—from Syria’s Al-Azem palace, to the Temple of Eshmoun in Lebanon, to the Mayan city of Chichen Itza—on the site.

Another from the “Awesome Past Lives I Never Knew My Colleagues Had” Files: I just learned that Tarik Abdel-Gawad, with whom I’ve been collaborating on AR stuff, programmed & performed the amazing “Box” projection-mapping robot demo with Bot & Dolly before Google acquired that company. It’s now a few years old but no less stunning:

Bot & Dolly produced this work to serve as both an artistic statement and technical demonstration. It is the culmination of multiple technologies, including large scale robotics, projection mapping, and software engineering. We believe this methodology has tremendous potential to radically transform theatrical presentations, and define new genres of expression.

Check out this peek behind the scenes:

[YouTube 1 & 2]

Now, before I tell you who makes this or what it’s called, check it out & tell me it’s not pretty slick:

Now, when you discover that it’s actually Microsoft Paint, do you go full Chris Farley warpath, or maybe just start spontaneously vomiting? Perhaps we need the equivalent of Swiftamine for apps. 🙂 

[YouTube 1 & 2]

It’s a running joke at Google that you can spend your whole career not knowing what the people nearby you are doing, or even who they are. (In fact, I continue to harbor a dream about creating an AR overlay that would solve this, but that’s another story.) As it happens, I just discovered that my teammate Bill has been making some really cool augmented reality experiments, including the open source Flight Paths:

Flight Paths is an experiment that transforms your room into a flight path visualization. Touch any horizontal surface and explore as flights take off from JFK or SFO and fly around your space. Learn more at g.co/arexperiments

Continual note to self: Ask people more about what they’re doing!

[YouTube]

This demo of the Snappers Facial Rig is pretty damn impressive. Now, how soon until front-facing depth cameras (a la those on iPhone X) can be paired with enough on-device rendering power to produce results like this?

[YouTube 1 & 2]

Re-creating performances in famous movies, capturing them with an iPhone X, and then compositing the results into those movies is, of course, 100% my kind of jam. Check out the whole thing (all great, from The Wire to Arrested Development), or just jump to the Full Metal Jacket portion.

[YouTube]

Sure, why not? 🙃⚔️

[YouTube]

Are Irish eyes smiling? I should ask my teammate Avneesh to scan me & find out:

Google Research presents a machine learning based approach to infer select facial action units and expressions entirely by analyzing a small part of the face while the user is engaged in a virtual reality experience. Specifically, images of the user’s eyes captured from an infrared (IR) gaze-tracking camera within a VR headset are sufficient to infer at least a subset of facial expressions without the use of any external cameras or additional sensors.

[YouTube]

Looks cool:

You can take in Discovery TRVLR for yourself using Google’s Daydream headset (or Google Cardboard). The show will be viewable on YouTube, DiscoveryVR.com, and on the Discovery VR app.

In La Paz, meet female wrestlers giving hope to domestic violence victims. Then cap it all off riding alongside a polar explorer through ice caves and the frozen tundra of Antarctica. 

See Engadget for more details.

[YouTube]

I’m eager to show the Micronaxx, who are studying Native American cultures in school:

Dr Jago Cooper, Curator, Head of the Americas at the British Museum, introduces Google Arts & Culture’s new collection on the preservation of the Maya Heritage.

The video showcases pioneering and cutting edge technologies that enable to preserve some unique traces of this Guatemalan civilization, inherited by British explorer Alfred Maudslay.

[YouTube]

My friend Ben Grossman & the team at Magnopus have cooked up what looks to be a fun, social VR experience for Oculus Rift:

[YouTube]

“With Poly,” says Google AR/VR lead Clay Bavor, “our mission is to organize the world’s 3D information and make it universally accessible and useful.”

Poly lets you quickly find 3D objects and scenes for use in your apps, and it was built from the ground up with AR and VR development in mind. It’s fully integrated with Tilt Brush and Blocks, and it also allows direct OBJ file upload, so there’s lots to discover and use.

Check it out:  

You’re frequently allowed to modify the models in case they don’t quite fit your needs, and you can share them as GIFs or explore them in VR viewers.

[YouTube]

Back in the day Steve DiVerdi implemented 3D, physics-based brushes in Photoshop, then joined Google Photos. Now he’s back at Adobe working on VR video tech. Check out his demo of “Sidewinder,” which leverages a Google Jump VR rig to capture numerous images, then synthesize new views to enable more interactive nav (hard to describe, easy to grok when watched):

[YouTube]

Check out five new Google Expeditions created in partnership with MLB.com:

These include virtual tours of Citi Field in New York and Oriole Park at Camden Yards in Baltimore, both of which are narrated by MLB Network’s Heidi Watney. You can also get behind-the-scenes access with career tours that showcase the lives of a baseball beat reporter and television broadcasters. We’re also bringing you a Statcast tour, so you can geek out Moneyball-style with the math and physics behind the game.

Immerse yourself in a 3D model built from Curiosity rover photos:

Today, we’re putting that same 3D model into an experience for everyone to explore. We call it Access Mars, and it lets you see what the scientists see. Get a real look at Curiosity’s landing site and other mission sites like Pahrump Hills and Murray Buttes. Plus, JPL will continuously update the data so you can see where Curiosity has just been in the past few days or weeks. All along the way, JPL scientist Katie Stack Morgan will be your guide, explaining key points about the rover, the mission, and some of the early findings.

[YouTube] [Via]

Curiscope takes introspection to a whole new level—and the puns’ll get under your skin:

Virtuali-Tee is a magic lens into a world inside the body. View through our free app on your phone or tablet to unlock a portal into your body’s vital organs. Jump into the pumping heart of an awesome anatomical adventure that brings learning to life in fully animated 3D using augmented and virtual reality technologies. Take a deep breath, dive into the bloodstream, and see for yourself.

[YouTube]

Holoportation is “a new type of 3D capture technology that allows high quality 3D models of people to be reconstructed, compressed, and transmitted anywhere in the world in real-time.”

When combined with mixed reality displays such as HoloLens, this technology allows users to see and interact with remote participants in 3D as if they are actually present in their physical space. Communicating and interacting with remote users becomes as natural as face to face communication.

Total Recall can’t be far behind…

[YouTube]

Creeptastic! But quick, cool, and impressive: Visit The University of Nottingham’s demo site, and check out the project site for more details. As The Verge writes,

“3D face reconstruction is a fundamental computer vision problem of extraordinary difficulty.” You usually need multiple pictures of the same face from different angles in order to map every contour. But, by feeding a bunch of photographs and corresponding 3D models into a neural network, the researchers were able to teach an AI system how to quickly extrapolate the shape of a face from a single photo.

[Via Alex Kauffmann]

Looks awesome:

This update lets you explore Street View imagery from 85 countries right within Earth VR. Just fly down closer to street level, check your controller to see if Street View is available and enter an immersive 360° photo. You’ll find photos from the Street View team and those shared by people all around the world.

Check out this multi-track song explorer from a cool podcast (see previous) & Google’s WebVR team:

What if you could step inside a song? Inside Music is a simple experiment that explores that idea. It features the music of Phoenix, Natalia Lafourcade, Perfume Genius, Alarm Will Sound, Clipping, and Ibeyi. 

if you’re a musician, you can explore your own songs in VR or put them up on the web for others to explore.

[YouTube 1 and 2] [Via]

I’m delighted that Google releasing a preview SDK of ARCore, bringing augmented reality capabilities to existing and future Android phones. Developers can start experimenting with it right now.

The team writes,

It works without any additional hardware, which means it can scale across the Android ecosystem. ARCore will run on millions of devices, starting today with the Pixel and Samsung’s S8, running 7.0 Nougat and above. We’re targeting 100 million devices at the end of the preview.

And:

We’re also working on Visual Positioning Service (VPS), a service which will enable world scale AR experiences well beyond a tabletop. And we think the Web will be a critical component of the future of AR, so we’re also releasing prototype browsers for web developers so they can start experimenting with AR, too. These custom browsers allow developers to create AR-enhanced websites and run them on both Android/ARCore and iOS/ARKit.

Super exciting times!

[YouTube]

The miracles & wonder continue:

Our approach is to instrument the environment leaving the user unburdened of any equipment, creating a seamless walk-up-and-play experience. We demonstrate this technology in a series of vignettes featuring humanoid animals.

Participants can not only see and hear these characters, they can also feel them on the bench through haptic feedback. Many of the characters also interact with users directly, either through speech or touch.

[YouTube]

When Periscope debuted two years ago, I thought it would quickly usher in an era of live multi-camera feeds of clubhouse champagne parties, dugout conversations, and more. Maybe it still will, but for most uses, it became clear (as maybe it should have been at the start) that very little content demands or rewards live viewing. The Snapchat-pioneered stories format, by contrast, feels live but offers just enough curation & control for both creators & viewers that it’s dramatically more compelling for most occasions.

Anyway, the jury remains out on VR, but I’m glad to see more experiments getting underway. Now Google is using the compact Jump camera rig to partner with MLB in telling young players’ stories. Engadget writes,

[T]he NFL partnered with Google Daydream to produce an exclusive series called All or Nothing last year. Major League Baseball has also collaborated with Google Daydream on a video game and the MLB.com At Bat app. Now, they’ve announced their latest partnership: “On the Verge,” which is a VR series that profiles up-and-coming baseball stars.

The first episodes are available now. They’ll also be available on MLB’s official YouTube account soon.

Viewer: “Where the hell should I look?”
Creator: “Where the hell do people look?”

Making compelling 360º content—like both pimpin’ & impin’—ain’t easy. Fortunately YouTube is adding some new analytical tools:

Today we’re introducing heatmaps for 360-degree and VR videos with over 1,000 views, which will give you specific insight into how your viewers are engaging with your content. With heatmaps, you’ll be able to see exactly what parts of your video are catching a viewer’s attention and how long they’re looking at a specific part of the video.

 Meanwhile they’ve started a new VR Creator Lab bootcamp:

Take your VR video creation to the next level. YouTube is taking applications for a 3 month learning and production intensive for VR creators. Participants will receive advanced education from leading VR instructors, 1:1 mentoring, and $30K – $40K in funding toward the production of their dream projects.

The application window has now closed (sorry I didn’t the news ’til now), but hopefully this will go well & future openings will emerge.