Category Archives: VR/AR

Creating a 3D model of you via a simple video

Check out this neat technique in action:

Science notes,

The researchers tested the method with a variety of body shapes, clothing, and backgrounds and found that it had an average accuracy within 5 millimeters, they will report in June at the Computer Vision and Pattern Recognition conference in Salt Lake City. The system can also reproduce the folding and wrinkles of fabric, but it struggles with skirts and long hair. With a model of you, the researchers can change your weight, clothing, and pose—and even make you perform a perfect pirouette. No practice necessary.



Beautiful projection mapping, with robots!

Another from the “Awesome Past Lives I Never Knew My Colleagues Had” Files: I just learned that Tarik Abdel-Gawad, with whom I’ve been collaborating on AR stuff, programmed & performed the amazing “Box” projection-mapping robot demo with Bot & Dolly before Google acquired that company. It’s now a few years old but no less stunning:


Bot & Dolly produced this work to serve as both an artistic statement and technical demonstration. It is the culmination of multiple technologies, including large scale robotics, projection mapping, and software engineering. We believe this methodology has tremendous potential to radically transform theatrical presentations, and define new genres of expression.

Check out this peek behind the scenes:

[YouTube 1 & 2]

Adobe’s “Project Puppetron” is now in beta

This super fun combo of style transfer & performance capture (see video below in case you missed the sneak peek last fall) is now accepting applications for beta testers:

Project Puppetron lets you capture your own face via webcam and, through a simple setup process, create a puppet of yourself in the style of a piece of referenced art.

[Y]ou perform various facial expressions and mouth shapes for lip sync, and then select the reference art and the level of stylization you want to apply to create a fully-realized, animated puppet.

Once Project Puppetron has created your puppet, you can perform your character or modify your puppet as you would any other puppet in Character Animator. Then, bring further dimension to your character’s performance with rigging, triggerable artwork, layer cycles, etc., through the broad array of tools offered in Character Animator.


[YouTube] [Via Margot Nack]

A beautiful visualization of flight paths in AR

It’s a running joke at Google that you can spend your whole career not knowing what the people nearby you are doing, or even who they are. (In fact, I continue to harbor a dream about creating an AR overlay that would solve this, but that’s another story.) As it happens, I just discovered that my teammate Bill has been making some really cool augmented reality experiments, including the open source Flight Paths:

Flight Paths is an experiment that transforms your room into a flight path visualization. Touch any horizontal surface and explore as flights take off from JFK or SFO and fly around your space. Learn more at

Continual note to self: Ask people more about what they’re doing!



Google Maps + Unity FTW!

Get ready for a whole new wave of AR gaming:

Per The Verge,

Unity integration will also allow developers to customize maps with what appears to be a great deal of flexibility and control. Things like buildings and roads are turned into objects, which developers can then tweak in the game engine. During a demonstration, Google showed off real-world maps that were transformed into sci-fi landscapes and fantasy realms, complete with dragons and treasure chests.

Jacoby says that one of the goals of the project was to help developers build detailed worlds using Maps data as a base to paint over. Developers can do things like choose particular kinds of buildings or locations — say, all stores or restaurants — and transform each one. A fantasy realm could turn all hotels into restorative inns, for instance, or anything else.