Imagine watching a live ballgame while being able to fly around the field or court in VR, viewing the action from any angle.
That’s the sort of future that could be enabled by new research from Microsoft. A team there has devised a way to capture live performances, generate 3D models, and stream the results. Check it out:
The team writes,
We present the first fully automated end-to-end solution to create high-quality free-viewpoint video encoded as a compact data stream. Our system records performances using a dense set of RGB and IR video cameras, generates dynamic textured surfaces, and compresses these to a streamable 3D video format. Four technical advances contribute to high fidelity and robustness: multimodal multi-view stereo fusing RGB, IR, and silhouette information; adaptive meshing guided by automatic detection of perceptually salient areas; mesh tracking to create temporally coherent subsequences; and encoding of tracked textured meshes as an MPEG video stream. Quantitative experiments demonstrate geometric accuracy, texture fidelity, and encoding efficiency. We release several datasets with calibrated inputs and processed results to foster future research.
[YouTube] [Via Ben Grossman]