It once took more than 100 cameras to scan a photorealistic avatar for virtual reality (VR), but now researchers at Meta have revealed that it can be done with just an iPhone.
Facebook first announced “codec avatars” in 2019 which, it learned, are a class of photorealistic face models that accurately represent a person’s geometry and textures in 3D and therefore virtual reality (VR). Avatars are almost indistinguishable from video and the company is hoping that lifetime avatars will be important in the future, especially given its Metaverse ambitions.
The codec called Meta MUGSY was using a special rig to generate the avatar, which consisted of 171 high-resolution cameras, as seen below. But as reported by insertIt was a recent breakthrough and is able to bypass MUGSY by using a scan from a smartphone with a front-facing depth sensor, such as an iPhone that has FaceID.
To capture a codec avatar, the phone must be held to a neutral face, then mimicking a series of 65 facial expressions. Researchers say the scanning process takes an average of three-and-a-half minutes — though users will have to wait six hours for a machine with four high-end GPUs to fully render the avatar. It is likely that if this product is made available to the general public, rendering will take place on the cloud GPU and not on the user’s device.
The iPhone scanning system uses the Universal Prior Model (UPM) “hypernetwork”, a neural network that helps generate person-specific codec avatars. This UPM hypernetwork is trained by scanning the faces of 255 diverse individuals using an advanced capture rig similar to MUGSY but with 90 cameras.
Other researchers have already demonstrated avatar generation from smartphone scans, but Meta claims that the quality of its results is the state of the art. However, the system is limited to scanning only a person’s head and is unable to interpret glasses or long hair.
Codec Avatar is still some time away from release
Meta Avatar currently has a basic cartoon art style, so this type of high-fidelity is still a long way off. On that note, the avatars of Meta featured on products like Oculus Quest 2 have actually been reduced in realism over time to better accommodate the complex environments in apps like Horizon Worlds.
Zuckerberg’s avatar
Back in February, Mark Zuckerberg said in an interview with Lex Friedman that in the future VR users could use “expressionist” avatars in casual games and “realistic” avatars in work meetings.
Facebook declined to say when the codec Avatar might become available to general users, but the company believes it is getting closer.
image credit: Photos via Photo Scan, Authentic Volumetric Avatars from Meta Reality Labs
