Facebook Reality Labs’ plans for building immersive audio will involve glasses, but could also use earbuds, like this in-ear monitor prototype.
Facebook’s future plans for AR smartglasses are still in the works, but one part of the equation that could soon see the light of day is smart audio. Spatial audio and augmented audio reality are two arenas in which Facebook is actively interested, and the company’s newest Facebook Reality Labs research update details how the company plans to make devices that can tune out the real world and make virtual objects sound like they’re right next to you, even if they’re not really there.
The two separate technologies, which Facebook is calling Audio Presence and Smart Hearing, are both parts of an audio puzzle the company’s trying to crack. Meanwhile, the larger augmented reality field remains an unresolved landscape, with players including Apple, Google, Microsoft, Magic Leap and others aiming to drive the next decade of innovation.
Facebook Reality Lab’s AR audio research is using prototype glasses frames with directional microphones for focusing attention.
Facebook originally planned a demo of its new audio tech during the upcoming Facebook Connect conference later this month. Instead, due to social-distancing concerns because of the coronavirus, reporters were briefed virtually in a chat with Facebook Reality Labs research head Michael Abrash and an audio research team.
Smart audio AR isn’t new: Bose had been experimenting with it for years before its AR audio division was apparently shuttered earlier this year. Apple’s AirPods Pro have active noise canceling and passthrough, and will be adding 3D spatial audio support this fall. Spatial audio is already a key feature in realistic VR, but Facebook’s next-wave plans focus more on conversations, attention and fidelity of audio, including 3D placement of invisible speakers in places that seem like they’re in the same room. “The most remarkable thing to me is that everything [about Facebook’s new audio tech] is absolutely feasible within a reasonable period of time,” Abrash said.
According to Abrash, it’s possible the audio tech will appear on VR headsets like the Oculus Quest eventually, or even other Facebook devices.
The experiments Facebook has been currently running using spatial audio involve using an anechoic chamber and a robot arm with a speaker array to surround a person and calculate their specific spatial audio geometry for their ears, which takes 30 minutes, so it’s hardly portable yet. The next step would be to shrink that process down to taking a photograph of an ear to get that measurement (which Facebook calls head-related transfer function, or HRTF). Smartglasses are still a considerable ways away, and the setups that Facebook has been using to test the audio once the measurements are taken are more like glasses with audio playback and focusing microphones, with no visual element yet.
Facebook Reality Lab Research’s audio group is also looking to make this audio tech assistive, listening and enhancing conversations and improving clarity of speech while filtering background noise, or other people speaking.
Some of Facebook’s concepts of 3D audio and audio-sensing superpowers remind me of the work Doppler Labs tried years ago, on its Here One earbuds with various ambient noise filters. Facebook’s research uses beamforming to focus audio, also using spatially aware prototypes worn by testers that have directional microphones. The company has made a custom in-ear monitor to create this audio, with aims to eventually build this tech into future products.
The spatial audio gear right now is pretty experimental at the moment.
I’ve been wondering for years whether audio augmented reality could be a doorway to making always-on AR glasses work. Audio can be ambient, and people already wear earbuds all the time and have learned to make it work in their lives. Always-on holographic video is another story. On 2D phones, heads-up AR experiments like Google Maps have shown the limits of heads-up displays in the real world, while audio directions can be much less intrusive. Maybe audio could work as a signal cue to trigger visuals afterwards. Facebook hasn’t made that process clear yet, but its blog post promises that this future audio could work to navigate AR-optimized Live Maps that Facebook is currently assembling of the spatially mapped world.
Abrash promises that the company’s advances in spatial audio have produced realistic 3D audio that seems like it’s coming from the real world and not from headphones, but the current research involves complicated ear scans and a specific, spatially mapped room. The company is working on developing algorithms that could potentially work with a simple camera ear scan at home, making the tech more accessible.
Would this maybe mean Facebook earbuds? AR apps on mobile? Future directions for the Oculus Quest? Maybe yes to all of these. It’s a sign that the bigger, thornier questions of attention and focus in immersive AR wearables still need to be thought through. But it also suggests that Facebook Reality Labs could eventually be working on some sort of headphone technology.
