Passthrough camera access is stirring quite a buzz in the XR community lately. We already know where Meta, Apple, and Pico stand, but Google’s plans for Android XR have kept us guessing. After getting in touch with the company, I’ve got the scoop: they’re developing an approach akin to what’s available on smartphones. Stick around for the details!
## The Camera Access Dilemma
Let’s back up a bit for those scratching their heads. New standalone VR headsets are essentially MR (mixed reality) headsets now, using front cameras to provide an RGB passthrough view of the world. This sets the stage for delightful MR apps like Cubism, Starship Home, and Pencil.
Developers dream of accessing these camera frames to use AI and computer vision algorithms for enriching a user’s real-world view dynamically. I’ve always believed that camera access is essential for true mixed reality, making applications context-aware of their surroundings. For instance, by experimenting with unauthorized camera access techniques on the Quest, I’ve prototyped an AI+MR app for interior design—a feat that wouldn’t be feasible otherwise.
Now, this all sounds exciting until you consider the counterpoint: privacy. A rogue developer could access camera frames, run them through AI detection, and extract sensitive information like national IDs or bank cards left on a table while using their app. And that’s to say nothing of potential misuse involving facial or bodily images.
It’s a tricky balance: protecting user privacy while unlocking the potential of mixed reality.
## The Behavior of XR Companies
Initially, there was unrestricted camera access without much fuss. Those of you who’ve been following me may remember NTW’s camera texture experiments on the Vive Focus back around 2019: think diminished reality, Aruco marker tracking, and sound reactivity.
As MR gained traction, companies started playing it safe by restricting camera frame access due to privacy concerns. Meta, Pico, HTC, and later Apple, all followed this trend.
This became the norm until the XR development community voiced the necessity for camera access, mounting pressure on XR vendors. Advocates like myself, Cix Liv, and Michael Gschwandtner called for transparent methods for accessing camera frames, enabling object recognition and computer vision with user consent. We questioned the disparity in approach compared to smartphones that simply require a user’s permission for camera use.
This push swayed XR companies, leading Meta to pledge a “Passthrough API” rollout earlier this year. But what’s Google planning for Android XR?
## Android XR to Treat the Headset Like a Phone
With Android dominating the smartphone OS market, developers can request camera access on phones with user permission. Google aims for seamless compatibility between Android apps and Android XR, mirroring smartphone behavior on its new OS. There was speculation, but I confirmed it via a detailed email chat with a Google spokesperson. Here’s her clarification on XR camera access:
Just like Android apps, XR developers can request access to camera frames with user permission. They can specify camera_id=0 for the world-facing stream (akin to the smartphone rear camera) or camera_id=1 for the selfie stream in standard Android terms. Both streams use standard Android Camera APIs: Camera2 and CameraX. If apps request the selfie stream, they receive an image of the user’s avatar from Avatar provider apps/services based on OpenXR tracked data like head, hand, and face movements. It’s awesome to see the standard camera handling classes like CameraX extend to Android XR headsets, allowing frame grabbing, media saving, and ML analysis, much the same as on phones.
The rear-facing camera stream equates to a recreated avatar, reminiscent of Apple’s Vision Pro approach. This is clever; it aligns Android XR’s behavior with smartphones—the “rear camera” views the world, while the “selfie camera” recreates the user’s face.
Google’s intent is crystal: to ensure existing Android apps operate seamlessly on Android XR, including those utilizing camera streams, fostering app stability. I’m thrilled by the consistency in permission requests across devices.
But maybe you’re wondering about raw camera access. Unfortunately, here’s Google’s stance:
“We aren’t providing paths for accessing non-standard (e.g., forward-facing, reconstructed inward) camera data yet.”
That hints at no immediate plans for non-standard stream access, although future enterprise solutions remain a possibility.
Moreover, what should Unity developers do? Android’s Camera2 and CameraX are native classes, yet adaptability remains. Unity could employ the WebcamTexture class for frame access. Alternately, one could fashion a JNI wrapper library for CameraX functions, bringing this capability to Unity developers.
## A Little Caveat About Android XR
Bear in mind, Android XR is still in its preview phase, with no official headset releases to date. Consequently, some elements discussed herein might shift upon official launch—though I doubt it’s likely, it’s worth noting.
## The Opening Up of Camera Access
As Google and Meta pave the path for camera access, it’s likely other XR companies will follow suit. Look out for 2025 to potentially usher in a wave of mixed reality innovation—I’m eager to see the developer community’s creativity flourish!
(Header image: Courtesy of Samsung)
Please note: This blog features ads and affiliate links that support its operations. Clicking an affiliate link will earn me a small commission, much to my delight. For the nitty-gritty details, see my full disclosure. Share the insight with fellow innovators!
