2.4 KiB
2.4 KiB
Developing camera features for XR devices, headsets, and AR glasses requires a shift from 2D pixel-pushing to 3D spatial awareness.
Understand the XR development mindset
| Concept | Mobile focus | XR focus |
|---|---|---|
| Input | Raw camera stream | Spatial tracking (visual-inertial odometry (VIO) or simultaneous localization and mapping (SLAM)) |
| Output | Screen viewfinder | Stereo passthrough and occlusion |
| Constraint | Battery life | Motion-to-photon latency (less than 20 ms) |
Follow the implementation guide
API selection
On XR devices, standard CameraX implementations are often restricted or
insufficient. Always use spatial software development kits (SDKs):
- ARCore: Use ARCore for plane detection, depth sensing, and motion tracking.
- OpenXR: Use OpenXR as the cross-platform standard for VR and AR rendering and input.
- OEM SDKs: Use manufacturer-specific libraries for hardware-accelerated passthrough.
Handle spatial passthrough
Unlike a 2D viewport, XR passthrough is often system-managed.
[Key requirement] Frame synchronization: Synchronize your application's frame clock with the headset's head-mounted display (HMD) pose.
// Example: Querying the spatial pose for the current camera frame
val headPose = xrSession.getHeadPose(frameTime)
val projectionMatrix = headPose.getProjectionMatrix(eyeIndex)
Manage depth and occlusion
Digital content must respect real-world depth to ensure accurate occlusion.
- Depth map : Access raw depth data using
ARCoreorSurfaceProcessorto create an occlusion mask. - Hardware buffers : Use
HardwareBufferto share camera frames directly with the GPU without CPU-side copies to minimize latency.
XR pitfalls
- The nausea limit: Any processing that delays the viewfinder by more than 20 ms causes user sickness. Don't perform image processing on the main thread.
- Privacy restrictions : Some XR devices return a black frame if you attempt to record the "Passthrough" layer. Check
Session.isRecordingSupported. - Field of view (FOV): The camera FOV possibly doesn't match the display FOV. Use the SDK's projection matrixes instead of calculating aspect ratios manually.
- Front buffer rendering : If the device supports it, use
FrontBufferRendererfor real-time overlays to bypass standard double-buffering latency.