skill: add Android skills
This commit is contained in:
198
.agents/skills/camerax/references/camera1-to-camerax.md
Normal file
198
.agents/skills/camerax/references/camera1-to-camerax.md
Normal file
@@ -0,0 +1,198 @@
|
||||
## Remove `Camera1` implementation
|
||||
|
||||
1. Delete all `android.hardware.Camera` instances.
|
||||
2. Delete `SurfaceView` and `SurfaceHolder.Callback` implementations `surfaceCreated`, `surfaceChanged`, and `surfaceDestroyed`.
|
||||
3. Remove custom lifecycle handling that opens or releases the camera in `onResume` or `onPause`.
|
||||
4. Remove manual matrix calculations for orientation.
|
||||
|
||||
## Initialize `ProcessCameraProvider`
|
||||
|
||||
Request the `ProcessCameraProvider` and bind use cases to the Activity or
|
||||
Fragment lifecycle.
|
||||
|
||||
|
||||
```kotlin
|
||||
val context = LocalContext.current
|
||||
val lifecycleOwner = LocalLifecycleOwner.current
|
||||
LaunchedEffect(context, lifecycleOwner) {
|
||||
val cameraProvider = ProcessCameraProvider.getInstance(context).await()
|
||||
|
||||
val cameraSelector = CameraSelector.Builder()
|
||||
.requireLensFacing(CameraSelector.LENS_FACING_BACK)
|
||||
.build()
|
||||
|
||||
val preview = Preview.Builder().build()
|
||||
val imageCapture = ImageCapture.Builder()
|
||||
.setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY)
|
||||
.build()
|
||||
|
||||
cameraProvider.unbindAll() // Unbind before rebinding
|
||||
|
||||
val camera = cameraProvider.bindToLifecycle(
|
||||
lifecycleOwner,
|
||||
cameraSelector,
|
||||
preview,
|
||||
imageCapture
|
||||
)
|
||||
val cameraControl = camera.cameraControl
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
## Implement the preview and tap-to-focus
|
||||
|
||||
Choose exactly one of the following patterns based on the app's UI toolkit:
|
||||
|
||||
### Option A: For Android Views
|
||||
|
||||
Use `androidx.camera.view.PreviewView`.
|
||||
|
||||
1. **Set up preview**:
|
||||
|
||||
|
||||
```kotlin
|
||||
preview.setSurfaceProvider(previewView.surfaceProvider)
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
2. **Handle tap-to-focus**:
|
||||
|
||||
|
||||
```kotlin
|
||||
val factory = previewView.meteringPointFactory
|
||||
val point = factory.createPoint(x, y) // x, y from touch event
|
||||
val action = FocusMeteringAction.Builder(point, FocusMeteringAction.FLAG_AF).build()
|
||||
cameraControl?.startFocusAndMetering(action)
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### Option B: For Jetpack Compose
|
||||
|
||||
Use `androidx.camera.compose.CameraXViewfinder`.
|
||||
|
||||
1. **Set up preview and SurfaceRequest**:
|
||||
|
||||
|
||||
```kotlin
|
||||
var surfaceRequest by remember { mutableStateOf<SurfaceRequest?>(null) }
|
||||
val preview = remember {
|
||||
Preview.Builder().build().apply {
|
||||
setSurfaceProvider { request -> surfaceRequest = request }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
2. **Render viewfinder**:
|
||||
|
||||
|
||||
```kotlin
|
||||
surfaceRequest?.let { request ->
|
||||
CameraXViewfinder(
|
||||
surfaceRequest = request,
|
||||
coordinateTransformer = coordinateTransformer,
|
||||
modifier = Modifier
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
3. **Handle tap-to-focus in Compose**:
|
||||
|
||||
|
||||
```kotlin
|
||||
// Inside your tap gesture handler...
|
||||
val surfaceCoords = with(coordinateTransformer) { offset.transform() }
|
||||
val factory = SurfaceOrientedMeteringPointFactory(
|
||||
request.resolution.width.toFloat(),
|
||||
request.resolution.height.toFloat()
|
||||
)
|
||||
val point = factory.createPoint(surfaceCoords.x, surfaceCoords.y)
|
||||
val action = FocusMeteringAction.Builder(point, FocusMeteringAction.FLAG_AF).build()
|
||||
cameraControl?.startFocusAndMetering(action)
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
4. **Update target rotation for Compose**:
|
||||
|
||||
|
||||
```kotlin
|
||||
LaunchedEffect(configuration) {
|
||||
if (!view.isInEditMode) {
|
||||
val rotation = view.display?.rotation ?: Surface.ROTATION_0
|
||||
imageCapture.targetRotation = rotation
|
||||
preview.targetRotation = rotation
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
## Capture a photo
|
||||
|
||||
Use the `ImageCapture` use case to take the picture. The `ImageProxy` handles
|
||||
rotation directly.
|
||||
|
||||
|
||||
```kotlin
|
||||
imageCapture.takePicture(
|
||||
cameraExecutor,
|
||||
object : ImageCapture.OnImageCapturedCallback() {
|
||||
override fun onCaptureSuccess(image: ImageProxy) {
|
||||
val buffer = image.planes[0].buffer
|
||||
val bytes = ByteArray(buffer.remaining())
|
||||
buffer.get(bytes)
|
||||
val bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
|
||||
|
||||
// Adjust rotation natively via ImageProxy
|
||||
val matrix = Matrix()
|
||||
matrix.postRotate(image.imageInfo.rotationDegrees.toFloat())
|
||||
if (lensFacing == CameraSelector.LENS_FACING_FRONT) {
|
||||
matrix.postScale(-1f, 1f) // Mirror for front camera
|
||||
}
|
||||
|
||||
val rotatedBitmap = Bitmap.createBitmap(
|
||||
bitmap, 0, 0, bitmap.width, bitmap.height, matrix, true
|
||||
)
|
||||
|
||||
// MUST close proxy
|
||||
image.close()
|
||||
}
|
||||
|
||||
override fun onError(exception: ImageCaptureException) {
|
||||
Log.e("CameraX", "Capture failed: ${exception.message}", exception)
|
||||
}
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
## Switch cameras
|
||||
|
||||
To flip between front and rear cameras, change the `CameraSelector` and
|
||||
retrigger the `ProcessCameraProvider` logic.
|
||||
|
||||
|
||||
```kotlin
|
||||
lensFacing = if (lensFacing == CameraSelector.LENS_FACING_BACK) {
|
||||
CameraSelector.LENS_FACING_FRONT
|
||||
} else {
|
||||
CameraSelector.LENS_FACING_BACK
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
## Follow constraints
|
||||
|
||||
- **Don't manage the camera lifecycle manually** : Bind the camera to a `LifecycleOwner` through the `ProcessCameraProvider`. Avoid manual camera open or close logic in `onResume` or `onPause`.
|
||||
- **Don't calculate focus matrices manually** : `MeteringPointFactory` handles coordinate transformations, including device rotation offsets. Avoid custom matrix implementations.
|
||||
- **Don't forget to close the `ImageProxy`** : Remember to invoke `image.close()` in the capture callback. Skipping this call locks the capture pipeline and interrupts subsequent photos.
|
||||
- **Don't wrap `PreviewView` in `AndroidView` for Compose code** : For Compose UI layouts, use `CameraXViewfinder`. Compiling `PreviewView` in an `AndroidView` is an earlier fallback option that introduces resizing issues.
|
||||
193
.agents/skills/camerax/references/camera2-to-camerax.md
Normal file
193
.agents/skills/camerax/references/camera2-to-camerax.md
Normal file
@@ -0,0 +1,193 @@
|
||||
Camera2 offers granular control but introduces boilerplate: managing
|
||||
`CameraDevice` states, `CameraCaptureSession` lifecycles, background threads,
|
||||
`HandlerThread`, and manual orientation calculations.
|
||||
|
||||
CameraX simplifies this by binding high-level `UseCase`s such as `Preview`,
|
||||
`ImageCapture`, and `ImageAnalysis` directly to Android lifecycles, handling
|
||||
thread management and device-specific workarounds automatically.
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Remove Camera2 boilerplate
|
||||
|
||||
Migrating to CameraX removes manual setup code:
|
||||
|
||||
1. **Delete manual thread handling** : Remove `HandlerThread`, `Handler`, and executors dedicated to camera background tasks. CameraX manages its own threads.
|
||||
2. **Remove session and device callbacks** : Delete `CameraDevice.StateCallback` and `CameraCaptureSession.StateCallback` implementations.
|
||||
3. **Remove surface management** : Remove manual routing of `Surface` objects from `TextureView` or `SurfaceView` to the camera device.
|
||||
4. **Remove manual orientation logic** : Delete calculations involving `CameraCharacteristics.SENSOR_ORIENTATION` and display rotation for adjusting preview and capture orientation.
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Initialize `ProcessCameraProvider`
|
||||
|
||||
Request the `ProcessCameraProvider` and bind your use cases to the
|
||||
`LifecycleOwner` activity or fragment. This replaces the
|
||||
`CameraManager.openCamera` flow.
|
||||
|
||||
|
||||
```kotlin
|
||||
val context = LocalContext.current
|
||||
val lifecycleOwner = LocalLifecycleOwner.current
|
||||
LaunchedEffect(context, lifecycleOwner) {
|
||||
val cameraProvider = ProcessCameraProvider.getInstance(context).await()
|
||||
|
||||
val cameraSelector = CameraSelector.Builder()
|
||||
.requireLensFacing(CameraSelector.LENS_FACING_BACK)
|
||||
.build()
|
||||
|
||||
val preview = Preview.Builder().build()
|
||||
val imageCapture = ImageCapture.Builder()
|
||||
.setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY)
|
||||
.build()
|
||||
|
||||
// ImageAnalysis is common when migrating from Camera2 ImageReader
|
||||
val imageAnalysis = ImageAnalysis.Builder()
|
||||
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
|
||||
.build()
|
||||
|
||||
cameraProvider.unbindAll()
|
||||
|
||||
val camera = cameraProvider.bindToLifecycle(
|
||||
lifecycleOwner,
|
||||
cameraSelector,
|
||||
preview,
|
||||
imageCapture,
|
||||
imageAnalysis
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Implement the preview and tap-to-focus
|
||||
|
||||
CameraX handles surface configuration automatically. Choose based on your UI
|
||||
toolkit:
|
||||
|
||||
### Option A: For Android Views
|
||||
|
||||
Use `androidx.camera.view.PreviewView` in your layout, and bind it to the
|
||||
`Preview` use case.
|
||||
|
||||
|
||||
```kotlin
|
||||
preview.setSurfaceProvider(previewView.surfaceProvider)
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### Option B: For Jetpack Compose
|
||||
|
||||
Use `androidx.camera.compose.CameraXViewfinder`.
|
||||
|
||||
|
||||
```kotlin
|
||||
var surfaceRequest by remember { mutableStateOf<SurfaceRequest?>(null) }
|
||||
val preview = remember {
|
||||
Preview.Builder().build().apply {
|
||||
setSurfaceProvider { request -> surfaceRequest = request }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Capture a photo
|
||||
|
||||
Replace `ImageReader` capture flows and
|
||||
`CaptureRequest.Builder.TEMPLATE_STILL_CAPTURE` with the `ImageCapture` use
|
||||
case. CameraX handles the rotation natively using the returned `ImageProxy`.
|
||||
|
||||
|
||||
```kotlin
|
||||
imageCapture.takePicture(
|
||||
cameraExecutor,
|
||||
object : ImageCapture.OnImageCapturedCallback() {
|
||||
override fun onCaptureSuccess(image: ImageProxy) {
|
||||
val buffer = image.planes[0].buffer
|
||||
val bytes = ByteArray(buffer.remaining())
|
||||
buffer.get(bytes)
|
||||
val bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
|
||||
if (bitmap != null) {
|
||||
val matrix = Matrix()
|
||||
matrix.postRotate(image.imageInfo.rotationDegrees.toFloat())
|
||||
if (lensFacing == CameraSelector.LENS_FACING_FRONT) {
|
||||
matrix.postScale(-1f, 1f)
|
||||
}
|
||||
|
||||
val rotatedBitmap = Bitmap.createBitmap(
|
||||
bitmap, 0, 0, bitmap.width, bitmap.height, matrix, true
|
||||
)
|
||||
}
|
||||
|
||||
// MUST close proxy
|
||||
image.close()
|
||||
}
|
||||
|
||||
override fun onError(exception: ImageCaptureException) {
|
||||
Log.e("CameraX", "Capture failed: ${exception.message}", exception)
|
||||
}
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Implement image analysis
|
||||
|
||||
If you were using `ImageReader` in Camera2 to access raw frames, e.g., for QR
|
||||
scanning or ML,, replace it with the CameraX `ImageAnalysis` use case.
|
||||
|
||||
|
||||
```kotlin
|
||||
imageAnalysis.setAnalyzer(cameraExecutor) { imageProxy ->
|
||||
try {
|
||||
val rotationDegrees = imageProxy.imageInfo.rotationDegrees
|
||||
// Process image here (e.g., run object detection)
|
||||
// ...
|
||||
} finally {
|
||||
// MUST close the imageProxy to avoid blocking the pipeline
|
||||
imageProxy.close()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Use Camera2 interop
|
||||
|
||||
If your app requires specific Camera2 configuration options, such as custom
|
||||
exposure modes or flash settings, that aren't exposed directly in CameraX,
|
||||
use `Camera2Interop` to apply them to your CameraX use cases.
|
||||
|
||||
|
||||
```kotlin
|
||||
// Use Camera2Interop to set Camera2-specific capture options
|
||||
val extender = Camera2Interop.Extender(imageCaptureBuilder)
|
||||
extender.setCaptureRequestOption(
|
||||
CaptureRequest.CONTROL_AE_MODE,
|
||||
CaptureRequest.CONTROL_AE_MODE_OFF
|
||||
).setCaptureRequestOption(
|
||||
CaptureRequest.FLASH_MODE,
|
||||
CaptureRequest.FLASH_MODE_TORCH
|
||||
)
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Follow constraints
|
||||
|
||||
- **Don't open the CameraDevice manually** : Let CameraX manage device opening and closing using `bindToLifecycle`.
|
||||
- **Don't forget to close `ImageProxy`** : In both `ImageCapture` and `ImageAnalysis` callbacks, you **must** call `image.close()`. Failure to do so will block the camera pipeline.
|
||||
- **Don't block the ImageAnalysis thread** : The analyzer runs frame-by-frame. If you need to perform heavy computations, offload them to a background worker and close the `ImageProxy` as soon as possible.
|
||||
58
.agents/skills/camerax/references/expert-blueprints.md
Normal file
58
.agents/skills/camerax/references/expert-blueprints.md
Normal file
@@ -0,0 +1,58 @@
|
||||
Complex camera features often fail due to "Agent Stall" or timeouts when
|
||||
attempted in a single turn. Use these blueprints to break tasks into manageable
|
||||
phases.
|
||||
|
||||
## Manual controls
|
||||
|
||||
### Phase one: ViewModel and state
|
||||
|
||||
1. Define a `ManualSettings` data class.
|
||||
2. Add a `MutableStateFlow<ManualSettings>` to your `CameraViewModel`.
|
||||
3. Implement the Jetpack Compose UI with sliders and switches to update this flow.
|
||||
|
||||
### Phase two: Controller wiring
|
||||
|
||||
1. In `CameraController.kt`, create a new function `updateManualSettings(settings: ManualSettings)`.
|
||||
2. Map these settings into the `CameraSystem` layer.
|
||||
|
||||
### Phase three: CameraX `Camera2Interop` wiring
|
||||
|
||||
1. In `CameraSession.kt`, use the `Camera2Interop.Extender` utility to access Camera2 capture request keys.
|
||||
2. Apply the hardware keys:
|
||||
- `CaptureRequest.SENSOR_SENSITIVITY` to set ISO sensitivity.
|
||||
- `CaptureRequest.SENSOR_EXPOSURE_TIME` to set exposure time.
|
||||
- `CaptureRequest.LENS_FOCUS_DISTANCE` to set focus distance.
|
||||
3. **Critical** : If manual exposure is active, set `CaptureRequest.CONTROL_AE_MODE` to `CameraMetadata.CONTROL_AE_MODE_OFF`. ---
|
||||
|
||||
## RAW and JPEG capture
|
||||
|
||||
### Phase one: Output configuration
|
||||
|
||||
1. Verify device support for RAW capture using `CameraInfo`.
|
||||
2. Configure `ImageCapture.Builder` with `OUTPUT_FORMAT_RAW_JPEG` or `OUTPUT_FORMAT_RAW`.
|
||||
|
||||
### Phase two: Implementation
|
||||
|
||||
1. Provide `ImageCapture.OutputFileOptions` for the target storage locations.
|
||||
2. Invoke `takePicture`. CameraX internally manages `DngCreator` to wrap RAW data with the required `CameraCharacteristics` and `CaptureResult` metadata.
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Apply image effects
|
||||
|
||||
### Phase one: Effects selection
|
||||
|
||||
1. Use the `androidx.media3:media3-effect` dependency.
|
||||
2. Use `RgbFilter` or `HslAdjustment` for standard color grading.
|
||||
|
||||
### Phase two: Application
|
||||
|
||||
1. Configure `Composition.Builder` or `MediaItem.Builder` with the list of effects.
|
||||
2. Inject the list of effects into the CameraX `Recorder` or `Preview` using the `setEffects` method.
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Low-light capture
|
||||
|
||||
For guidance on Night Mode Extensions and Low Light Boost,
|
||||
[low-light.md](https://developer.android.com/agents/skills/camera/camerax/references/low-light).
|
||||
83
.agents/skills/camerax/references/foldables.md
Normal file
83
.agents/skills/camerax/references/foldables.md
Normal file
@@ -0,0 +1,83 @@
|
||||
Foldable devices introduce unique challenges for camera applications, including
|
||||
dynamic layout changes, multiple display orientations, and physical device
|
||||
postures, such as tabletop and book modes.
|
||||
|
||||
## Manage fold states and postures
|
||||
|
||||
| State | Posture | User interaction | Implementation goal |
|
||||
|---|---|---|---|
|
||||
| `FLAT` | Standard | Full screen preview | Conventional mobile phone layout. |
|
||||
| `HALF_OPENED` | Tabletop | Lower half for controls | Split-screen layout, viewfinder on top, controls on bottom. |
|
||||
| `HALF_OPENED` | Book | Side-by-side | Viewfinder on one panel, gallery and controls on the other. |
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Follow the implementation guide
|
||||
|
||||
### Detect posture changes
|
||||
|
||||
Use the Jetpack WindowManager library to observe the device's hinge state and
|
||||
fold layout.
|
||||
|
||||
|
||||
```kotlin
|
||||
lifecycleScope.launch {
|
||||
lifecycle.repeatOnLifecycle(Lifecycle.State.STARTED) {
|
||||
windowInfoTracker.windowLayoutInfo(activity)
|
||||
.collect { layoutInfo ->
|
||||
val displayFeature = layoutInfo.displayFeatures
|
||||
.filterIsInstance<FoldingFeature>()
|
||||
.firstOrNull()
|
||||
|
||||
updateCameraLayout(displayFeature)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### Handle tabletop mode
|
||||
|
||||
In Tabletop mode, horizontal fold, you should move the viewfinder to the top
|
||||
half of the screen and the controls to the bottom half to prevent the user from
|
||||
seeing a "bent" image.
|
||||
|
||||
- **Identify orientation:** Check `FoldingFeature.orientation`.
|
||||
- **Calculate geometry:** Use `FoldingFeature.bounds` to identify the hinge's physical location on the screen.
|
||||
- **Update UI:** Apply padding or constraints to move the `PreviewView` above the hinge.
|
||||
|
||||
### Coordinate mapping and `Viewport`
|
||||
|
||||
When the UI layout changes due to a fold, you **must** update the `Viewport` to
|
||||
ensure that tap-to-focus and image capture coordinates remain accurate.
|
||||
|
||||
|
||||
```kotlin
|
||||
val viewport = ViewPort.Builder(Rational(viewfinder.width, viewfinder.height), display.rotation)
|
||||
.setScaleType(ViewPort.FILL_CENTER)
|
||||
.build()
|
||||
|
||||
val useCaseGroup = UseCaseGroup.Builder()
|
||||
.addUseCase(preview)
|
||||
.setViewPort(viewport)
|
||||
.build()
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### Rear display mode
|
||||
|
||||
Some foldables allow using the rear camera with the cover display while the
|
||||
device is unfolded.
|
||||
|
||||
- **Verification:** If available through OEM SDKs or Android 14 (API level 34) or higher, check `DeviceState.REAR_DISPLAY_STATE`.
|
||||
- **Logic:** Handle preview detachment and reattachment on different display surfaces with varying aspect ratios.
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Foldable pitfalls
|
||||
|
||||
- **Hinge distortion:** Don't span the camera preview across a hinge in `HALF_OPENED` state.
|
||||
- **Physical orientation:** The camera sensor's physical orientation relative to the screen often changes when you fold or unfold the device. Always rely on `CameraInfo.getSensorRotationDegrees`.
|
||||
- **Latency:** Rebinding `UseCase` objects during a fold event is expensive. Use the internal scaling of `PreviewView` before performing a full `bindToLifecycle` reconfiguration.
|
||||
60
.agents/skills/camerax/references/immutability.md
Normal file
60
.agents/skills/camerax/references/immutability.md
Normal file
@@ -0,0 +1,60 @@
|
||||
Many Android APIs are designed with immutability in mind to prevent race
|
||||
conditions in async environments. However, this often trips up developers used
|
||||
to mutable builder patterns.
|
||||
|
||||
## Common immutable classes
|
||||
|
||||
The following classes use fluent APIs that **return a new instance**. You must
|
||||
reassign the variable.
|
||||
|
||||
| Class | Methods that return a new instance | Result if not reassigned |
|
||||
|---|---|---|
|
||||
| `PendingRecording` | `withAudioEnabled`, `asPersistentRecording` | Audio isn't recorded. |
|
||||
| `ImageCapture.Builder` | `setTargetRotation`, `setTargetResolution` | The output has the wrong orientation. |
|
||||
| `Recorder.Builder` | `setQualitySelector`, `setExecutor` | The recording uses the default quality. |
|
||||
| `Viewport.Builder` | `setScaleType`, `setLayoutDirection` | The viewfinder is stretched. |
|
||||
|
||||
## Use standard patterns
|
||||
|
||||
### CameraX video recording
|
||||
|
||||
To set up video recording, use the following code:
|
||||
|
||||
|
||||
```kotlin
|
||||
// WRONG
|
||||
run {
|
||||
val pending = recorder.prepareRecording(context, opts)
|
||||
pending.withAudioEnabled() // This returns a new instance which is ignored
|
||||
val active = pending.start(exec, listener)
|
||||
}
|
||||
|
||||
// CORRECT
|
||||
run {
|
||||
val pending = recorder.prepareRecording(context, opts)
|
||||
.withAudioEnabled() // Chaining works
|
||||
val active = pending.start(exec, listener)
|
||||
}
|
||||
|
||||
// ALSO CORRECT
|
||||
run {
|
||||
var pending = recorder.prepareRecording(context, opts)
|
||||
pending = pending.withAudioEnabled() // Reassignment
|
||||
val active = pending.start(exec, listener)
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### Viewport construction
|
||||
|
||||
To set up the viewport, use the following code:
|
||||
|
||||
|
||||
```kotlin
|
||||
val viewport = ViewPort.Builder(Rational(width, height), displayRotation)
|
||||
.setScaleType(ViewPort.FILL_CENTER)
|
||||
.build()
|
||||
```
|
||||
|
||||
<br />
|
||||
124
.agents/skills/camerax/references/low-light.md
Normal file
124
.agents/skills/camerax/references/low-light.md
Normal file
@@ -0,0 +1,124 @@
|
||||
This guide covers implementing low-light features using **Night mode
|
||||
extensions** and **Low Light Boost (LLB)**.
|
||||
|
||||
## Choosing the right tool
|
||||
|
||||
| Feature | Used for | Implementation | UX impact |
|
||||
|---|---|---|---|
|
||||
| **Night mode** | High-quality stills | `ExtensionsManager` | Possibly requires user to hold still for several seconds. |
|
||||
| **LLB, AE mode** | Real-time preview or video | `Camera2Interop`, CameraX utility | Hardware drops the frame rate to increase brightness. |
|
||||
| **LLB, Play services** | Real-time preview and video | `SurfaceProcessor` | Software-based brightening; maintains a higher frame rate. |
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Night mode extension
|
||||
|
||||
CameraX Extensions provide access to the device's built-in computational
|
||||
photography pipeline.
|
||||
|
||||
### Basic setup
|
||||
|
||||
To set up the extension, initialize the extension manager:
|
||||
|
||||
|
||||
```kotlin
|
||||
// Use ListenableFuture.await() extension function for coroutine support
|
||||
val extensionsManager = ExtensionsManager.getInstanceAsync(context, cameraProvider).await()
|
||||
if (extensionsManager.isExtensionAvailable(cameraSelector, ExtensionMode.NIGHT)) {
|
||||
val nightSelector = extensionsManager.getExtensionEnabledCameraSelector(
|
||||
cameraSelector, ExtensionMode.NIGHT
|
||||
)
|
||||
cameraProvider.bindToLifecycle(lifecycleOwner, nightSelector, imageCapture, preview)
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### Comprehensive features
|
||||
|
||||
- **Image postview** : Display a low-resolution image immediately while the multi-frame processing occurs.
|
||||
|
||||
```kotlin
|
||||
val imageCapture = ImageCapture.Builder()
|
||||
.setPostviewEnabled(true)
|
||||
.build()
|
||||
```
|
||||
- **Extension strength** : Let users control the intensity of the night effect.
|
||||
|
||||
```kotlin
|
||||
// Set the strength of the active extension (e.g. NIGHT mode intensity)
|
||||
val extensionsManager = ExtensionsManager.getInstanceAsync(context, cameraProvider).await()
|
||||
val extensionsControl = extensionsManager.getCameraExtensionsControl(camera.cameraControl)
|
||||
extensionsControl?.setExtensionStrength(strength)
|
||||
```
|
||||
- **Capture progress** : Show a UI progress bar for long exposures.
|
||||
|
||||
```kotlin
|
||||
// Use the suspend extension function for takePicture to avoid callback boilerplate
|
||||
try {
|
||||
val result = imageCapture.takePicture(outputOptions)
|
||||
// Use result.savedUri or other fields
|
||||
} catch (e: ImageCaptureException) {
|
||||
// Handle capture failure
|
||||
}
|
||||
```
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Low-light boost
|
||||
|
||||
LLB is designed for preview and video streams where you prefer high frame rates.
|
||||
|
||||
### AE mode
|
||||
|
||||
The built-in CameraX way to prioritize brightness. It modifies the hardware's
|
||||
auto-exposure algorithm.
|
||||
|
||||
- **Activation** : Use `CameraControl.enableLowLightBoostAsync`.
|
||||
- **Implementation** :
|
||||
|
||||
```kotlin
|
||||
// Enable Low Light Boost (LLB) natively in CameraX 1.4+
|
||||
camera.cameraControl.enableLowLightBoostAsync(true)
|
||||
```
|
||||
- **Monitoring** : Observe `CameraInfo.lowLightBoostState` to track when the hardware actively applies the enhancement.
|
||||
|
||||
### Google Play services LLB
|
||||
|
||||
It's a multi-step implementation that uses a session-based `SurfaceProcessor`.
|
||||
|
||||
**Dependency** : `com.google.android.gms:play-services-camera-low-light-boost`
|
||||
|
||||
To implement Google Play services LLB, follow these core steps:
|
||||
|
||||
1. **Initialize client** : `val client = LowLightBoost.getClient`.
|
||||
2. **Implement `SurfaceProcessor`** :
|
||||
- **Manage session** : Call `client.createSession`.
|
||||
- **Forward required metadata** : Observe the camera's `TotalCaptureResult` stream and forward every result to the session: `session.processCaptureResult`.
|
||||
- **Provide surface** : Get the input surface from the session, `session.getCameraSurface`, and provide it to the camera's `SurfaceRequest`.
|
||||
- **Lifecycle** : Release the session, `session.release`, when the processor is closed or the `SurfaceRequest` completes.
|
||||
3. **Wire using `CameraEffect`** :
|
||||
|
||||
```kotlin
|
||||
val effect = SimpleCameraEffect(
|
||||
CameraEffect.PREVIEW or CameraEffect.VIDEO_CAPTURE,
|
||||
executor,
|
||||
llbSurfaceProcessor
|
||||
) { throw it }
|
||||
|
||||
// Add to UseCaseGroup
|
||||
val useCaseGroup = UseCaseGroup.Builder()
|
||||
.addUseCase(preview)
|
||||
.addUseCase(videoCapture)
|
||||
.addEffect(effect)
|
||||
.build()
|
||||
```
|
||||
4. **Scene detection** : Use `session.setSceneDetectorCallback` to receive `boostStrength` updates for real-time UI indicators.
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Implementation notes
|
||||
|
||||
- **Thread safety** : Always handle `ExtensionsManager` and `LowLightBoostClient` initialization asynchronously.
|
||||
- **FPS trade-offs**: AE mode LLB often drops the frame rate significantly to increase brightness.
|
||||
- **Compatibility** : Extensions, Night Mode, possibly conflict with `ConcurrentCamera`. Always verify support before binding.
|
||||
75
.agents/skills/camerax/references/mlkit-spatial.md
Normal file
75
.agents/skills/camerax/references/mlkit-spatial.md
Normal file
@@ -0,0 +1,75 @@
|
||||
When you use ML Kit for features such as face mesh, object detection, or pose
|
||||
detection, the most common failure point is the coordinate disparity between the
|
||||
analysis image and the viewfinder UI.
|
||||
|
||||
## The mapping mindset
|
||||
|
||||
| Dimension | Analysis frame | Viewfinder UI on the screen |
|
||||
|---|---|---|
|
||||
| Resolution | Fixed, such as 640x480 | Dynamic, such as 1080x2400 |
|
||||
| Rotation | 0° for the raw buffer | 90° or 270° in portrait or landscape mode |
|
||||
| Origin | Top-left of buffer at (0,0) | Top-left of screen at (0,0) |
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Follow the implementation guide
|
||||
|
||||
### Coordinate transformation matrix
|
||||
|
||||
Android provides the `Viewport` and `UseCaseGroup` APIs to calculate the
|
||||
transformation matrix automatically. **Don't** calculate aspect ratio scaling
|
||||
manually.
|
||||
|
||||
|
||||
```kotlin
|
||||
val transform = previewView.viewPort?.let { viewPort ->
|
||||
// Use CameraX's built-in coordinate mapper
|
||||
viewPort.getTransformationMatrix(imageProxy.imageInfo.rotationDegrees)
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### Handling the "double rotation" bug
|
||||
|
||||
ML Kit results, bounding boxes, are relative to the **rotated buffer**. If the
|
||||
device is in portrait, the buffer is often 480x640, landscape, but the screen
|
||||
is 1080x1920.
|
||||
|
||||
To map the coordinates, use the following workflow:
|
||||
|
||||
1. Query `imageProxy.imageInfo.rotationDegrees`.
|
||||
2. Pass this rotation to the ML Kit `InputImage`.
|
||||
3. Use the `MappingUtils.transformRect` method to map the result `Rect` to the screen.
|
||||
|
||||
### Face mesh and pose normalization
|
||||
|
||||
For high-precision spatial analysis, for example, "Is the user's hand at a
|
||||
specific screen button?", use **normalized coordinates from 0.0 to 1.0**.
|
||||
|
||||
|
||||
```kotlin
|
||||
// Example: Converting a Pose landmark to a Screen Coordinate
|
||||
val screenX = landmark.position.x / analysisWidth * screenWidth
|
||||
val screenY = landmark.position.y / analysisHeight * screenHeight
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
**Warning** : Always account for **mirrored lenses** . If the `LENS_FACING_FRONT`
|
||||
is used, you must flip the X-coordinate: `actualX = screenWidth - screenX`.
|
||||
|
||||
### Overlays and canvas clipping
|
||||
|
||||
Use a custom `GraphicOverlay` view on top of the `PreviewView`.
|
||||
|
||||
- **Buffer lock** : Ensure your `GraphicOverlay` clears its canvas every time a new `ImageAnalysis` frame is processed to prevent "ghosting" of bounding boxes.
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Spatial pitfalls
|
||||
|
||||
- **The "stretched box" bug** : Caused by assuming the Analysis Frame aspect ratio, 4:3, matches the screen aspect ratio, 21:9. Use `PreviewView.SCALE_TYPE_FILL_CENTER` and map coordinates accordingly.
|
||||
- **Latency** : If ML processing exceeds 50 ms, the bounding box trails behind the user's face.
|
||||
- **Fix** : Use `ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST` to avoid queuing stale frames.
|
||||
- **Sensor vs. display rotation** : On some tablets, the sensor is mounted horizontally. Always query `display.rotation` and `cameraInfo.sensorRotationDegrees`.
|
||||
42
.agents/skills/camerax/references/modern-apis.md
Normal file
42
.agents/skills/camerax/references/modern-apis.md
Normal file
@@ -0,0 +1,42 @@
|
||||
Always prefer these various abstractions over legacy Camera2 or early CameraX
|
||||
implementations.
|
||||
|
||||
## Compare APIs
|
||||
|
||||
| Use case | Legacy or verbose way | Recommendation |
|
||||
|---|---|---|
|
||||
| QR or face scanning | `ImageAnalysis.Analyzer` and manual ByteBuffer math | **`MlKitAnalyzer`**, which automates coordinate mapping and multi-format support |
|
||||
| Post-processing | Custom OpenGL shaders or `SurfaceTexture` | **`Media3Effect`** for composable, declarative effects |
|
||||
| Dual camera | Manual binding of two UseCases | **`ConcurrentCamera`**, which provides built-in support in CameraX 1.3 and higher |
|
||||
| High dynamic range | Manual bit-depth and profile config | **`DynamicRange`** , which uses `DYNAMIC_RANGE_HLG10` or `SDR` |
|
||||
| Zoom and focus | `Camera2Interop` for CameraX-to-Camera2 mapping | **`CameraControl.setZoomRatio`** or **`setLinearZoom`** |
|
||||
|
||||
## Hardware awareness
|
||||
|
||||
Modern APIs abstract away the complexity of hardware diversity.
|
||||
|
||||
- **`CameraSelector`** : Use `DEFAULT_BACK_CAMERA` or `DEFAULT_FRONT_CAMERA` instead of hardcoding camera IDs. Use `filter` if you need specific lens capabilities.
|
||||
- **Extensions** : Before enabling advanced modes, such as night, bokeh, and face retouch, use `ExtensionsManager` to query whether the device supports them.
|
||||
- **Foldables** : Observe `Lifecycle` and `Viewport` updates to handle posture changes, like a half-opened posture, on foldable devices.
|
||||
|
||||
## Required dependencies
|
||||
|
||||
Add the following dependencies to your `libs.versions.toml` file:
|
||||
|
||||
# CameraX ML Kit
|
||||
|
||||
androidx-camera-mlkit-vision = { group = "androidx.camera", name =
|
||||
"camera-mlkit-vision", version.ref = "camerax" }
|
||||
|
||||
# Media3 effects
|
||||
|
||||
androidx-media3-effect = { group = "androidx.media3", name = "media3-effect",
|
||||
version.ref = "media3" }
|
||||
|
||||
# Camera extensions
|
||||
|
||||
androidx-camera-extensions = { group = "androidx.camera", name =
|
||||
"camera-extensions", version.ref = "camerax" }
|
||||
|
||||
Refer to the official [CameraX Release Notes](https://developer.android.com/jetpack/androidx/releases/camera) for the
|
||||
stable versions.
|
||||
69
.agents/skills/camerax/references/testing.md
Normal file
69
.agents/skills/camerax/references/testing.md
Normal file
@@ -0,0 +1,69 @@
|
||||
Automated testing for camera features is notoriously difficult because you
|
||||
can't easily mock physical hardware, lighting, or motion. This guide provides
|
||||
patterns for reliable, hermetic camera tests.
|
||||
|
||||
## Develop a testing mindset
|
||||
|
||||
| Challenge | Conventional approach | Camera technical approach |
|
||||
|---|---|---|
|
||||
| **Frameworks** | `Mockito` or `MockK` | **Fakes over mocks** |
|
||||
| **Assertions** | `assertEquals`, `assertTrue` | **Google Truth, `assertThat`** |
|
||||
| **Environment** | Implied environments | **Explicit `@RunWith` annotations** |
|
||||
| **Async operations** | `Thread.sleep` | **Explicit `timeoutMillis` or `IdlingResource`** |
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Follow the implementation guide
|
||||
|
||||
### Fakes over mocks
|
||||
|
||||
**Don't use Mockito.** Relying on mocks for complex, rapidly changing interfaces
|
||||
like `ImageProxy` or `CameraInfo` makes tests brittle. Instead, build "Fake"
|
||||
implementations that verify state rather than behavior.
|
||||
|
||||
|
||||
```kotlin
|
||||
// Create a Fake ImageProxy for ML Testing (Fakes over Mocks)
|
||||
val fakeImage = FakeImageProxy(w = 640, h = 480)
|
||||
|
||||
// Feed the fake buffer into your analyzer
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### Mock camera capabilities
|
||||
|
||||
Use `FakeAppConfig` from `androidx.camera:camera-testing` to simulate specific
|
||||
hardware constraints in tests, such as a device without a flash.
|
||||
|
||||
|
||||
```kotlin
|
||||
// Use awaitInstance() extension function for coroutine-based provider retrieval
|
||||
val cameraProvider = ProcessCameraProvider.awaitInstance(context)
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### Use Truth assertions
|
||||
|
||||
Use Google Truth, `assertThat`, instead of standard JUnit assertions. It
|
||||
provides more readable assertion chains and useful failure messages.
|
||||
|
||||
### Test asynchronous lifecycles
|
||||
|
||||
Camera initialization is asynchronous. Use `IdlingResource` to ensure
|
||||
your test waits for the `UseCase` to be bound before asserting.
|
||||
|
||||
To test asynchronous lifecycles, use the following pattern:
|
||||
|
||||
1. Wrap the `ProcessCameraProvider` initialization in a `CountDownLatch` or `IdlingResource`.
|
||||
2. Assert only after the `cameraControl` instance is non-null.
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Testing pitfalls
|
||||
|
||||
- **Resource leaks** : To prevent "Camera in Use" errors, in your `@After` block, call `cameraProvider.unbindAll`.
|
||||
- **The "flaky initializer"**: Camera tests often fail on CI because the "Virtual Camera" takes too long to warm up. Use a sufficient explicit timeout for the first initialization.
|
||||
- **Permission blockers** : To bypass the system permission dialogs, in your Espresso tests, use `GrantPermissionRule`.
|
||||
- **Resolution mismatch** : Tests on emulators often default to 640x480. Ensure your `ResolutionSelector` handles this low-res fallback correctly.
|
||||
85
.agents/skills/camerax/references/thermals.md
Normal file
85
.agents/skills/camerax/references/thermals.md
Normal file
@@ -0,0 +1,85 @@
|
||||
Camera operations are among the most power-intensive tasks on mobile devices.
|
||||
Without proactive management, the system throttle hardware, drop frames, or
|
||||
force-close the camera app.
|
||||
|
||||
## The thermal management strategy
|
||||
|
||||
| Priority | Strategy | Implementation |
|
||||
|---|---|---|
|
||||
| **1. Inform** | Use case hints | Provide `StreamUseCase` to allow the OS to optimize hardware. |
|
||||
| **2. Monitor** | Thermal state listener | Observe `PowerManager.addThermalStatusListener`. |
|
||||
| **3. Act** | Graceful degradation | Dynamically reduce FPS, resolution, or disable demanding effects such as HDR. |
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Follow the implementation guide
|
||||
|
||||
### Stream use case optimization
|
||||
|
||||
Android 13 (API level 33) introduced `StreamUseCase`. This is the **single most
|
||||
effective** way to tell the hardware how to balance quality versus power.
|
||||
|
||||
|
||||
```kotlin
|
||||
// In CameraX: Set the hint on your Use Case
|
||||
val preview = Preview.Builder()
|
||||
.setTargetName("Preview")
|
||||
.apply {
|
||||
Camera2Interop.Extender(this).setStreamUseCase(
|
||||
CameraMetadata.SCALER_AVAILABLE_STREAM_USE_CASES_VIDEO_CALL.toLong()
|
||||
)
|
||||
}
|
||||
.build()
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
Review the following key use cases for stream optimization:
|
||||
|
||||
- `PREVIEW`: This option is the default and provides a balanced configuration.
|
||||
- `STILL_CAPTURE`: This option provides high-quality capture for short bursts.
|
||||
- `VIDEO_RECORD`: This option maintains sustained power and is optimized for encoding.
|
||||
- `VIDEO_CALL`: This option minimizes power consumption for long-duration sessions.
|
||||
|
||||
### Monitor thermal status
|
||||
|
||||
Don't wait for a crash. Monitor the `PowerManager` status and react before
|
||||
`THERMAL_STATUS_CRITICAL`.
|
||||
|
||||
|
||||
```kotlin
|
||||
val powerManager = context.getSystemService(Context.POWER_SERVICE) as PowerManager
|
||||
powerManager.addThermalStatusListener { status ->
|
||||
when (status) {
|
||||
PowerManager.THERMAL_STATUS_MODERATE -> {
|
||||
// Signal to UI: "Device is warming up"
|
||||
}
|
||||
PowerManager.THERMAL_STATUS_SEVERE -> {
|
||||
// ACTION: Reduce Frame Rate from 60fps to 30fps
|
||||
// ACTION: Disable HDR or High-Quality Post-processing
|
||||
}
|
||||
PowerManager.THERMAL_STATUS_CRITICAL -> {
|
||||
// ACTION: Close the camera session to prevent hardware damage
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### Graceful degradation tiers
|
||||
|
||||
| Tier | Action | User impact |
|
||||
|---|---|---|
|
||||
| **Mild** | Stop background analysis using ML Kit | Minimal |
|
||||
| **Moderate** | Cap frame rate to 30 FPS | Noticeable but smooth |
|
||||
| **Severe** | Drop resolution from 1080p to 720p | Significant visual change |
|
||||
| **Critical** | Shut down the session | App unusable in safe mode |
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Thermal pitfalls
|
||||
|
||||
- **The "double work" bug**: Don't run two high-resolution streams---such as a preview and a video capture stream---at different aspect ratios unless necessary. This forces the image signal processor (ISP) to perform double the scaling work, which generates excessive heat.
|
||||
- **Surface overload** : Don't use multi-step `SurfaceProcessor` or `Media3Effect` chains during `THERMAL_STATUS_SEVERE`.
|
||||
- **Flash heat** : Flash or torch usage generates high thermal load. Proactively disable the flash if thermal status is `SEVERE`.
|
||||
76
.agents/skills/camerax/references/wear-os.md
Normal file
76
.agents/skills/camerax/references/wear-os.md
Normal file
@@ -0,0 +1,76 @@
|
||||
Developing camera features for Wear OS is rarely about the watch's own lens, if
|
||||
it even has one. It's almost always about creating a **Remote Viewfinder** to
|
||||
control the phone's camera.
|
||||
|
||||
## The Wear OS remote mindset
|
||||
|
||||
| Feature | Phone camera app | Wear OS remote app |
|
||||
|---|---|---|
|
||||
| **Screen** | Rectangular (Large) | **Circular and less than 2 inches in size** |
|
||||
| **Connectivity** | Local Hardware | **Bluetooth or Wi-Fi data layer API** |
|
||||
| **Latency** | Direct and less than 20 ms | **Networked, 100 ms to 500 ms** |
|
||||
| **Interaction** | Multi-touch gestures | **Rotary input or single taps** |
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Follow the implementation guide
|
||||
|
||||
### The circular UI challenge
|
||||
|
||||
Wear OS devices are often round. Standard rectangular layouts clip corner
|
||||
buttons.
|
||||
|
||||
Follow these blueprint recommendations:
|
||||
|
||||
- Use `Horologist` or `Wear Compose` libraries.
|
||||
- Use `ScalingLazyColumn` for lists so that items stay within the "safe zone" of the circular display.
|
||||
- **Preview scaling**: Crop the center of the rectangular phone viewfinder to fit the circular watch screen.
|
||||
|
||||
### Stream the viewfinder
|
||||
|
||||
You can't send a raw 60 fps stream over Bluetooth. compress and throttle.
|
||||
|
||||
|
||||
```kotlin
|
||||
// Example: Sending a viewfinder frame to the watch
|
||||
val bitmap = previewView.bitmap // Capture current frame
|
||||
if (bitmap != null) {
|
||||
val compressed = compressToJpeg(bitmap, quality = 50)
|
||||
val request = PutDataMapRequest.create("/camera/preview").apply {
|
||||
dataMap.putAsset("image", Asset.createFromBytes(compressed))
|
||||
}
|
||||
Wearable.getDataClient(context).putDataItem(request.asPutDataRequest())
|
||||
}
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
**Optimization** : Cap the watch preview at **10-15 fps** to preserve battery and
|
||||
bandwidth.
|
||||
|
||||
### Remote triggers and syncing
|
||||
|
||||
Use the `MessageClient` for low-latency commands like "Take Photo" or "Switch
|
||||
Camera."
|
||||
|
||||
|
||||
```kotlin
|
||||
// Watch sends a trigger to the phone
|
||||
Wearable.getMessageClient(context).sendMessage(nodeId, "/camera/capture", null)
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### Rotary input support
|
||||
|
||||
On devices that support it, use the physical crown, Rotary Input, to control
|
||||
**Zoom** or **Exposure**.
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Wear OS pitfalls
|
||||
|
||||
- **Corner clipping** : Placing a **Close** button in the top-right corner of a square layout makes it impossible to tap the button on a round watch display.
|
||||
- **Battery drain**: Sustained Bluetooth data transfer, viewfinder sync, drains watch battery. Proactively close the remote app if the phone screen is turned off.
|
||||
- **Node discovery** : The phone is possibly connected to multiple "Nodes" (watches, earbuds, or tablets). Ensure your `CapabilityClient` filters for the specific `camera_remote_host` capability.
|
||||
- **Disconnect handling**: If the watch disconnects, the phone camera must stop its high-power preview to save energy.
|
||||
54
.agents/skills/camerax/references/xr.md
Normal file
54
.agents/skills/camerax/references/xr.md
Normal file
@@ -0,0 +1,54 @@
|
||||
Developing camera features for XR devices, headsets, and AR glasses requires a
|
||||
shift from 2D pixel-pushing to 3D spatial awareness.
|
||||
|
||||
## Understand the XR development mindset
|
||||
|
||||
| Concept | Mobile focus | **XR focus** |
|
||||
|---|---|---|
|
||||
| **Input** | Raw camera stream | **Spatial tracking (visual-inertial odometry (VIO) or simultaneous localization and mapping (SLAM))** |
|
||||
| **Output** | Screen viewfinder | **Stereo passthrough and occlusion** |
|
||||
| **Constraint** | Battery life | **Motion-to-photon latency (less than 20 ms)** |
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## Follow the implementation guide
|
||||
|
||||
### API selection
|
||||
|
||||
On XR devices, standard `CameraX` implementations are often restricted or
|
||||
insufficient. Always use spatial software development kits (SDKs):
|
||||
|
||||
- **ARCore**: Use ARCore for plane detection, depth sensing, and motion tracking.
|
||||
- **OpenXR**: Use OpenXR as the cross-platform standard for VR and AR rendering and input.
|
||||
- **OEM SDKs**: Use manufacturer-specific libraries for hardware-accelerated passthrough.
|
||||
|
||||
### Handle spatial passthrough
|
||||
|
||||
Unlike a 2D viewport, XR passthrough is often system-managed.
|
||||
|
||||
**\[Key requirement\] Frame synchronization**: Synchronize your application's
|
||||
frame clock with the headset's head-mounted display (HMD) pose.
|
||||
|
||||
```kotlin
|
||||
// Example: Querying the spatial pose for the current camera frame
|
||||
val headPose = xrSession.getHeadPose(frameTime)
|
||||
val projectionMatrix = headPose.getProjectionMatrix(eyeIndex)
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
### Manage depth and occlusion
|
||||
|
||||
Digital content must respect real-world depth to ensure accurate occlusion.
|
||||
|
||||
- **Depth map** : Access raw depth data using `ARCore` or `SurfaceProcessor` to create an occlusion mask.
|
||||
- **Hardware buffers** : Use `HardwareBuffer` to share camera frames directly with the GPU without CPU-side copies to minimize latency.
|
||||
|
||||
*** ** * ** ***
|
||||
|
||||
## XR pitfalls
|
||||
|
||||
- **The nausea limit**: Any processing that delays the viewfinder by more than 20 ms causes user sickness. Don't perform image processing on the main thread.
|
||||
- **Privacy restrictions** : Some XR devices return a black frame if you attempt to record the "Passthrough" layer. Check `Session.isRecordingSupported`.
|
||||
- **Field of view (FOV)**: The camera FOV possibly doesn't match the display FOV. Use the SDK's projection matrixes instead of calculating aspect ratios manually.
|
||||
- **Front buffer rendering** : If the device supports it, use `FrontBufferRenderer` for real-time overlays to bypass standard double-buffering latency.
|
||||
Reference in New Issue
Block a user