skill: add Android skills

This commit is contained in:
2026-07-14 10:34:54 +08:00
parent e260c02629
commit ab82fb7cf0
182 changed files with 62992 additions and 0 deletions

View File

@@ -0,0 +1,198 @@
## Remove `Camera1` implementation
1. Delete all `android.hardware.Camera` instances.
2. Delete `SurfaceView` and `SurfaceHolder.Callback` implementations `surfaceCreated`, `surfaceChanged`, and `surfaceDestroyed`.
3. Remove custom lifecycle handling that opens or releases the camera in `onResume` or `onPause`.
4. Remove manual matrix calculations for orientation.
## Initialize `ProcessCameraProvider`
Request the `ProcessCameraProvider` and bind use cases to the Activity or
Fragment lifecycle.
```kotlin
val context = LocalContext.current
val lifecycleOwner = LocalLifecycleOwner.current
LaunchedEffect(context, lifecycleOwner) {
val cameraProvider = ProcessCameraProvider.getInstance(context).await()
val cameraSelector = CameraSelector.Builder()
.requireLensFacing(CameraSelector.LENS_FACING_BACK)
.build()
val preview = Preview.Builder().build()
val imageCapture = ImageCapture.Builder()
.setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY)
.build()
cameraProvider.unbindAll() // Unbind before rebinding
val camera = cameraProvider.bindToLifecycle(
lifecycleOwner,
cameraSelector,
preview,
imageCapture
)
val cameraControl = camera.cameraControl
}
```
<br />
## Implement the preview and tap-to-focus
Choose exactly one of the following patterns based on the app's UI toolkit:
### Option A: For Android Views
Use `androidx.camera.view.PreviewView`.
1. **Set up preview**:
```kotlin
preview.setSurfaceProvider(previewView.surfaceProvider)
```
<br />
2. **Handle tap-to-focus**:
```kotlin
val factory = previewView.meteringPointFactory
val point = factory.createPoint(x, y) // x, y from touch event
val action = FocusMeteringAction.Builder(point, FocusMeteringAction.FLAG_AF).build()
cameraControl?.startFocusAndMetering(action)
```
<br />
### Option B: For Jetpack Compose
Use `androidx.camera.compose.CameraXViewfinder`.
1. **Set up preview and SurfaceRequest**:
```kotlin
var surfaceRequest by remember { mutableStateOf<SurfaceRequest?>(null) }
val preview = remember {
Preview.Builder().build().apply {
setSurfaceProvider { request -> surfaceRequest = request }
}
}
```
<br />
2. **Render viewfinder**:
```kotlin
surfaceRequest?.let { request ->
CameraXViewfinder(
surfaceRequest = request,
coordinateTransformer = coordinateTransformer,
modifier = Modifier
)
}
```
<br />
3. **Handle tap-to-focus in Compose**:
```kotlin
// Inside your tap gesture handler...
val surfaceCoords = with(coordinateTransformer) { offset.transform() }
val factory = SurfaceOrientedMeteringPointFactory(
request.resolution.width.toFloat(),
request.resolution.height.toFloat()
)
val point = factory.createPoint(surfaceCoords.x, surfaceCoords.y)
val action = FocusMeteringAction.Builder(point, FocusMeteringAction.FLAG_AF).build()
cameraControl?.startFocusAndMetering(action)
```
<br />
4. **Update target rotation for Compose**:
```kotlin
LaunchedEffect(configuration) {
if (!view.isInEditMode) {
val rotation = view.display?.rotation ?: Surface.ROTATION_0
imageCapture.targetRotation = rotation
preview.targetRotation = rotation
}
}
```
<br />
## Capture a photo
Use the `ImageCapture` use case to take the picture. The `ImageProxy` handles
rotation directly.
```kotlin
imageCapture.takePicture(
cameraExecutor,
object : ImageCapture.OnImageCapturedCallback() {
override fun onCaptureSuccess(image: ImageProxy) {
val buffer = image.planes[0].buffer
val bytes = ByteArray(buffer.remaining())
buffer.get(bytes)
val bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
// Adjust rotation natively via ImageProxy
val matrix = Matrix()
matrix.postRotate(image.imageInfo.rotationDegrees.toFloat())
if (lensFacing == CameraSelector.LENS_FACING_FRONT) {
matrix.postScale(-1f, 1f) // Mirror for front camera
}
val rotatedBitmap = Bitmap.createBitmap(
bitmap, 0, 0, bitmap.width, bitmap.height, matrix, true
)
// MUST close proxy
image.close()
}
override fun onError(exception: ImageCaptureException) {
Log.e("CameraX", "Capture failed: ${exception.message}", exception)
}
}
)
```
<br />
## Switch cameras
To flip between front and rear cameras, change the `CameraSelector` and
retrigger the `ProcessCameraProvider` logic.
```kotlin
lensFacing = if (lensFacing == CameraSelector.LENS_FACING_BACK) {
CameraSelector.LENS_FACING_FRONT
} else {
CameraSelector.LENS_FACING_BACK
}
```
<br />
## Follow constraints
- **Don't manage the camera lifecycle manually** : Bind the camera to a `LifecycleOwner` through the `ProcessCameraProvider`. Avoid manual camera open or close logic in `onResume` or `onPause`.
- **Don't calculate focus matrices manually** : `MeteringPointFactory` handles coordinate transformations, including device rotation offsets. Avoid custom matrix implementations.
- **Don't forget to close the `ImageProxy`** : Remember to invoke `image.close()` in the capture callback. Skipping this call locks the capture pipeline and interrupts subsequent photos.
- **Don't wrap `PreviewView` in `AndroidView` for Compose code** : For Compose UI layouts, use `CameraXViewfinder`. Compiling `PreviewView` in an `AndroidView` is an earlier fallback option that introduces resizing issues.

View File

@@ -0,0 +1,193 @@
Camera2 offers granular control but introduces boilerplate: managing
`CameraDevice` states, `CameraCaptureSession` lifecycles, background threads,
`HandlerThread`, and manual orientation calculations.
CameraX simplifies this by binding high-level `UseCase`s such as `Preview`,
`ImageCapture`, and `ImageAnalysis` directly to Android lifecycles, handling
thread management and device-specific workarounds automatically.
*** ** * ** ***
## Remove Camera2 boilerplate
Migrating to CameraX removes manual setup code:
1. **Delete manual thread handling** : Remove `HandlerThread`, `Handler`, and executors dedicated to camera background tasks. CameraX manages its own threads.
2. **Remove session and device callbacks** : Delete `CameraDevice.StateCallback` and `CameraCaptureSession.StateCallback` implementations.
3. **Remove surface management** : Remove manual routing of `Surface` objects from `TextureView` or `SurfaceView` to the camera device.
4. **Remove manual orientation logic** : Delete calculations involving `CameraCharacteristics.SENSOR_ORIENTATION` and display rotation for adjusting preview and capture orientation.
*** ** * ** ***
## Initialize `ProcessCameraProvider`
Request the `ProcessCameraProvider` and bind your use cases to the
`LifecycleOwner` activity or fragment. This replaces the
`CameraManager.openCamera` flow.
```kotlin
val context = LocalContext.current
val lifecycleOwner = LocalLifecycleOwner.current
LaunchedEffect(context, lifecycleOwner) {
val cameraProvider = ProcessCameraProvider.getInstance(context).await()
val cameraSelector = CameraSelector.Builder()
.requireLensFacing(CameraSelector.LENS_FACING_BACK)
.build()
val preview = Preview.Builder().build()
val imageCapture = ImageCapture.Builder()
.setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY)
.build()
// ImageAnalysis is common when migrating from Camera2 ImageReader
val imageAnalysis = ImageAnalysis.Builder()
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.build()
cameraProvider.unbindAll()
val camera = cameraProvider.bindToLifecycle(
lifecycleOwner,
cameraSelector,
preview,
imageCapture,
imageAnalysis
)
}
```
<br />
*** ** * ** ***
## Implement the preview and tap-to-focus
CameraX handles surface configuration automatically. Choose based on your UI
toolkit:
### Option A: For Android Views
Use `androidx.camera.view.PreviewView` in your layout, and bind it to the
`Preview` use case.
```kotlin
preview.setSurfaceProvider(previewView.surfaceProvider)
```
<br />
### Option B: For Jetpack Compose
Use `androidx.camera.compose.CameraXViewfinder`.
```kotlin
var surfaceRequest by remember { mutableStateOf<SurfaceRequest?>(null) }
val preview = remember {
Preview.Builder().build().apply {
setSurfaceProvider { request -> surfaceRequest = request }
}
}
```
<br />
*** ** * ** ***
## Capture a photo
Replace `ImageReader` capture flows and
`CaptureRequest.Builder.TEMPLATE_STILL_CAPTURE` with the `ImageCapture` use
case. CameraX handles the rotation natively using the returned `ImageProxy`.
```kotlin
imageCapture.takePicture(
cameraExecutor,
object : ImageCapture.OnImageCapturedCallback() {
override fun onCaptureSuccess(image: ImageProxy) {
val buffer = image.planes[0].buffer
val bytes = ByteArray(buffer.remaining())
buffer.get(bytes)
val bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
if (bitmap != null) {
val matrix = Matrix()
matrix.postRotate(image.imageInfo.rotationDegrees.toFloat())
if (lensFacing == CameraSelector.LENS_FACING_FRONT) {
matrix.postScale(-1f, 1f)
}
val rotatedBitmap = Bitmap.createBitmap(
bitmap, 0, 0, bitmap.width, bitmap.height, matrix, true
)
}
// MUST close proxy
image.close()
}
override fun onError(exception: ImageCaptureException) {
Log.e("CameraX", "Capture failed: ${exception.message}", exception)
}
}
)
```
<br />
*** ** * ** ***
## Implement image analysis
If you were using `ImageReader` in Camera2 to access raw frames, e.g., for QR
scanning or ML,, replace it with the CameraX `ImageAnalysis` use case.
```kotlin
imageAnalysis.setAnalyzer(cameraExecutor) { imageProxy ->
try {
val rotationDegrees = imageProxy.imageInfo.rotationDegrees
// Process image here (e.g., run object detection)
// ...
} finally {
// MUST close the imageProxy to avoid blocking the pipeline
imageProxy.close()
}
}
```
<br />
*** ** * ** ***
## Use Camera2 interop
If your app requires specific Camera2 configuration options, such as custom
exposure modes or flash settings, that aren't exposed directly in CameraX,
use `Camera2Interop` to apply them to your CameraX use cases.
```kotlin
// Use Camera2Interop to set Camera2-specific capture options
val extender = Camera2Interop.Extender(imageCaptureBuilder)
extender.setCaptureRequestOption(
CaptureRequest.CONTROL_AE_MODE,
CaptureRequest.CONTROL_AE_MODE_OFF
).setCaptureRequestOption(
CaptureRequest.FLASH_MODE,
CaptureRequest.FLASH_MODE_TORCH
)
```
<br />
*** ** * ** ***
## Follow constraints
- **Don't open the CameraDevice manually** : Let CameraX manage device opening and closing using `bindToLifecycle`.
- **Don't forget to close `ImageProxy`** : In both `ImageCapture` and `ImageAnalysis` callbacks, you **must** call `image.close()`. Failure to do so will block the camera pipeline.
- **Don't block the ImageAnalysis thread** : The analyzer runs frame-by-frame. If you need to perform heavy computations, offload them to a background worker and close the `ImageProxy` as soon as possible.

View File

@@ -0,0 +1,58 @@
Complex camera features often fail due to "Agent Stall" or timeouts when
attempted in a single turn. Use these blueprints to break tasks into manageable
phases.
## Manual controls
### Phase one: ViewModel and state
1. Define a `ManualSettings` data class.
2. Add a `MutableStateFlow<ManualSettings>` to your `CameraViewModel`.
3. Implement the Jetpack Compose UI with sliders and switches to update this flow.
### Phase two: Controller wiring
1. In `CameraController.kt`, create a new function `updateManualSettings(settings: ManualSettings)`.
2. Map these settings into the `CameraSystem` layer.
### Phase three: CameraX `Camera2Interop` wiring
1. In `CameraSession.kt`, use the `Camera2Interop.Extender` utility to access Camera2 capture request keys.
2. Apply the hardware keys:
- `CaptureRequest.SENSOR_SENSITIVITY` to set ISO sensitivity.
- `CaptureRequest.SENSOR_EXPOSURE_TIME` to set exposure time.
- `CaptureRequest.LENS_FOCUS_DISTANCE` to set focus distance.
3. **Critical** : If manual exposure is active, set `CaptureRequest.CONTROL_AE_MODE` to `CameraMetadata.CONTROL_AE_MODE_OFF`. ---
## RAW and JPEG capture
### Phase one: Output configuration
1. Verify device support for RAW capture using `CameraInfo`.
2. Configure `ImageCapture.Builder` with `OUTPUT_FORMAT_RAW_JPEG` or `OUTPUT_FORMAT_RAW`.
### Phase two: Implementation
1. Provide `ImageCapture.OutputFileOptions` for the target storage locations.
2. Invoke `takePicture`. CameraX internally manages `DngCreator` to wrap RAW data with the required `CameraCharacteristics` and `CaptureResult` metadata.
*** ** * ** ***
## Apply image effects
### Phase one: Effects selection
1. Use the `androidx.media3:media3-effect` dependency.
2. Use `RgbFilter` or `HslAdjustment` for standard color grading.
### Phase two: Application
1. Configure `Composition.Builder` or `MediaItem.Builder` with the list of effects.
2. Inject the list of effects into the CameraX `Recorder` or `Preview` using the `setEffects` method.
*** ** * ** ***
## Low-light capture
For guidance on Night Mode Extensions and Low Light Boost,
[low-light.md](https://developer.android.com/agents/skills/camera/camerax/references/low-light).

View File

@@ -0,0 +1,83 @@
Foldable devices introduce unique challenges for camera applications, including
dynamic layout changes, multiple display orientations, and physical device
postures, such as tabletop and book modes.
## Manage fold states and postures
| State | Posture | User interaction | Implementation goal |
|---|---|---|---|
| `FLAT` | Standard | Full screen preview | Conventional mobile phone layout. |
| `HALF_OPENED` | Tabletop | Lower half for controls | Split-screen layout, viewfinder on top, controls on bottom. |
| `HALF_OPENED` | Book | Side-by-side | Viewfinder on one panel, gallery and controls on the other. |
*** ** * ** ***
## Follow the implementation guide
### Detect posture changes
Use the Jetpack WindowManager library to observe the device's hinge state and
fold layout.
```kotlin
lifecycleScope.launch {
lifecycle.repeatOnLifecycle(Lifecycle.State.STARTED) {
windowInfoTracker.windowLayoutInfo(activity)
.collect { layoutInfo ->
val displayFeature = layoutInfo.displayFeatures
.filterIsInstance<FoldingFeature>()
.firstOrNull()
updateCameraLayout(displayFeature)
}
}
}
```
<br />
### Handle tabletop mode
In Tabletop mode, horizontal fold, you should move the viewfinder to the top
half of the screen and the controls to the bottom half to prevent the user from
seeing a "bent" image.
- **Identify orientation:** Check `FoldingFeature.orientation`.
- **Calculate geometry:** Use `FoldingFeature.bounds` to identify the hinge's physical location on the screen.
- **Update UI:** Apply padding or constraints to move the `PreviewView` above the hinge.
### Coordinate mapping and `Viewport`
When the UI layout changes due to a fold, you **must** update the `Viewport` to
ensure that tap-to-focus and image capture coordinates remain accurate.
```kotlin
val viewport = ViewPort.Builder(Rational(viewfinder.width, viewfinder.height), display.rotation)
.setScaleType(ViewPort.FILL_CENTER)
.build()
val useCaseGroup = UseCaseGroup.Builder()
.addUseCase(preview)
.setViewPort(viewport)
.build()
```
<br />
### Rear display mode
Some foldables allow using the rear camera with the cover display while the
device is unfolded.
- **Verification:** If available through OEM SDKs or Android 14 (API level 34) or higher, check `DeviceState.REAR_DISPLAY_STATE`.
- **Logic:** Handle preview detachment and reattachment on different display surfaces with varying aspect ratios.
*** ** * ** ***
## Foldable pitfalls
- **Hinge distortion:** Don't span the camera preview across a hinge in `HALF_OPENED` state.
- **Physical orientation:** The camera sensor's physical orientation relative to the screen often changes when you fold or unfold the device. Always rely on `CameraInfo.getSensorRotationDegrees`.
- **Latency:** Rebinding `UseCase` objects during a fold event is expensive. Use the internal scaling of `PreviewView` before performing a full `bindToLifecycle` reconfiguration.

View File

@@ -0,0 +1,60 @@
Many Android APIs are designed with immutability in mind to prevent race
conditions in async environments. However, this often trips up developers used
to mutable builder patterns.
## Common immutable classes
The following classes use fluent APIs that **return a new instance**. You must
reassign the variable.
| Class | Methods that return a new instance | Result if not reassigned |
|---|---|---|
| `PendingRecording` | `withAudioEnabled`, `asPersistentRecording` | Audio isn't recorded. |
| `ImageCapture.Builder` | `setTargetRotation`, `setTargetResolution` | The output has the wrong orientation. |
| `Recorder.Builder` | `setQualitySelector`, `setExecutor` | The recording uses the default quality. |
| `Viewport.Builder` | `setScaleType`, `setLayoutDirection` | The viewfinder is stretched. |
## Use standard patterns
### CameraX video recording
To set up video recording, use the following code:
```kotlin
// WRONG
run {
val pending = recorder.prepareRecording(context, opts)
pending.withAudioEnabled() // This returns a new instance which is ignored
val active = pending.start(exec, listener)
}
// CORRECT
run {
val pending = recorder.prepareRecording(context, opts)
.withAudioEnabled() // Chaining works
val active = pending.start(exec, listener)
}
// ALSO CORRECT
run {
var pending = recorder.prepareRecording(context, opts)
pending = pending.withAudioEnabled() // Reassignment
val active = pending.start(exec, listener)
}
```
<br />
### Viewport construction
To set up the viewport, use the following code:
```kotlin
val viewport = ViewPort.Builder(Rational(width, height), displayRotation)
.setScaleType(ViewPort.FILL_CENTER)
.build()
```
<br />

View File

@@ -0,0 +1,124 @@
This guide covers implementing low-light features using **Night mode
extensions** and **Low Light Boost (LLB)**.
## Choosing the right tool
| Feature | Used for | Implementation | UX impact |
|---|---|---|---|
| **Night mode** | High-quality stills | `ExtensionsManager` | Possibly requires user to hold still for several seconds. |
| **LLB, AE mode** | Real-time preview or video | `Camera2Interop`, CameraX utility | Hardware drops the frame rate to increase brightness. |
| **LLB, Play services** | Real-time preview and video | `SurfaceProcessor` | Software-based brightening; maintains a higher frame rate. |
*** ** * ** ***
## Night mode extension
CameraX Extensions provide access to the device's built-in computational
photography pipeline.
### Basic setup
To set up the extension, initialize the extension manager:
```kotlin
// Use ListenableFuture.await() extension function for coroutine support
val extensionsManager = ExtensionsManager.getInstanceAsync(context, cameraProvider).await()
if (extensionsManager.isExtensionAvailable(cameraSelector, ExtensionMode.NIGHT)) {
val nightSelector = extensionsManager.getExtensionEnabledCameraSelector(
cameraSelector, ExtensionMode.NIGHT
)
cameraProvider.bindToLifecycle(lifecycleOwner, nightSelector, imageCapture, preview)
}
```
<br />
### Comprehensive features
- **Image postview** : Display a low-resolution image immediately while the multi-frame processing occurs.
```kotlin
val imageCapture = ImageCapture.Builder()
.setPostviewEnabled(true)
.build()
```
- **Extension strength** : Let users control the intensity of the night effect.
```kotlin
// Set the strength of the active extension (e.g. NIGHT mode intensity)
val extensionsManager = ExtensionsManager.getInstanceAsync(context, cameraProvider).await()
val extensionsControl = extensionsManager.getCameraExtensionsControl(camera.cameraControl)
extensionsControl?.setExtensionStrength(strength)
```
- **Capture progress** : Show a UI progress bar for long exposures.
```kotlin
// Use the suspend extension function for takePicture to avoid callback boilerplate
try {
val result = imageCapture.takePicture(outputOptions)
// Use result.savedUri or other fields
} catch (e: ImageCaptureException) {
// Handle capture failure
}
```
*** ** * ** ***
## Low-light boost
LLB is designed for preview and video streams where you prefer high frame rates.
### AE mode
The built-in CameraX way to prioritize brightness. It modifies the hardware's
auto-exposure algorithm.
- **Activation** : Use `CameraControl.enableLowLightBoostAsync`.
- **Implementation** :
```kotlin
// Enable Low Light Boost (LLB) natively in CameraX 1.4+
camera.cameraControl.enableLowLightBoostAsync(true)
```
- **Monitoring** : Observe `CameraInfo.lowLightBoostState` to track when the hardware actively applies the enhancement.
### Google Play services LLB
It's a multi-step implementation that uses a session-based `SurfaceProcessor`.
**Dependency** : `com.google.android.gms:play-services-camera-low-light-boost`
To implement Google Play services LLB, follow these core steps:
1. **Initialize client** : `val client = LowLightBoost.getClient`.
2. **Implement `SurfaceProcessor`** :
- **Manage session** : Call `client.createSession`.
- **Forward required metadata** : Observe the camera's `TotalCaptureResult` stream and forward every result to the session: `session.processCaptureResult`.
- **Provide surface** : Get the input surface from the session, `session.getCameraSurface`, and provide it to the camera's `SurfaceRequest`.
- **Lifecycle** : Release the session, `session.release`, when the processor is closed or the `SurfaceRequest` completes.
3. **Wire using `CameraEffect`** :
```kotlin
val effect = SimpleCameraEffect(
CameraEffect.PREVIEW or CameraEffect.VIDEO_CAPTURE,
executor,
llbSurfaceProcessor
) { throw it }
// Add to UseCaseGroup
val useCaseGroup = UseCaseGroup.Builder()
.addUseCase(preview)
.addUseCase(videoCapture)
.addEffect(effect)
.build()
```
4. **Scene detection** : Use `session.setSceneDetectorCallback` to receive `boostStrength` updates for real-time UI indicators.
*** ** * ** ***
## Implementation notes
- **Thread safety** : Always handle `ExtensionsManager` and `LowLightBoostClient` initialization asynchronously.
- **FPS trade-offs**: AE mode LLB often drops the frame rate significantly to increase brightness.
- **Compatibility** : Extensions, Night Mode, possibly conflict with `ConcurrentCamera`. Always verify support before binding.

View File

@@ -0,0 +1,75 @@
When you use ML Kit for features such as face mesh, object detection, or pose
detection, the most common failure point is the coordinate disparity between the
analysis image and the viewfinder UI.
## The mapping mindset
| Dimension | Analysis frame | Viewfinder UI on the screen |
|---|---|---|
| Resolution | Fixed, such as 640x480 | Dynamic, such as 1080x2400 |
| Rotation | 0° for the raw buffer | 90° or 270° in portrait or landscape mode |
| Origin | Top-left of buffer at (0,0) | Top-left of screen at (0,0) |
*** ** * ** ***
## Follow the implementation guide
### Coordinate transformation matrix
Android provides the `Viewport` and `UseCaseGroup` APIs to calculate the
transformation matrix automatically. **Don't** calculate aspect ratio scaling
manually.
```kotlin
val transform = previewView.viewPort?.let { viewPort ->
// Use CameraX's built-in coordinate mapper
viewPort.getTransformationMatrix(imageProxy.imageInfo.rotationDegrees)
}
```
<br />
### Handling the "double rotation" bug
ML Kit results, bounding boxes, are relative to the **rotated buffer**. If the
device is in portrait, the buffer is often 480x640, landscape, but the screen
is 1080x1920.
To map the coordinates, use the following workflow:
1. Query `imageProxy.imageInfo.rotationDegrees`.
2. Pass this rotation to the ML Kit `InputImage`.
3. Use the `MappingUtils.transformRect` method to map the result `Rect` to the screen.
### Face mesh and pose normalization
For high-precision spatial analysis, for example, "Is the user's hand at a
specific screen button?", use **normalized coordinates from 0.0 to 1.0**.
```kotlin
// Example: Converting a Pose landmark to a Screen Coordinate
val screenX = landmark.position.x / analysisWidth * screenWidth
val screenY = landmark.position.y / analysisHeight * screenHeight
```
<br />
**Warning** : Always account for **mirrored lenses** . If the `LENS_FACING_FRONT`
is used, you must flip the X-coordinate: `actualX = screenWidth - screenX`.
### Overlays and canvas clipping
Use a custom `GraphicOverlay` view on top of the `PreviewView`.
- **Buffer lock** : Ensure your `GraphicOverlay` clears its canvas every time a new `ImageAnalysis` frame is processed to prevent "ghosting" of bounding boxes.
*** ** * ** ***
## Spatial pitfalls
- **The "stretched box" bug** : Caused by assuming the Analysis Frame aspect ratio, 4:3, matches the screen aspect ratio, 21:9. Use `PreviewView.SCALE_TYPE_FILL_CENTER` and map coordinates accordingly.
- **Latency** : If ML processing exceeds 50 ms, the bounding box trails behind the user's face.
- **Fix** : Use `ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST` to avoid queuing stale frames.
- **Sensor vs. display rotation** : On some tablets, the sensor is mounted horizontally. Always query `display.rotation` and `cameraInfo.sensorRotationDegrees`.

View File

@@ -0,0 +1,42 @@
Always prefer these various abstractions over legacy Camera2 or early CameraX
implementations.
## Compare APIs
| Use case | Legacy or verbose way | Recommendation |
|---|---|---|
| QR or face scanning | `ImageAnalysis.Analyzer` and manual ByteBuffer math | **`MlKitAnalyzer`**, which automates coordinate mapping and multi-format support |
| Post-processing | Custom OpenGL shaders or `SurfaceTexture` | **`Media3Effect`** for composable, declarative effects |
| Dual camera | Manual binding of two UseCases | **`ConcurrentCamera`**, which provides built-in support in CameraX 1.3 and higher |
| High dynamic range | Manual bit-depth and profile config | **`DynamicRange`** , which uses `DYNAMIC_RANGE_HLG10` or `SDR` |
| Zoom and focus | `Camera2Interop` for CameraX-to-Camera2 mapping | **`CameraControl.setZoomRatio`** or **`setLinearZoom`** |
## Hardware awareness
Modern APIs abstract away the complexity of hardware diversity.
- **`CameraSelector`** : Use `DEFAULT_BACK_CAMERA` or `DEFAULT_FRONT_CAMERA` instead of hardcoding camera IDs. Use `filter` if you need specific lens capabilities.
- **Extensions** : Before enabling advanced modes, such as night, bokeh, and face retouch, use `ExtensionsManager` to query whether the device supports them.
- **Foldables** : Observe `Lifecycle` and `Viewport` updates to handle posture changes, like a half-opened posture, on foldable devices.
## Required dependencies
Add the following dependencies to your `libs.versions.toml` file:
# CameraX ML Kit
androidx-camera-mlkit-vision = { group = "androidx.camera", name =
"camera-mlkit-vision", version.ref = "camerax" }
# Media3 effects
androidx-media3-effect = { group = "androidx.media3", name = "media3-effect",
version.ref = "media3" }
# Camera extensions
androidx-camera-extensions = { group = "androidx.camera", name =
"camera-extensions", version.ref = "camerax" }
Refer to the official [CameraX Release Notes](https://developer.android.com/jetpack/androidx/releases/camera) for the
stable versions.

View File

@@ -0,0 +1,69 @@
Automated testing for camera features is notoriously difficult because you
can't easily mock physical hardware, lighting, or motion. This guide provides
patterns for reliable, hermetic camera tests.
## Develop a testing mindset
| Challenge | Conventional approach | Camera technical approach |
|---|---|---|
| **Frameworks** | `Mockito` or `MockK` | **Fakes over mocks** |
| **Assertions** | `assertEquals`, `assertTrue` | **Google Truth, `assertThat`** |
| **Environment** | Implied environments | **Explicit `@RunWith` annotations** |
| **Async operations** | `Thread.sleep` | **Explicit `timeoutMillis` or `IdlingResource`** |
*** ** * ** ***
## Follow the implementation guide
### Fakes over mocks
**Don't use Mockito.** Relying on mocks for complex, rapidly changing interfaces
like `ImageProxy` or `CameraInfo` makes tests brittle. Instead, build "Fake"
implementations that verify state rather than behavior.
```kotlin
// Create a Fake ImageProxy for ML Testing (Fakes over Mocks)
val fakeImage = FakeImageProxy(w = 640, h = 480)
// Feed the fake buffer into your analyzer
```
<br />
### Mock camera capabilities
Use `FakeAppConfig` from `androidx.camera:camera-testing` to simulate specific
hardware constraints in tests, such as a device without a flash.
```kotlin
// Use awaitInstance() extension function for coroutine-based provider retrieval
val cameraProvider = ProcessCameraProvider.awaitInstance(context)
```
<br />
### Use Truth assertions
Use Google Truth, `assertThat`, instead of standard JUnit assertions. It
provides more readable assertion chains and useful failure messages.
### Test asynchronous lifecycles
Camera initialization is asynchronous. Use `IdlingResource` to ensure
your test waits for the `UseCase` to be bound before asserting.
To test asynchronous lifecycles, use the following pattern:
1. Wrap the `ProcessCameraProvider` initialization in a `CountDownLatch` or `IdlingResource`.
2. Assert only after the `cameraControl` instance is non-null.
*** ** * ** ***
## Testing pitfalls
- **Resource leaks** : To prevent "Camera in Use" errors, in your `@After` block, call `cameraProvider.unbindAll`.
- **The "flaky initializer"**: Camera tests often fail on CI because the "Virtual Camera" takes too long to warm up. Use a sufficient explicit timeout for the first initialization.
- **Permission blockers** : To bypass the system permission dialogs, in your Espresso tests, use `GrantPermissionRule`.
- **Resolution mismatch** : Tests on emulators often default to 640x480. Ensure your `ResolutionSelector` handles this low-res fallback correctly.

View File

@@ -0,0 +1,85 @@
Camera operations are among the most power-intensive tasks on mobile devices.
Without proactive management, the system throttle hardware, drop frames, or
force-close the camera app.
## The thermal management strategy
| Priority | Strategy | Implementation |
|---|---|---|
| **1. Inform** | Use case hints | Provide `StreamUseCase` to allow the OS to optimize hardware. |
| **2. Monitor** | Thermal state listener | Observe `PowerManager.addThermalStatusListener`. |
| **3. Act** | Graceful degradation | Dynamically reduce FPS, resolution, or disable demanding effects such as HDR. |
*** ** * ** ***
## Follow the implementation guide
### Stream use case optimization
Android 13 (API level 33) introduced `StreamUseCase`. This is the **single most
effective** way to tell the hardware how to balance quality versus power.
```kotlin
// In CameraX: Set the hint on your Use Case
val preview = Preview.Builder()
.setTargetName("Preview")
.apply {
Camera2Interop.Extender(this).setStreamUseCase(
CameraMetadata.SCALER_AVAILABLE_STREAM_USE_CASES_VIDEO_CALL.toLong()
)
}
.build()
```
<br />
Review the following key use cases for stream optimization:
- `PREVIEW`: This option is the default and provides a balanced configuration.
- `STILL_CAPTURE`: This option provides high-quality capture for short bursts.
- `VIDEO_RECORD`: This option maintains sustained power and is optimized for encoding.
- `VIDEO_CALL`: This option minimizes power consumption for long-duration sessions.
### Monitor thermal status
Don't wait for a crash. Monitor the `PowerManager` status and react before
`THERMAL_STATUS_CRITICAL`.
```kotlin
val powerManager = context.getSystemService(Context.POWER_SERVICE) as PowerManager
powerManager.addThermalStatusListener { status ->
when (status) {
PowerManager.THERMAL_STATUS_MODERATE -> {
// Signal to UI: "Device is warming up"
}
PowerManager.THERMAL_STATUS_SEVERE -> {
// ACTION: Reduce Frame Rate from 60fps to 30fps
// ACTION: Disable HDR or High-Quality Post-processing
}
PowerManager.THERMAL_STATUS_CRITICAL -> {
// ACTION: Close the camera session to prevent hardware damage
}
}
}
```
<br />
### Graceful degradation tiers
| Tier | Action | User impact |
|---|---|---|
| **Mild** | Stop background analysis using ML Kit | Minimal |
| **Moderate** | Cap frame rate to 30 FPS | Noticeable but smooth |
| **Severe** | Drop resolution from 1080p to 720p | Significant visual change |
| **Critical** | Shut down the session | App unusable in safe mode |
*** ** * ** ***
## Thermal pitfalls
- **The "double work" bug**: Don't run two high-resolution streams---such as a preview and a video capture stream---at different aspect ratios unless necessary. This forces the image signal processor (ISP) to perform double the scaling work, which generates excessive heat.
- **Surface overload** : Don't use multi-step `SurfaceProcessor` or `Media3Effect` chains during `THERMAL_STATUS_SEVERE`.
- **Flash heat** : Flash or torch usage generates high thermal load. Proactively disable the flash if thermal status is `SEVERE`.

View File

@@ -0,0 +1,76 @@
Developing camera features for Wear OS is rarely about the watch's own lens, if
it even has one. It's almost always about creating a **Remote Viewfinder** to
control the phone's camera.
## The Wear OS remote mindset
| Feature | Phone camera app | Wear OS remote app |
|---|---|---|
| **Screen** | Rectangular (Large) | **Circular and less than 2 inches in size** |
| **Connectivity** | Local Hardware | **Bluetooth or Wi-Fi data layer API** |
| **Latency** | Direct and less than 20 ms | **Networked, 100 ms to 500 ms** |
| **Interaction** | Multi-touch gestures | **Rotary input or single taps** |
*** ** * ** ***
## Follow the implementation guide
### The circular UI challenge
Wear OS devices are often round. Standard rectangular layouts clip corner
buttons.
Follow these blueprint recommendations:
- Use `Horologist` or `Wear Compose` libraries.
- Use `ScalingLazyColumn` for lists so that items stay within the "safe zone" of the circular display.
- **Preview scaling**: Crop the center of the rectangular phone viewfinder to fit the circular watch screen.
### Stream the viewfinder
You can't send a raw 60 fps stream over Bluetooth. compress and throttle.
```kotlin
// Example: Sending a viewfinder frame to the watch
val bitmap = previewView.bitmap // Capture current frame
if (bitmap != null) {
val compressed = compressToJpeg(bitmap, quality = 50)
val request = PutDataMapRequest.create("/camera/preview").apply {
dataMap.putAsset("image", Asset.createFromBytes(compressed))
}
Wearable.getDataClient(context).putDataItem(request.asPutDataRequest())
}
```
<br />
**Optimization** : Cap the watch preview at **10-15 fps** to preserve battery and
bandwidth.
### Remote triggers and syncing
Use the `MessageClient` for low-latency commands like "Take Photo" or "Switch
Camera."
```kotlin
// Watch sends a trigger to the phone
Wearable.getMessageClient(context).sendMessage(nodeId, "/camera/capture", null)
```
<br />
### Rotary input support
On devices that support it, use the physical crown, Rotary Input, to control
**Zoom** or **Exposure**.
*** ** * ** ***
## Wear OS pitfalls
- **Corner clipping** : Placing a **Close** button in the top-right corner of a square layout makes it impossible to tap the button on a round watch display.
- **Battery drain**: Sustained Bluetooth data transfer, viewfinder sync, drains watch battery. Proactively close the remote app if the phone screen is turned off.
- **Node discovery** : The phone is possibly connected to multiple "Nodes" (watches, earbuds, or tablets). Ensure your `CapabilityClient` filters for the specific `camera_remote_host` capability.
- **Disconnect handling**: If the watch disconnects, the phone camera must stop its high-power preview to save energy.

View File

@@ -0,0 +1,54 @@
Developing camera features for XR devices, headsets, and AR glasses requires a
shift from 2D pixel-pushing to 3D spatial awareness.
## Understand the XR development mindset
| Concept | Mobile focus | **XR focus** |
|---|---|---|
| **Input** | Raw camera stream | **Spatial tracking (visual-inertial odometry (VIO) or simultaneous localization and mapping (SLAM))** |
| **Output** | Screen viewfinder | **Stereo passthrough and occlusion** |
| **Constraint** | Battery life | **Motion-to-photon latency (less than 20 ms)** |
*** ** * ** ***
## Follow the implementation guide
### API selection
On XR devices, standard `CameraX` implementations are often restricted or
insufficient. Always use spatial software development kits (SDKs):
- **ARCore**: Use ARCore for plane detection, depth sensing, and motion tracking.
- **OpenXR**: Use OpenXR as the cross-platform standard for VR and AR rendering and input.
- **OEM SDKs**: Use manufacturer-specific libraries for hardware-accelerated passthrough.
### Handle spatial passthrough
Unlike a 2D viewport, XR passthrough is often system-managed.
**\[Key requirement\] Frame synchronization**: Synchronize your application's
frame clock with the headset's head-mounted display (HMD) pose.
```kotlin
// Example: Querying the spatial pose for the current camera frame
val headPose = xrSession.getHeadPose(frameTime)
val projectionMatrix = headPose.getProjectionMatrix(eyeIndex)
```
<br />
### Manage depth and occlusion
Digital content must respect real-world depth to ensure accurate occlusion.
- **Depth map** : Access raw depth data using `ARCore` or `SurfaceProcessor` to create an occlusion mask.
- **Hardware buffers** : Use `HardwareBuffer` to share camera frames directly with the GPU without CPU-side copies to minimize latency.
*** ** * ** ***
## XR pitfalls
- **The nausea limit**: Any processing that delays the viewfinder by more than 20 ms causes user sickness. Don't perform image processing on the main thread.
- **Privacy restrictions** : Some XR devices return a black frame if you attempt to record the "Passthrough" layer. Check `Session.isRecordingSupported`.
- **Field of view (FOV)**: The camera FOV possibly doesn't match the display FOV. Use the SDK's projection matrixes instead of calculating aspect ratios manually.
- **Front buffer rendering** : If the device supports it, use `FrontBufferRenderer` for real-time overlays to bypass standard double-buffering latency.