tt/VibeCoding
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

skill: add Android skills

Browse Source
This commit is contained in:
tongtongstudio
2026-07-14 10:34:54 +08:00
parent e260c02629
commit ab82fb7cf0
182 changed files with 62992 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

83
.agents/skills/android-cli/references/interact.md Normal file
View File

@@ -0,0 +1,83 @@
# Tools
Run `android layout --help` and `android screen --help`.
## UI Dump
`android layout` returns a flat JSON list of the UI elements on screen.
`android layout --diff` returns a flat JSON list of the UI elements that have changed since the last call to `layout` or `layout --diff`
Each JSON object represents a UI element in the Android app. The following properties may be present:
- `text` - any literal text the element contains
- `resourceId` - the Android resource id used to refer to the element
- `contentDesc` - a description of a UI element for use by accessibility tools
- `interactions` - the set of user interactions the element supports. May contain one or more of: `checkable`, `clickable`, `focusable`, `scrollable`, `long-clickable`, `password`
- `state` - the set of states the element is in. May contain one or more of `checked`, `focused`, `selected`
- `bounds` - the screen coordinates of the bounding rectangle of the element, in the format `[min X,min Y][max X, max Y]`
- `center` - the screen coordinates of the center of the element, in the format `[x,y]`
- `off-screen` - if true, the element is in the UI hierarchy but not visible; it may require scrolling to view.
Use `layout` as a primary means of examining an Android app. Use `layout --diff` to focus on changes and to keep your context small.
Example: When entering digits into a calculator, use `layout --diff` to output only the digit readout element.
`layout` may fail due to the app displaying a WebView or animation; in these cases, use `android screen --annotate` to inspect the app.
This failure will likely resolve after navigating away from the current screen.
## Screenshot
`android screen capture -o <file path>` saves a PNG of the current device screen to `<file path>`
Use `screen capture` as a secondary means of examining an Android app
Examples:
- Understanding the content of an on-screen image
- Looking at a `WebView` (web content does not always appear in the ui dump)
- Trying to find a UI element by its visual appearance
**IMPORTANT**: Always *VISUALLY* examine the PNG image returned from `android screen` BEFORE doing anything else.
## Annotated Screenshot
`android screen capture --annotate -o <file path>`
`android screen resolve --screen <path> --string <string>`
The `--annotate` command adds numerical labels and bounding boxes around UI elements. Use this command to locate UI elements that cannot
be located in the `layout` output.
**IMPORTANT**: When using `android screen --annotate`, always *VISUALLY* examine the resulting PNG file.
To refer to these labels in input commands, use `screen resolve` to convert labels into coordinates:
`android screen resolve --screen <file path> --string "#3"` returns `<x coord of region 3> <y coord of region 3>`
To save turns, you can combine shell commands:
`adb shell input $(android screen resolve --screen screen.png --string "tap #34")`
This command taps on region #34 from `screen.png`
## Input
Use `adb shell input` for interacting with Android devices.
Refer to the `"interactions"` property of an element for what interactions can be performed on a particular element.
Interact with UI elements with their `center` coordinate or their `bounds` coordinates:
```json
{
"key": -248568265,
"class": "android.widget.Button",
"bounds": "[138,9][167,38]",
"center": "[152,23]"
}
```
To tap on this button, you would execute `adb shell input tap 152 23`. This taps the center.
```json
{
"key": 12487234,
"class": "com.example.ui.ScrollableList",
"bounds": "[100,200][400,600]",
"center": "[250,400]"
}
```
To scroll down on this list, you would execute `adb shell input swipe 250 400 600 500`. This swipes from the center to the bottom over 500ms.
# Android Interaction Rules
1. Always ensure text input fields have `"focused"` in their `"state"` list before entering text
2. If an element has `"scrollable"` in its `"interactions"` list, try scrolling it when looking for missing UI elements
2. Always scroll slowly when executing scroll inputs. The 5th argument to `adb shell input swipe` controls scroll duration.
3. Content may take time to load; if a `layout` is missing information after you take an action, wait a few seconds, then perform `layout --diff` to see if anything changes.

97
.agents/skills/android-cli/references/journeys.md Normal file
View File

@@ -0,0 +1,97 @@
A journey is an XML-specified test of an Android app's behavior. It consists of a list of `<action>` elements. For example:
```xml
<journey name="My Journey">
<description>
A sample journey to illustrate the format
</description>
<actions>
<action>
Tap the "Home" icon
</action>
<action>
Verify that the app is on its Home screen
</action>
</actions>
</journey>
```
Evaluate a journey by proceeding through the `<actions>` list in sequential order. Evaluate each `<action>` block individually.
A journey succeeds if all elements in the `<actions>` list succeed.
A journey is a test case for an app. The journey XML is the source of truth; if the app disagrees with the journey, the app has failed.
Additionally, if the app exits, crashes, or freezes, journey evaluation stops and the journey fails.
**IMPORTANT** - Execute each step EXACTLY as written, and independently of other steps! If an action says to `"tap the first search result"`,
you MUST find the search results and tap the first one. Do this even if you believe you know the intent behind the action.
## Taking Actions
Some `<action>` elements specify UI interactions to perform on the running Android app. Perform the interaction and verify that the app does
not crash or behave in an unexpected manner. This is the *only* verification you should perform for an `<action>`.
If the interaction cannot be performed as specified, the journey fails.
Example:
```<action>Click the red button</action>```
If you determine a red button is not present in the UI, the journey fails.
If the text of an `<action>` specifies a list of actions, break it into sub-actions and evaluate them individually:
Example:
```<action>Search for soda and add the first result to the cart</action>```
This should be evaluated as:
```
<action>Search for soda</action>
<action>Add the first result to the cart</action>
```
If an `<action>` contains something that is not a specification for a UI interaction, alert the user that the journey is malformed and exit
early, specifying the error in question.
## Verifying Expectations
`<action>` elements that begin with "check" or "verify" specify expectations for the current state of the Android app. Determine the current
state of the app and check if the expectations are met.
Determine the current state of the app by inspecting the current screen of the device without interacting with it.
Example:
```<action>Check if "Switch 2" is visible on the screen</action>```
This requires only inspecting the current screen, not scrolling or interacting. If "Switch 2" is not currently visible, the action fails.
If the expectations are not met, mark the `<action>` as a failure and the journey evaluation ends. A single `<action>` may contain
multiple expectations.
Example:
```<action>Verify that the app is on the Home screen, the Home icon is blue, and the temperature is displayed</action>```
This `<action>` fails if ANY of the following are false:
- The app is on the Home screen
- There is a Home icon, and it is blue
- A temperature is displayed
## Handling failure
When running a journey, evaluate it as a test. Failure is acceptable, and often expected. Proper reporting of failures is the priority.
Keep debugging and troubleshooting to a minimum; assume that tools are showing you the correct output every time. The goal is to determine
if the *current* Android app can correctly handle the *current* steps outlined in the journey. Suggestions for bug fixes, clarification, or
other improvements should be kept to journey evaluation summary at the end.
## Summarizing
For each `<action>` you evaluated, output JSON describing the results.
```
{
"journey:", The name of the journey
"results:" [
{
// A string containing the full text of the <action>
"action": "Click the blue button,
// "PASSED" if the instruction was evaluated, "FAILED" if the instruction could not be evaluated, or "SKIPPED" if journey evaluation ended early because an instruction failed
"status": "PASSED",
// A list of the ADB commands executed while evaluating the instruction,
"commands": [ "adb input swipe 490 200 500 500 500", "adb input tap 45 920" ],
// Failure reasons, feedback, or other useful information
"comment": "The journey step doesn't specify that the button requires scrolling to see",
},
{
"action": "The home screen is shown",
"status": "FAILED",
"comment": "The settings page was shown",
},
]
}
```