233 lines
7.5 KiB
Markdown
233 lines
7.5 KiB
Markdown
# GUI Automation Tools
|
||
|
||
KDE desktop automation via D-Bus and accessibility interfaces. All tools use `call_tool`.
|
||
|
||
**Dependencies**: `gdbus` (always), `wmctrl`/`xdotool` (window ops), `ydotool` (Wayland input), `spectacle` (screenshots), `python3-gobject` + AT-SPI2 typelib (accessibility).
|
||
|
||
---
|
||
|
||
## gui_clipboard
|
||
|
||
Read/write the KDE clipboard via Klipper.
|
||
|
||
**Args**: `[command]` or `[command, value]`
|
||
|
||
Commands: `get`, `set`, `history`, `clear`
|
||
|
||
```
|
||
call_tool: calls=[{tool: "gui_clipboard", args: ["get"]}]
|
||
call_tool: calls=[{tool: "gui_clipboard", args: ["set", "hello world"]}]
|
||
call_tool: calls=[{tool: "gui_clipboard", args: ["history", "2"]}]
|
||
```
|
||
|
||
---
|
||
|
||
## gui_notify
|
||
|
||
Show desktop notifications with optional interactive action buttons. Waits for user response when actions are specified.
|
||
|
||
**Args**: `[summary]` or `[summary, body, --flags...]`
|
||
|
||
Flags: `--actions=label1,label2,...`, `--timeout=ms`, `--icon=name`, `--urgency=low|normal|critical`
|
||
|
||
```
|
||
call_tool: calls=[{tool: "gui_notify", args: ["Build complete", "All tests passed"]}]
|
||
call_tool: calls=[{tool: "gui_notify", args: ["Deploy?", "Ready to push to prod", "--actions=Yes,No", "--urgency=critical"]}]
|
||
```
|
||
|
||
**Returns**: For action notifications, prints the chosen label (e.g., `Yes`) or `dismissed`/`timeout`.
|
||
|
||
---
|
||
|
||
## gui_screenshot
|
||
|
||
Capture screenshots. Uses spectacle (KDE) with fallbacks to ImageMagick.
|
||
|
||
**Args**: `[mode]` with optional flags
|
||
|
||
Modes: `active` (default), `screen`, `area`, `window`
|
||
|
||
Flags: `--output=path`, `--screen=N`, `--title=substr`
|
||
|
||
```
|
||
call_tool: calls=[{tool: "gui_screenshot", args: ["active"]}]
|
||
call_tool: calls=[{tool: "gui_screenshot", args: ["screen", "--output=/tmp/shot.png"]}]
|
||
call_tool: calls=[{tool: "gui_screenshot", args: ["window", "--title=Firefox"]}]
|
||
```
|
||
|
||
**Returns**: output file path and size.
|
||
|
||
---
|
||
|
||
## gui_windows
|
||
|
||
Manage windows: list, focus, close, move, resize, virtual desktops.
|
||
|
||
**Args**: `[command, ...params]`
|
||
|
||
Commands:
|
||
- `list` — all windows with geometry and state
|
||
- `focus <title>` — raise and focus
|
||
- `close <title>` — close window
|
||
- `minimize <title>` / `maximize <title>`
|
||
- `move <title> <x> <y>` — reposition
|
||
- `resize <title> <w> <h>` — resize
|
||
- `desktops` — list virtual desktops
|
||
- `desktop <n>` — switch to desktop N
|
||
|
||
```
|
||
call_tool: calls=[{tool: "gui_windows", args: ["list"]}]
|
||
call_tool: calls=[{tool: "gui_windows", args: ["focus", "Kate"]}]
|
||
call_tool: calls=[{tool: "gui_windows", args: ["move", "Terminal", "0", "0"]}]
|
||
call_tool: calls=[{tool: "gui_windows", args: ["resize", "Firefox", "1920", "1080"]}]
|
||
call_tool: calls=[{tool: "gui_windows", args: ["desktop", "2"]}]
|
||
```
|
||
|
||
Title matching is case-insensitive substring.
|
||
|
||
---
|
||
|
||
## gui_apps
|
||
|
||
Launch, list, or activate applications.
|
||
|
||
**Args**: `[command]` or `[command, app_name]`
|
||
|
||
Commands: `list` (running), `installed` (all .desktop), `launch <name>`, `activate <title_or_class>`
|
||
|
||
```
|
||
call_tool: calls=[{tool: "gui_apps", args: ["launch", "firefox"]}]
|
||
call_tool: calls=[{tool: "gui_apps", args: ["activate", "dolphin"]}]
|
||
call_tool: calls=[{tool: "gui_apps", args: ["installed"]}]
|
||
```
|
||
|
||
---
|
||
|
||
## gui_input
|
||
|
||
Simulate keyboard and mouse input. Uses ydotool (Wayland) or xdotool (X11).
|
||
|
||
**Args**: `[command, ...params]`
|
||
|
||
Commands:
|
||
- `key <combo>` — press key combo (e.g., `ctrl+c`, `Return`, `alt+F4`, `super`)
|
||
- `type <text>` — type a string character by character
|
||
- `click [button] [x y]` — mouse click (1=left, 2=middle, 3=right)
|
||
- `move <x> <y> [--relative]` — move cursor
|
||
- `scroll <up|down|left|right> [amount]`
|
||
|
||
```
|
||
call_tool: calls=[{tool: "gui_input", args: ["key", "ctrl+s"]}]
|
||
call_tool: calls=[{tool: "gui_input", args: ["type", "Hello, world!"]}]
|
||
call_tool: calls=[{tool: "gui_input", args: ["click", "1", "500", "300"]}]
|
||
call_tool: calls=[{tool: "gui_input", args: ["scroll", "down", "5"]}]
|
||
```
|
||
|
||
**Caution**: Input simulation is a write operation — confirm intent before using `key` with destructive combos (alt+F4, ctrl+w) or `click` on unknown targets.
|
||
|
||
---
|
||
|
||
## gui_activities
|
||
|
||
Manage KDE Plasma Activities.
|
||
|
||
**Args**: `[command]` or `[command, name_or_id]`
|
||
|
||
Commands: `list`, `current`, `switch <name_or_id>`, `create <name>`, `remove <name_or_id>`
|
||
|
||
```
|
||
call_tool: calls=[{tool: "gui_activities", args: ["list"]}]
|
||
call_tool: calls=[{tool: "gui_activities", args: ["switch", "Development"]}]
|
||
call_tool: calls=[{tool: "gui_activities", args: ["create", "Research"]}]
|
||
```
|
||
|
||
---
|
||
|
||
## gui_files
|
||
|
||
Search files via Baloo (KDE file indexer). Falls back to `locate`.
|
||
|
||
**Args**: `[query, ...flags]`
|
||
|
||
Flags: `--type=document|image|video|audio|folder`, `--path=dir`, `--limit=N`
|
||
|
||
```
|
||
call_tool: calls=[{tool: "gui_files", args: ["meeting notes"]}]
|
||
call_tool: calls=[{tool: "gui_files", args: ["invoice", "--type=document", "--path=/home/lkn/doc"]}]
|
||
call_tool: calls=[{tool: "gui_files", args: ["screenshot", "--type=image", "--limit=10"]}]
|
||
```
|
||
|
||
---
|
||
|
||
## gui_shortcuts
|
||
|
||
Manage and trigger KDE global keyboard shortcuts.
|
||
|
||
**Args**: `[command, ...params]`
|
||
|
||
Commands:
|
||
- `list` — all shortcut components
|
||
- `list <component>` — shortcuts for that component
|
||
- `get <component> <name>` — current key binding
|
||
- `trigger <component> <name>` — invoke the shortcut action
|
||
- `set <component> <name> <keys>` — change key binding
|
||
|
||
```
|
||
call_tool: calls=[{tool: "gui_shortcuts", args: ["list"]}]
|
||
call_tool: calls=[{tool: "gui_shortcuts", args: ["list", "kwin"]}]
|
||
call_tool: calls=[{tool: "gui_shortcuts", args: ["trigger", "kwin", "Overview"]}]
|
||
call_tool: calls=[{tool: "gui_shortcuts", args: ["trigger", "plasmashell", "show dashboard"]}]
|
||
```
|
||
|
||
---
|
||
|
||
## gui_accessibility
|
||
|
||
Full GUI automation via the AT-SPI2 accessibility tree. Inspect, navigate, and interact with any application's UI elements.
|
||
|
||
**Args**: `[command, app_name, ...params]`
|
||
|
||
Commands:
|
||
- `apps` — list all accessible applications (no app_name needed)
|
||
- `tree <app> [depth]` — dump widget tree (default depth: 4)
|
||
- `inspect <app> [depth]` — brief overview (default depth: 2)
|
||
- `find <app> <role> [name]` — find elements by role and optional name
|
||
- `read <app> <path>` — read text/value of element
|
||
- `click <app> <path>` — activate/click element
|
||
- `type <app> <path> <text>` — type into element
|
||
- `action <app> <path> <action_name>` — invoke named action
|
||
|
||
Paths use slash-separated indices (`0/2/1`) or `role:name` patterns (`push button:OK`).
|
||
|
||
```
|
||
call_tool: calls=[{tool: "gui_accessibility", args: ["apps"]}]
|
||
call_tool: calls=[{tool: "gui_accessibility", args: ["inspect", "Kate"]}]
|
||
call_tool: calls=[{tool: "gui_accessibility", args: ["find", "Firefox", "push button", "Reload"]}]
|
||
call_tool: calls=[{tool: "gui_accessibility", args: ["click", "Firefox", "push button:Reload"]}]
|
||
call_tool: calls=[{tool: "gui_accessibility", args: ["type", "Kate", "0/0/text:untitled", "Hello world"]}]
|
||
call_tool: calls=[{tool: "gui_accessibility", args: ["tree", "Dolphin", "3"]}]
|
||
```
|
||
|
||
**Workflow**: Use `apps` → `inspect` → `find` to locate elements, then `click`/`type`/`action` to interact.
|
||
|
||
**Requires**: `python3-gobject` and AT-SPI2 typelib (`dev-libs/at-spi2-core` on Gentoo).
|
||
|
||
---
|
||
|
||
## gui_brightness
|
||
|
||
Get/set screen brightness via PowerDevil. Accepts **percentages** (0–100), not raw hardware values.
|
||
|
||
**Args**: `[command]` or `[command, value]`
|
||
|
||
Commands: `get`, `set <percent>`, `up [step]`, `down [step]`
|
||
|
||
```
|
||
call_tool: calls=[{tool: "gui_brightness", args: ["get"]}]
|
||
call_tool: calls=[{tool: "gui_brightness", args: ["set", "50"]}]
|
||
call_tool: calls=[{tool: "gui_brightness", args: ["up", "10"]}]
|
||
call_tool: calls=[{tool: "gui_brightness", args: ["down", "25"]}]
|
||
```
|
||
|
||
**Note**: The underlying PowerDevil API uses absolute values (0 to ~10000). This tool handles the conversion — always pass percentages.
|