Computer-use agents are usually evaluated with a deceptively simple question: did the agent finish the task? That is useful, but it is not enough for an accessibility-first system. An agent can complete a task while requiring exhausting corrections, hiding important state changes, or making it difficult for the person to stop and recover.
This is version 0.1 of an open evaluation framework for accessible computer-use agents. The machine-readable protocol is available as JSON. It defines what should be measured before any particular product is declared accessible or reliable; it does not publish benchmark results or claim independent validation.
What should an accessible computer-use agent measure?
Task completion remains the starting point, but the unit of evaluation should be the full interaction between person, input method, agent, and interface. The framework therefore measures seven dimensions: completion, input effort, correction burden, recovery, safety, latency, and user control.
Completion asks whether the intended end state was reached. Input effort counts the commands, gestures, breaths, gaze selections, taps, or other deliberate inputs required. Correction burden records how often the user has to restate intent or repair an incorrect action. Recovery measures whether the system can return to a safe state after failure.
Safety separates reversible navigation from actions with external consequences, such as sending a message, deleting a file, or confirming a purchase. Latency measures both system response time and the time until the user can understand what happened. User control asks whether the person can inspect, pause, cancel, and resume the action without switching to a less accessible input method.
Task families
A useful benchmark should include ordinary work rather than a collection of visually convenient demos. Version 0.1 proposes six task families: navigation, text entry and editing, information retrieval, file management, communication, and recovery from a deliberately introduced error.
Each task needs a declared start state, target state, allowed applications, consequence level, supported input method, and observable completion rule. A result is not comparable when one run starts from a clean desktop and another starts with the correct window already focused.
Why task completion alone fails
Imagine two agents that both send the correct email. The first needs one clear command, reads back the recipient and message, asks for confirmation, and exposes an accessible cancel action. The second needs eight corrections, selects the wrong recipient twice, and sends immediately when it finally understands. A binary success metric calls those runs equal. The user experience clearly is not.
Accessibility evaluation must also avoid treating a disabled participant as a device for validating a predetermined product claim. Co-design should influence the task set, failure categories, acceptable trade-offs, and interpretation of results. Participation requires informed consent, appropriate compensation, and a clear process for removing recordings or data.
Reporting a result
Every published run should identify the software version, operating system, application versions, model and provider where relevant, input hardware, network conditions, task definition, number of attempts, and whether the evaluator is a product team member, an independent researcher, or a participant using the system for their own access needs.
Aggregate scores should never erase failure severity. A benchmark report should publish the task-level record and separate harmless retries from irreversible or privacy-sensitive mistakes. It should also state what was not tested.
What version 0.1 does not claim
This framework is a proposal, not a standard, clinical assessment, or certification. It has not yet been validated through a representative study. It is being published early so accessibility practitioners, assistive-technology users, researchers, and other computer-use teams can challenge the task definitions before results harden around weak metrics.
The protocol is informed by the principles behind WCAG 2.2 and the User Agent Accessibility Guidelines, while focusing specifically on agent-mediated desktop actions. For project context, see the Fluent case study. Corrections and source changes are tracked through the site's public evidence policy.