Some Buttons Are More Explosive Than Others

TL;DR

• A computer-use agent should classify a proposed action by consequence, recoverability, uncertainty, and exposure before it touches the interface.
• Confidence helps the agent aim; it does not grant permission. A model can be 99% sure that it found the correct button and still need approval to press it.
• Harmless navigation can run automatically and be verified afterward, while sending, publishing, purchasing, changing permissions, and permanent deletion need a preview, confirmation, or handoff.
• The policy belongs between planning and execution, where it can limit the executor's actual capability instead of politely asking the planner to behave.
• The reproducible starting points are OSWorld and OS-Harm; my proposed Haunted Desktop suite adds stale screens, layout shifts, duplicate clicks, expired Undo, prompt injection, and stop requests during motion.

Clicking Bold and clicking Send look almost identical to an executor: the pointer goes down, the pointer comes up, and the application changes state. One action makes six words thicker; the other can notify 4,000 people that you attached the PDF containing your grocery list.

I am building Fluent, a computer-use agent for accessibility that lets people control a computer through voice and other accessible inputs. A person using voice, gaze, or a switch cannot always grab the mouse when the agent starts improvising, so the system needs to move quickly through harmless work while recognizing the exact moment a click becomes a commitment.

The hard part is not cursor control; it is authority.

Coordinates do not describe consequences

A basic computer-use loop looks like observe -> choose -> click -> observe, which is enough to move a cursor but not enough to decide whether moving it is a good idea. Before execution, the planner should describe the intended state change in semantic terms: operation, target, scope, audience, expected result, and recovery path.

Click at 842, 611 contains none of that information; it is an instruction with the situational awareness of a Roomba, while the danger can change even when the verb stays the same. Deleting a generated thumbnail is cheap if it remains in Trash, while deleting the only copy of a medical document may be catastrophic; similarly, run can mean running a unit test or running payroll, which is why a list of scary verbs will always miss the interesting failures.

The policy needs to evaluate the proposed state transition, not the surface gesture that triggers it.

Risk is a vector, not one comforting number

I would keep the action's blast radius as four separate values:

B(action, state) = [consequence, recoverability, uncertainty, exposure]

Consequence is the worst credible result in the current state. Renaming one local file and renaming 80,000 production objects share a verb, but they do not share a consequence.

Recoverability describes the real path back, including whether Undo is complete, available long enough, and usable through the person's input method. A three-second Undo toast that disappears before a screen reader announces it is decorative safety.

Uncertainty covers intent, target, visible state, hidden state, and expected outcome; model confidence captures only part of this because a model can be extremely confident about a screenshot that became stale 200 milliseconds ago.

Exposure measures the boundary being crossed. A local draft differs from a shared document, which differs from a public post, financial transaction, legal commitment, or medical action; once external state changes, another person or system may react before recovery begins.

Keeping these values separate prevents a good average from laundering a catastrophic dimension. Ninety-nine percent confidence does not cancel an irreversible transfer; the average may look green while one input is actively on fire.

Five action tiers without the committee meeting

The vector still needs to produce a control decision, so I map actions into five contextual tiers:

1. Tier 0: Observe. Read text, inspect the accessibility tree, or check window state; execute automatically inside the user's privacy boundary.
2. Tier 1: Navigate. Scroll, focus, open menus, or change tabs; execute and verify that the expected view appeared.
3. Tier 2: Reversible edit. Type into a draft, apply formatting, or move a backed-up file; announce the plan when useful, preserve recovery, and verify the target.
4. Tier 3: Commitment boundary. Send, publish, share, purchase, submit, change permissions, or permanently delete; render the exact operation and ask before execution.
5. Tier 4: Restricted. Perform bulk deletion, large transfers, credential changes, medical actions, or operations affecting many people; require stronger authentication, another reviewer, a handoff, or no capability at all.

The tier belongs to the action in context rather than to the verb. Deleting a backed-up screenshot may be Tier 2, emptying the trash may be Tier 3, and deleting prod-final-FINAL-actually-final can wait for a human with excellent posture and a recent backup.

Confidence helps the agent aim; it does not issue a permit

Low confidence is one reason to ask the user, while high consequence is a separate reason; mixing the two into one score hides the policy boundary that matters.

If the agent is unsure whether a harmless menu lives under View or Window, it can open one and inspect the result; if it is completely certain that it found Delete Account, congratulations -- it still asks.

The confirmation UI should display the executor's real arguments, including the recipient, amount, permission change, selected files, or content diff. A free-form summary can drift away from the pending action between planning and execution, so the user should approve what the computer will do rather than the bedtime story the model wrote about it.

Asking 47 times is also a failure

Confirming every click sounds safe until the computer becomes Clippy after completing law school. For an accessibility workflow, each prompt may require speech, breath, a switch press, gaze dwell, or another deliberate input; the agent has saved no work if it returns every tiny decision to the user.

A CHI 2026 study on confirmation timing evaluated intermediate checkpoints in multi-step agent tasks; among 48 participants, 81% preferred intermediate confirmation over confirmation only at the end, while task completion time fell by 13.54%. The result does not validate this accessibility framework, but it does show that confirmation placement matters more than simply maximizing the number of interruptions.

A separate study comparing four oversight strategies, also with 48 participants, found that plan-based strategies were associated with lower rates of problematic actions, although no single strategy was uniformly best and runtime intervention did not become equally reliable once risky behavior was already visible. The useful design target is therefore minimum necessary interruption: group harmless steps, make the commitment boundary legible, and ask once with the information required for judgment.

WCAG 2.2 already makes a related distinction for legal, financial, and data submissions: important actions should be reversible, checked, or reviewable before finalization, while ordinary saves do not each require confirmation. The guidance is written for web accessibility, not autonomous agents, but its separation between routine editing and serious consequence maps cleanly onto this problem.

Put policy between planning and execution

The safety layer cannot remain another sentence in the planner prompt. The planner should propose a semantic action, a separate policy should classify it, and the executor should receive only the capability allowed by that decision.

type ProposedAction = {
  operation: "send_email";
  target: "all@company.com";
  scope: 4000;
  external: true;
  reversible: false;
  expectedPostcondition: "message appears in Sent";
};

// perceive -> propose -> classify -> gate -> execute -> verify

The gate returns one of four decisions: execute, announce and execute, confirm, or hand off. Drafting an email does not grant permission to send it, and attaching one file does not grant unrestricted access to every folder; each execution token should describe the permitted operation, target, scope, and expiry.

After execution, verification checks the semantic postcondition rather than merely checking whether the click occurred. Did the message enter Sent with the intended recipient and attachment? Did the permission change for the correct account? Did one file move, or did the selection quietly expand to 600?

When verification fails, the system should stop the action chain, capture the observed state, identify what changed, calculate the remaining blast radius, and offer bounded recovery options. BackBench formalizes recovery as its own planning problem across 50 computer-use tasks; this is more useful than automatically sending a second email titled "PLEASE IGNORE" and doubling the audience's interest.

Try to break the policy before a user does

This model needs a reproducible test rather than a tasteful architecture diagram followed by a victory lap. OSWorld provides an open desktop environment, task setup, agent interfaces, execution traces, and evaluation code; OS-Harm builds on that environment with 150 tasks covering deliberate misuse, prompt injection, and model misbehavior.

git clone https://github.com/xlang-ai/OSWorld
git clone https://github.com/tml-epfl/os-harm

I would add a smaller adversarial suite called The Haunted Desktop, where an ordinary task receives one controlled failure: a layout shift moves the destructive button, lag invites a duplicate click, a selection expands silently, an on-screen document instructs the agent to ignore the user, Undo expires, or the user says Stop while the pointer is already moving.

The report should separate task completion from safety outcomes: harmful actions by tier, missed confirmations, unnecessary confirmations, stop latency, postcondition-verification coverage, recovery success, recovery side effects, and deliberate user inputs. Ten annoying prompts and one unauthorized transfer are both defects, but they are not eleven units of the same defect.

Sources and artifacts

• Fluent
• WCAG 2.2: Error Prevention for Legal, Financial, and Data Actions
• When Should Users Check?
• Comparing Human Oversight Strategies for Computer-Use Agents
• Human-Guided Harm Recovery for Computer Use Agents
• OSWorld repository
• OS-Harm repository

All embedded visuals are original to this draft; the raster illustrations were generated with OpenAI image generation, while the precise policy diagrams were constructed from the stated model rather than copied from external images.

Conclusion

A useful computer-use agent should move quickly through harmless work and become extremely boring around irreversible work; it can scroll, inspect, open, draft, and organize without scheduling a committee meeting for every click, but it should slow down when an action leaves the screen, changes another person's access, spends money, deletes durable data, or speaks in the user's name.

That balance matters for accessibility because autonomy is supposed to reduce effort, while control preserves the user's authority. The policy is not complete until disabled users and accessibility practitioners test it against hostile desktop states, expensive input methods, and recovery paths that work outside a demo.

The cursor may be a loaded weapon; at minimum, the agent should know which direction it is pointing.