Help that knows where you are
Instructions for computer software often assume that a person can see the screen. A screen-reader user may hear the focused control but still lack the wider context needed to understand what changed or where to go next. AskEase is a research prototype designed to answer questions using the state of the computer at that moment. [1]
The system combines a desktop screenshot, structured information about the screen, recent screen-reader output, conversation history and software documentation. A language model uses that context to produce step-by-step guidance written for screen-reader navigation. [1]
A small controlled comparison
Twelve blind screen-reader users each tried unfamiliar tasks with AskEase and with the help tools they would ordinarily use. The task sets covered GitHub Copilot and Microsoft Word. The researchers counted only tasks completed correctly within 20 minutes. [1]
1.5 of 2 tasks
The average completed with AskEase. Every participant completed at least one task.
0.5 of 2 tasks
The average completed with the baseline help tools in the same study.
Six participants completed both tasks with AskEase within the time limit, compared with two using baseline tools. Participants also reported less physical demand, effort and frustration. This was a short study of a prototype, not evidence about long-term independent use. [1]
Why correct guidance matters more here
The prototype sometimes lacked enough screen detail, missed shortcut conflicts or produced an incorrect step. A sighted user may notice that an instruction does not match the screen. Someone receiving the interface through speech can have a harder time checking the same mistake. [1]
Participants treated trust as provisional. The authors argue that future systems should make their behavior easier to verify instead of asking users to accept a confident answer. [1]
What this study leaves open
Participants were deliberately unfamiliar with the target software and generally had moderate experience with language-model tools. The results may not transfer to expert users or to people who have never used this kind of assistant. A longer study would be needed to see whether benefits persist after the novelty wears off. [1]
AskEase currently depends on NVDA and Windows, and its software documentation was collected manually. Capturing screenshots and screen-reader traces also creates privacy risks because those inputs can contain sensitive information. Wider use would require more platforms, stronger verification and privacy safeguards. [1]
Sources & context
One original user study reported in a conference paper. The arXiv and conference links refer to the same work.
From Struggle to Success: Context-Aware Guidance for Screen Reader Users in Computer Use
Chen and colleagues · CHI 2026 · January 26, 2026 preprint