What is a screen-aware voice assistant?
A screen-aware voice assistant combines real-time voice conversation with shared screen context. When the user enables screen awareness and shares their display, HEIDI can see what is on screen, highlight UI elements, guide through forms and workflows, and answer questions using Operational Knowledge — with human approval before sensitive clicks or actions.
Capabilities
What you can do with HEIDI
See what you share
When screen awareness is enabled, HEIDI understands the visible UI context.
Highlight the right element
HEIDI can point to fields, buttons, and regions on your shared screen.
Short-term screen context
Designed for focused sessions — onboarding, troubleshooting, training — not indefinite recording.
Knowledge-grounded guidance
Answers reference approved SOPs, manuals, and operational maps.
Clicks require confirmation
UI interactions that change state wait for explicit user confirmation.
Use cases
Where teams apply HEIDI
Real workflows that benefit from visual design, automation, and governance.
SOP onboarding with voice + screen share
A new hire shares their screen; HEIDI walks through the approved procedure step by step.
Can you see my screen? Guide me to the next step
HEIDI reads the form, highlights the field you need, and explains what to enter.
Troubleshoot a workflow step with UI highlights
Support Ops shares the screen; HEIDI identifies where the process diverged from the SOP.
Train a new hire through a browser workflow
HEIDI narrates each step, highlights controls, and answers questions from the training manual.
Requirements session: extract criteria from shared context
During a live review, HEIDI captures structured requirements from what is visible and discussed.
How it works
From chaos to clarity in 4 steps
Enable screen awareness
Turn on the screen awareness skill in your HEIDI configuration.
Share your display
Share the browser tab or window HEIDI should observe.
Speak and listen
Real-time voice with live transcription — ask questions naturally.
Ground and highlight
HEIDI queries knowledge and highlights relevant UI elements on screen.
Confirm before acting
Clicks and system actions wait for your explicit approval.
Avoid these
Common mistakes (and how to avoid them)
Sharing screen without enabling the skill
Screen awareness must be explicitly enabled — HEIDI does not observe your display by default.
Enable screen awareness in configuration before starting a guided session.
Expecting HEIDI to click without confirmation
UI clicks that change application state require explicit user confirmation.
Use highlights and voice guidance; approve clicks when you are ready.
Listen → See → Ground → Decide → Act
Screen-aware sessions follow the same operational loop as every HEIDI voice interaction:
- Listen — realtime speech and transcription
- See — shared screen context when enabled
- Ground — query Operational Knowledge and policies
- Decide — summarise, recommend, highlight
- Act — only with approval for sensitive steps
This is designed for focused sessions — onboarding a new hire, troubleshooting a stuck workflow, or capturing requirements — not for passive always-on surveillance.