Eyes, hands and memory for an agent driving the iOS Simulator.
An agent driving the Simulator is slow for three reasons, and only the first is obvious: every look is a wait, every step is a round trip, and nothing is ever remembered. simframe fixes all three.
Two jobs on a real React Native field app, end to end and submitted for real: create a service request, and add an asset. One run per setup, 2026-10-03.
| Who drove | Both jobs | Calls | Screenshots | Tokens |
|---|---|---|---|---|
| Opus, Claude Code’s built-in simulator tool, no simframe | yes | ~176 | 112 | 296k |
| Opus + simframe | request only* | ~50 | ≥4 | 118k |
| Opus briefing a Sonnet driver + simframe | yes | ~88 | few | 128k |
Same model, with and without simframe: 2.5× fewer tokens and 3.5× fewer calls. The Sonnet driver did everything the no-simframe run did with 2.3× fewer tokens on a cheaper model. * The session’s own permission check blocked that run’s asset upload. Not near human speed yet, measured: a person who knows the app did both jobs in about 1.5 minutes (medians of five runs, 39 s and 49 s); the agents took 13–16 minutes, 9–10× slower, because every model turn costs seconds. Full table and its limits.
The same four-tab navigation flow, on a real production app:
| Question | Before | With simframe |
|---|---|---|
| Look at the screen | ~130–400 ms, blocking | ~20 ms, already captured |
| “Did anything change?” | a full image | ~2 ms, text only |
| A five-step flow | 5+ model round trips | one call |
| Finding a control | read tree (~570 ms) + reason about it | ~1 ms from memory |
| The same flow, third run | no improvement — every run is the first | 3304 ms, 4/4 from memory |
No mis-taps in any run. What is left is mostly the app's own animation and data load.
# Claude Code claude mcp add --scope user simframe -- npx -y simframe mcp # or use it directly npm install -g simframe simframe doctor
Capture starts on the first call, targets the booted simulator, and stops itself 15 minutes after the last request.
Each layer works without the ones below it, and doctor tells you which
you have. Watching the screen needs nothing but Xcode.
| Capability | Needs | Without it |
|---|---|---|
| Watch, wait, recall | nothing extra | — |
| Read labels + coordinates from pixels | swiftc | falls back to the accessibility tree alone |
| Tap, type, swipe | idb | simframe observes but cannot touch |
Every change question is really “changed since when?” — and the answer is almost never “since the previous frame”. A UI transition is over in about 700 ms, so comparing consecutive frames tells a caller that polls every few seconds “nothing changed”, even though the screen is completely different from when it last looked.
So simframe compares against the last frame you observed. Over MCP that is automatic. From the CLI you capture a baseline before acting:
H=$(simframe mark) # ...tap, launch, navigate... simframe wait --since=$H # change, then settle simframe state --since=$H # what moved, as text
An accessibility tree is a promise apps do not always keep. Tested against a real production app, its custom tab bar published no children at all, its icon buttons carried unreadable private-use glyphs, and its React Native text inputs were absent entirely — the controls used most were exactly the ones that could not be tapped by name.
So simframe reads the screen two ways and remembers the result:
OCR also beats measuring by eye. On one tab bar the first tab's centre sat at x = 62, not the x = 40 an even five-way split predicts — a silent mis-tap on every attempt.
A frame hash changes whenever any pixel group changes — a clock digit, one new row of data — which makes it useless as a key for “have I seen this screen before?”. The layout hash crops the status bar and takes a difference hash over a 12×24 grid.
| Same screen, revisited while settled | 0–4 |
| Same screen, but loading vs loaded | 48–98 |
| Different screens | 77–113 |
Only one of those errors is dangerous. Matching the wrong screen would tap the wrong control, and that needs two different screens within 12 bits of each other — the closest pair ever measured was 77. Failing to recognise a screen you have seen is harmless: it rebuilds the map and taps correctly. So the tolerance sits far below the collision floor rather than being tuned to maximise hits.
Three sessions drove the same task family on the same production app, and the third one existed only to hold a variable still. Pass 2 was faster than pass 1 by twenty-one model round trips — but it had a warm graph and an operator who had already driven the app. Pass 3 used a fresh operator against the warm graph.
| Pass | Operator | Graph | round trips |
|---|---|---|---|
| 1 | fresh | cold | 33 |
| 3 | fresh | warm | 24 |
| 2 | experienced | warm | 12 |
The graph is worth ~27 % of the round trips. Operator knowledge is worth the rest — half of everything left after the graph had taken its share. A second reporter, with no way to isolate the graph, derived ~23 % from escalation rate per step without knowing that number.
One app, one operator per pass, so operator variance and operator experience are not separated from each other. Round trips are not seconds. The part worth taking away is the direction: the memory is real and it is the smaller half, and what an operator learns is the larger half that no tool here carries yet.
Nothing here is written for a particular app. What varies is how much accessibility tree an app publishes, and simframe is built to degrade rather than fail:
A control the app declares but never names — an icon-only overflow menu, a back chevron —
is listed with its coordinates and can be tapped by #ref rather than by
name, and the map says how many such controls are on the screen. A view the app never
declared accessible at all is invisible to any tree, ours included; the same line says
so, because a count of zero on a screen that has one would be the more expensive answer.
Run simframe ui on any screen to see exactly what simframe can see, each
target marked ax or ocr. Two honest caveats: OCR reads
text, so a purely graphical icon is invisible to it, and the confirm vocabulary
(APPLY, OK, SAVE…) is English.
iPhone 17 Pro, iOS 26.5, Apple Silicon, default settings.
| Warm frame read | ~20 ms |
| State check, text only | ~2 ms |
| Accessibility tree read | ~570 ms |
| On-device OCR of a full frame | ~290 ms |
| Screen map: first visit / remembered | ~600 ms / ~1 ms |
Raw simctl io screenshot, for comparison | ~130 ms, every look |
| CPU | 1.1 % idle · 3.1 % active |
| Frame memory | ~60 s of screen, ~2.7 MB |
The supervisor answers three words — wait, retry,
stop — when a step fails, before the failure reaches the model.
We assumed capacity barely mattered for that and were asked to prove it. One
simulator pass recorded 22 real failures; every arm then answered the identical
22, three times, briefed from the same source text and constrained to the same
three-word schema.
| arm | accuracy | median |
|---|---|---|
| always the commonest answer | 55 % | — |
| a threshold on a number we already compute | 95 % | 0 ms |
| Apple Foundation Models (~3B, on-device) | 77 / 82 / 86 % | ~640 ms |
| qwen3:8b (4-bit, 5.2 GB) | 91 % | 919 ms |
| qwen3:14b (4-bit, 9.3 GB) | 82 % | 1,489 ms |
And the number a supervisor should actually be judged on is not accuracy at all. On the same 22 failures it handled 9 locally with no round trip and escalated 13 — at 10–16 s a round trip, roughly 90–145 s saved.
The larger model scored lower than the smaller one. A free
threshold beat all three on a population it was never fitted to. And the accuracy
ranking inverts the safety ranking: every arm errs in one direction only, Apple’s
all wait where stop was right and both Qwen arms’ all
stop where wait was right — and a wrong stop
abandons a plan that would have worked. Reporting only the percentages would have
recommended the wrong model. Full conditions and the reasons to distrust the
threshold are in
What We Believed First.
settle waits for the screen to stop. Stillness is decided from a mean over a
grid of the frame, and that has one blind spot worth knowing about before you rely on it:
a small animation does not move a mean. Measured on a screen built to
never settle — a spinner rotating continuously, frames arriving every 77–95 ms:
| mean difference between frames | 0.00196 (threshold 0.004) |
| largest single-cell difference | 0.0275 |
what stableForMs reported | 79,207 ms |
So a settle can return satisfied while part of the screen is still moving. It now
says so — settled after 63ms (a 13x13 region is still animating) —
because the capture daemon localises the animation and publishes its bounding box. A
settle that times out names where the movement is instead:
the movement is top right (78% of it), with a region map under it.
Deliberately reported rather than acted on. Refusing to settle whenever something is animating would be right for a spinner and wrong for a blinking text cursor, which is also small and also never stops. Separating those two needs a measurement that has not been made, and picking a threshold before making it is how the other numbers on this page got themselves reversed.
abstain word cost it a third of
its accuracy while it never used the word once. Both are measured; neither is
shipped on.simctl cannot capture a physical device.