Eyes, hands and memory for an agent driving the iOS Simulator.
An agent driving the Simulator is slow for three reasons, and only the first is obvious: every look is a wait, every step is a round trip, and nothing is ever remembered. simframe fixes all three.
The same four-tab navigation flow, on a real production app:
| Question | Before | With simframe |
|---|---|---|
| Look at the screen | ~130–400 ms, blocking | ~20 ms, already captured |
| “Did anything change?” | a full image | ~2 ms, text only |
| A five-step flow | 5+ model round trips | one call |
| Finding a control | read tree (~570 ms) + reason about it | ~1 ms from memory |
| The same flow, third run | no improvement — every run is the first | 3304 ms, 4/4 from memory |
No mis-taps in any run. What is left is mostly the app's own animation and data load.
# Claude Code claude mcp add --scope user simframe -- npx -y simframe mcp # or use it directly npm install -g simframe simframe doctor
Capture starts on the first call, targets the booted simulator, and stops itself 15 minutes after the last request.
Each layer works without the ones below it, and doctor tells you which
you have. Watching the screen needs nothing but Xcode.
| Capability | Needs | Without it |
|---|---|---|
| Watch, wait, recall | nothing extra | — |
| Read labels + coordinates from pixels | swiftc | falls back to the accessibility tree alone |
| Tap, type, swipe | idb | simframe observes but cannot touch |
Every change question is really “changed since when?” — and the answer is almost never “since the previous frame”. A UI transition is over in about 700 ms, so comparing consecutive frames tells a caller that polls every few seconds “nothing changed”, even though the screen is completely different from when it last looked.
So simframe compares against the last frame you observed. Over MCP that is automatic. From the CLI you capture a baseline before acting:
H=$(simframe mark) # ...tap, launch, navigate... simframe wait --since=$H # change, then settle simframe state --since=$H # what moved, as text
An accessibility tree is a promise apps do not always keep. Tested against a real production app, its custom tab bar published no children at all, its icon buttons carried unreadable private-use glyphs, and its React Native text inputs were absent entirely — the controls used most were exactly the ones that could not be tapped by name.
So simframe reads the screen two ways and remembers the result:
OCR also beats measuring by eye. On one tab bar the first tab's centre sat at x = 62, not the x = 40 an even five-way split predicts — a silent mis-tap on every attempt.
A frame hash changes whenever any pixel group changes — a clock digit, one new row of data — which makes it useless as a key for “have I seen this screen before?”. The layout hash crops the status bar and takes a difference hash over a 12×24 grid.
| Same screen, revisited while settled | 0–4 |
| Same screen, but loading vs loaded | 48–98 |
| Different screens | 77–113 |
Only one of those errors is dangerous. Matching the wrong screen would tap the wrong control, and that needs two different screens within 12 bits of each other — the closest pair ever measured was 77. Failing to recognise a screen you have seen is harmless: it rebuilds the map and taps correctly. So the tolerance sits far below the collision floor rather than being tuned to maximise hits.
Nothing here is written for a particular app. What varies is how much accessibility tree an app publishes, and simframe is built to degrade rather than fail:
Run simframe ui on any screen to see exactly what simframe can see, each
target marked ax or ocr. Two honest caveats: OCR reads
text, so a purely graphical icon is invisible to it, and the confirm vocabulary
(APPLY, OK, SAVE…) is English.
iPhone 17 Pro, iOS 26.5, Apple Silicon, default settings.
| Warm frame read | ~20 ms |
| State check, text only | ~2 ms |
| Accessibility tree read | ~570 ms |
| On-device OCR of a full frame | ~290 ms |
| Screen map: first visit / remembered | ~600 ms / ~1 ms |
Raw simctl io screenshot, for comparison | ~130 ms, every look |
| CPU | 1.1 % idle · 3.1 % active |
| Frame memory | ~60 s of screen, ~2.7 MB |
simctl cannot capture a physical device.