ZeptapDriving the iPhone
The operating loop and calibrated rules that make an agent reliable on a real iPhone - targeting, scrolling, typing, recovery and safety.
A real iPhone differs from a browser: there is no DOM, taps land on pixels, and animations take time. These rules come from eval runs on a real iPhone, and extend the instructions the server sends in its initialize response.
The loop
- Look. Start with
screenshot. Read the screen before acting on it. - Act. Make one action.
- Verify. Read the screenshot the action returns. It is taken after the screen settles, so it shows the result. Continue only when it shows what you expected.
When a result note says screen still changing, call wait before the next action: an animation or load is still running, and a tap now could land on a moving target.
Targeting
- Prefer text. For any visible label, use
tap_text(orfind_textfor coordinates). OCR finds the label's exact center; estimating from the image is the most common cause of missed taps. - Tap centers. When tapping by coordinates, aim at the middle of the control.
- Pick among duplicates with
tap_text'sindex(0 is the topmost match), or narrow withfind_textfirst. - Icons without text (toolbar glyphs, toggles): tap by coordinates from the latest screenshot.
- Ignore two overlays. The small dark dot is the pointer. The large translucent circle is the AssistiveTouch button. Neither is part of the app. When the AssistiveTouch button covers a control, scroll the content out from under it.
- The status bar always reads 9:41 in captures. It is not the real time, and battery and signal are not real either.
Scrolling
| Need | Use | Calibration |
|---|---|---|
| Move a list a precise distance | drag at the default 800 ms | Content moves about 0.85× the drag distance, with no momentum |
| Reveal content further down | drag from a lower point to a higher one | For example from_y: 1000 → to_y: 400 |
| Jump far through a long list | flick up | One flick can cover a whole Settings list |
| Turn a picker wheel | drag | About 50 px per step |
| Move a slider | drag along the track |
Start scroll gestures in the content area, away from the screen edges: an edge swipe from the left goes back, and one from the bottom goes home.
After each scroll, re-find the target (find_text or tap_text) rather than reusing coordinates from before the scroll.
Typing
- Focus first. Tap the field, then confirm the keyboard or cursor is showing before
type_text. - US-keyboard ASCII only. Any other character (accents, emoji, curly quotes,
₹) rejects the whole call with nothing typed. Rephrase the text in ASCII. - iOS edits what you type. Autocorrect, auto-capitalization and smart quotes can change it. Read the returned screenshot, and fix mistakes with
press_keybackspace(userepeat) or select-all withpress_keya+command. - Submit with
press_keyenter, or tap the on-screen button. - Long text types at about 80 ms per character.
Navigating and recovering
| Situation | Do |
|---|---|
| Start an app | open_app with the name shown under its icon |
| Leave an app | home |
| One screen back | back, or tap the app's back button |
| Spotlight, a menu, a sheet or the keyboard is open | dismiss |
| Control Center or Notification Center is open | home |
| Lock screen, passcode pad, or no screen image | unlock, once. If it fails, tell the user |
| Unsure where you are | home, then start again from the Home Screen |
Settings, Safari and other system apps restore their last screen when reopened. After open_app, read the screenshot before assuming you are on the app's first screen.
Safety
Confirm with the user before any action that is hard to undo or reaches other people: sending a message or email, posting, buying, deleting, changing an account or a privacy setting, or accepting a permission prompt. The phone is real and the actions are real.
Leave the phone as you found it: undo test data you created, and restore settings you changed.