Agents got good at the web quickly. Pages have a DOM, forms have names, and there are mature browser-automation tools. But a lot of what people do on a phone happens inside native apps that have no website and no public API: messaging, ride-hailing, delivery, banking, and very often the app you are building right now.
For an agent to reach those apps, it needs a phone. Not a picture of a phone, and not a mock that replays canned screens: a full Android system with real apps installed, where a tap does what a tap does.
This post is about what that phone has to offer an LLM, and the API shape that falls out of it. The examples come from ERIUS PHONE (the cloud Android phone we build), but the design points apply to any phone tool you hand to a model.
What “a whole phone” means
The minimum set, as we see it:
- Real apps. Your own APK, or apps from the Play Store. A screenshot simulator can’t run your release build; it can only show you pictures of it.
- State that persists. Sessions, sign-ins and app data that are still there on the next run, the way a real device keeps them. Agents doing multi-day tasks need this; QA agents need the opposite (a clean install), and both should be explicit choices.
- Isolation. One phone per account, its own storage, network and IP, not a pooled device that someone else’s agent touched five minutes ago.
- A readable screen. Not only pixels. More on this below; it’s the most important point.
- The same inputs a person has. Tap, long-press, type, swipe, scroll, the hardware keys, plus app-level actions (launch, stop, open a URL, send an intent, install, uninstall).
The screen has to be readable as text
The obvious design is “give the model a screenshot and let it click coordinates”. It works in demos and degrades in production:
- A full-resolution phone screenshot (1440×3120) is about 3 MB as PNG. That is more than many MCP clients accept in one tool result, and it burns context on every step.
- The model has to estimate positions. On a dense settings screen that estimate is often a row off.
Android already has a structured description of the screen: the accessibility tree. Expose it, and put a short handle on everything actionable:
GET /device/:id/snapshot?format=text
- Group
- ScrollView [ref=1]
- Group
- Text [ref=5] "Contacts" (Contacts)
- Text [ref=7] "Camera" (Camera)Now the action is exact and cheap:
POST /device/:id/tap {"ref": 5}Screenshots are still needed, for WebViews, Flutter, games, canvas, anything the tree can’t see. Make them a deliberate second step, and small by default (a JPEG with the longest side around 1024 px is tens of KB). An endpoint that returns the image and the tree together (we call it /observe) saves a round trip when the agent genuinely needs both.
Handles must expire
The single most useful rule we added: refs reset on every snapshot.ref=5 means “item 5 in the snapshot you just read”, nothing more.
Without that rule, agents reuse handles from three screens ago and tap something unrelated, which is the worst kind of failure because nothing reports an error. With it, a stale handle fails loudly (ElementNotFound) and the model takes a new look.
For actions that should survive a screen change, offer a selector instead of a handle:
POST /device/:id/tap {"text": "Network & internet"}
POST /device/:id/tap {"contentDesc": "Settings"}The server takes its own fresh snapshot, picks the best match, and taps its nearest tappable parent (labels are often on a child of the thing that is actually clickable).
Coordinates need a frame of reference
When the agent does fall back to pixels, it usually read them off a scaled image. If the API silently assumes device pixels, a tap at (360, 640) on a half-size screenshot lands in the wrong quarter of the screen. The fix is boring and effective: let the caller say which image the coordinates came from.
POST /device/:id/tap {"x": 360, "y": 640, "of": [720, 1560]}Waiting is a first-class action
Phones are asynchronous. A tap that opens a screen returns before the screen exists. Agents that type immediately lose keystrokes into a field that isn’t focused yet. A wait primitive that blocks until some text appears (with a timeout and a distinct error, 408 WaitTimeout) removes a whole class of flaky retries:
POST /device/:id/wait {"text": "Internet", "timeout": 10000}One action at a time, per phone
A touchscreen is a single shared resource. If an agent fires ten taps in parallel, the result depends on scheduling, not on intent. We serialize actions per device and cap the queue (more than 8 queued requests gets 429 device_busy), while different phones run fully in parallel. That makes “run the same flow on five phones” easy and “race yourself on one phone” impossible.
Errors are for the model to read
Every error has one shape and a machine-readable code:
{
"ok": false,
"error": "forbidden",
"message": "device not found or not accessible with this key",
"requestId": "3c3eb4a4f9151d4d"
}Pass these through to the model unchanged. SelectorNotFound with the selector it tried leads the model to look again and pick a different label. no_launcher_activity tells it the package can’t be launched that way. A generic “action failed” just makes it retry the same thing.
One security detail worth copying: a device id that isn’t yours gets the same 403 whether or not it exists, so ids can’t be discovered by guessing.
What is still honestly hard
We’d rather you hear the limits from us:
- A cloud phone is a full Android system in an isolated container, not a handset. Apps that check for virtual devices can tell, and some banking apps and games refuse to run.
- No SIM means no phone number, no SMS, no cellular calls.
- Typing through adb is printable ASCII only: no emoji, no accented or non-Latin text.
- The accessibility tree is only as good as the app’s accessibility labels. Some apps are mostly unlabeled, and then you are back to pixels.
If you want to try this
The phones are in early access, by invitation. The MCP server that exposes all of the above as tools for Claude, Cursor and other MCP clients is MIT-licensed: github.com/Protremix/erius-phone-mcp.
If you’ve built your own phone tool for agents, we’d like to compare notes, especially on unlabeled apps.
