Most teams put safety rules in the prompt and hope the model obeys. The Buddy project does the opposite, and it works better. Buddy is an open-source Android app where an AI agent (Claude Code, Codex, OpenCode or Google’s Antigravity CLI) reads your phone screen, taps and types. After a few weeks of real use, its builder shared five lessons. Here they are, as a guide you can reuse for any tool-using agent.
The Key Idea
A prompt is a suggestion. The app around it is the enforcement. When the two disagree, the prompt can win, and that is where things break. Think of the prompt as a polite request and the code as a locked door. A polite request works most of the time. A locked door works every time, including the day the model gets creative, confused or overconfident. The goal is not to distrust the model. The goal is to make sure the worst case is a blocked action instead of a bad one.
Old Way vs New Way
1. Safety rules
- Old way: put “confirm before changing settings” in the prompt. It reads clearly, it feels safe, and it is only as reliable as the model’s mood on a given run.
- New way: enforce it in the app. One user set their camera app to “Trust” and still got asked before every switch, because the prompt overruled the lock. Now send, pay and delete buttons in messaging, shopping and banking apps stop on a card until you approve. A model once picked “Buy now” instead of “Add to cart”. That check is why nothing got ordered. The card shows what the agent is about to press, so you approve a specific action, not a vague intention.
2. Tool descriptions
- Old way: list tool names and trust the model to figure out the rest. A name like
taptells the model almost nothing about arguments, optional fields or limits. - New way: show the signatures. When Antigravity was added, Gemini ignored the phone tools, tried
lsin a shell, got refused and quit silently. A short first-message note fixed it: signatures liketap(element?, x?, y?)plus the line “Shell commands are not available in this chat”. Notice the second half. Saying what is missing prevents the model from reaching for habits it learned elsewhere.
3. Refusals
- Old way: let the agent stall when a tool call is refused. Many models treat a refusal as a dead end and simply stop, with no error and no explanation.
- New way: send one nudge that starts with “(From Buddy, not the user)”. The model then knows the app is talking, not the user changing the task. Keep the nudge short and practical, for example: “That call was blocked. Try another tool or ask the user.” One clear sentence beats a paragraph of apology.
4. Where answers come from
- Old way: assume the agent will read the answer off the screen.
- New way: say it explicitly. In a benchmark, one model couldn’t get Calculator working and answered 1234 × 5678 from its own math. Asked for the current temperature, it twice returned today’s high and low. Models fill gaps with what they already know. The answer looked right, which is exactly why it is dangerous. If the number must come from the app, the prompt has to say so, and your tests should check where it came from, not only whether it matches.
5. Screen input
- Old way: send screenshots to a vision model for everything. It works, but every image costs tokens and time, and small text can get misread.
- New way: read the accessibility tree as text. It is faster and cheaper, and it gives exact labels instead of pixel guesses. Keep screenshots for truly visual things like maps. 🗺️
Practical Steps
- List every action in your agent that is hard to undo (send, pay, delete, publish). Gate each one in code, not in wording. Start with the single scariest button and work down.
- Paste real tool signatures into the first message, and name what is not available.
- Write a stock recovery message for refusals, and label its sender.
- For any task where the answer must come from a tool, write “the answer must come from X” in the prompt. Then spot check a few runs to confirm the agent really used X.
- Default to text-based inputs. Add images only when text can’t carry the information.
Why It Matters
Each lesson moves a decision out of the model’s guesswork and into something you control. That is the pattern: less hoping, more structure. ✅ The payoff is predictable behavior, fewer silent failures and cheaper runs, which is what you want from anything that touches your phone, your money or your messages.
The project is free and open source at github.com/ghorbelhamdi/buddy-android. Pick one of the five steps above, apply it to your own agent this week, and see which failure disappears first.
What I learned prompting AI agents to use a real Android phone
by u/3amae404 in PromptEngineering