Let LLMs control Android apps in the background – open-source project, no root required
I built an open-source prototype that connects LLMs (via OpenAI-compatible APIs) to real Android apps – and it runs completely in the background.
The AI doesn’t just generate text – it actually does things: tap, type, scroll, long-press, copy/paste, go back. It’s like giving an LLM a phone to use, but on a virtual screen that doesn’t interfere with what you’re doing on your physical phone.
Key bits:
· Works with any OpenAI-compatible endpoint that supports streaming + tool calls
· The model receives UI node trees (or OCR results) and responds with tool calls
· All actions are executed on the virtual display – not your main screen
· Runs on any Android 10+ device – no root, no special hardware
If you’ve been wanting to experiment with phone-controlling agents but didn’t want to buy a dedicated device, this might be useful.
GitHub: https://github.com/android-notes/ShadowAuto
Demo video: https://github.com/android-notes/ShadowAuto/blob/main/shadowauto.mp4
Happy to discuss the architecture – the VirtualDisplay setup, the tool-call loop, OCR fallback, etc. Let me know what you think!