EyeSense
AI visual assistance for blind and low-vision Android users
An Android-first Flutter app that describes scenes, reads printed text aloud fully offline, and answers spoken questions about a captured image. It was built around TalkBack, explicit consent before any image leaves the phone, and clear limits on what the AI may say.
At a glance
- Status
- Mobile prototype
- Type of work
- Innovation showcase project
- Context
- MUA Innovation Week 2026
- Years
- 2026
- Categories
- Mobile & Accessibility · Applied AI & Agents · Civic & Social Impact
- Source code
- Public repository
Technology
Problem
Blind and low-vision people need fast, spoken descriptions of printed text and their surroundings, through controls that work with a screen reader. They also need honesty about what an AI system cannot safely tell them.
Who it’s for
Blind and low-vision Android users. The interface is available in English and Kiswahili.
What I built
- Describe Scene, Read Text and Ask a Question flows, with Stop and Repeat controls throughout.
- Read Text runs fully offline with Google ML Kit, so printed text works without a connection.
- TalkBack semantics, interruptible text-to-speech, speech commands, a maximum-contrast theme, Atkinson Hyperlegible type and support for 200% text scale.
- A Cloudflare Worker that holds the shared provider key and enforces authorisation, rate limits, concurrency caps and a budget cap.
- Optional bring-your-own Groq key, kept in secure storage on the device.
- Explicit consent before an image is sent for cloud processing, and an output safety supervisor that filters what is spoken.
Architecture
- CaptureCamera snapshot on the device; nothing is uploaded without consent.
- On deviceGoogle ML Kit OCR for Read Text, fully offline.
- Cloud (opt-in)Cloudflare Worker → Groq vision model for scene description and follow-up questions.
- SafetyOutput policy supervisor removes unsafe guidance, such as navigation or road-crossing instructions.
- SpeechInterruptible text-to-speech designed to work alongside TalkBack.
Engineering challenges
Not over-promising
The app refuses navigation and road-crossing guidance. A vision model cannot judge traffic or distance reliably enough for that.
Protecting a shared API key in a mobile app
The provider key lives in the Worker, not the APK. The Worker adds authorisation, rate limits, concurrency caps and a spending cap, and users can bring their own key instead.
Working without a connection
Text reading uses on-device ML Kit, so the most common task keeps working offline.
Quality and trust
- The README reports 24 Flutter tests and 10 Worker tests passing.
- Consent prompt before cloud processing; secure storage for user-supplied keys.
Results
| Measure | Value | Evidence |
|---|---|---|
| Automated tests | 24 Flutter · 10 Worker | README-reported |
Limitations
- Prototype: on-device TalkBack testing and a formal contrast audit are still pending.
- The ReadingAgent summarisation step is a stub and has been deferred.
- No public APK has been released; this page describes the engineering, not a store listing.
- Not a navigation aid.