Voice-Driven AI Assistant for Web-Based ERP Systems

Task

Working in an ERP means many screens and many fields, and an assistant usually needs an integration with each system. We built EFRION AI Autopilot, our own product, to test whether one assistant can work on a web-based ERP as it is: no configuration, no integration and no API keys on the customer's side.

What we did

• Built the assistant as a Chrome extension on Gemini Live: the user speaks a request, and the extension streams the audio.
• Sent the model a screenshot every 1.5 seconds, together with a simplified accessibility tree of the page.
• Connected a FastAPI backend over a two-way WebSocket to turn the model's function calls into six page actions: click, type, scroll, navigate, read and highlight.
• Made the assistant announce its plan in an on-screen panel and show each target with an animated cursor before it clicks.
• Added a check after every click that the page actually changed, with another attempt in a different way if it did not.
• Added a spoken "undo" that reverses the last of up to ten actions, and an optional safety lock that asks before submitting, deleting or leaving a page.

Result

Public at ai.efrion.com: the extension can be downloaded, and a demo shows spoken commands carried out step by step in an ERP screen.