Voice-Driven AI Assistant That Operates Any Web-Based ERP

Task

Working in an ERP means many screens and many fields, and an assistant usually needs an integration with each system. EFRION, a B2B software company, wanted one that works on any web-based ERP as it is: no configuration, no integration and no API keys on the customer's side.

What we did

• Built the assistant as a Chrome extension on Gemini Live: the user speaks a request, and the extension streams the audio.
• Sent the model a screenshot every 1.5 seconds, together with a simplified accessibility tree of the page.
• Connected a FastAPI backend over a two-way WebSocket to turn the model's function calls into six page actions: click, type, scroll, navigate, read and highlight.
• Made the assistant announce its plan in an on-screen panel and show each target with an animated cursor before it clicks.
• Added a check after every click that the page actually changed, with another attempt in a different way if it did not.
• Added a spoken "undo" that reverses the last of up to ten actions, and an optional safety lock that asks before submitting, deleting or leaving a page.

Result

Public at ai.efrion.com: the extension can be downloaded, and a demo shows spoken commands carried out step by step in an ERP screen.