On-Device UI Clicking with Jev

Jev uses a local CoreML model to segment on-screen buttons and UI elements, while on-device OCR reads their labels. It scores the elements to choose one to click, then clicks and reruns detection.

View on X

Milind S@milindlabs𝕏

Okay so Jev can actually do computer use really well Without any screenshots, or LLMs and no Pixels leave my mac I dont even read the Dom elements A local CoreML model segments every button and UI element on screen. On-device OCR reads the labels. That text is all Jev gets. It returns a probability across those elements and tells me the best one to click. Then it clicks, re-runs detection, and dec

Sep 17, 2026X postsView on X

It uses no screenshots or LLMs, reads no DOM elements, and keeps pixels on the Mac.

Also filed under Agents & browsers

Evaluates Tool-Using Agents with Jev in CI
The project evaluates a real tool-using AI agent using code checks, a calibrated Jev judge, and an LLM judge. It is wired into CI.
Watchdog for AI Coding Agents
Jev Watch is a watchdog for AI coding agents that stops Codex or OpenCode when they loop, stall, or drift off task, then resumes the same session with a correction.
Custom Agent Harnesses with Jev
Elvis uses Jev to build custom harnesses and reports success with guardrails, routing, and verifiers.
Fraud Investigation Agent
The author is building a fraud investigation agent using Jev and TigerGraph. Its flow goes from an alert through graph evidence, a Bayesian update, a pattern, Jev’s extra lookup, policy, a verdict, and human approval.