Computer-use training & evaluation index
Computer-use RL environments, training data & evaluations.
Compare evidence for providers of interactive GUI tasks, resettable environments and verification for model training. This snapshot covers a selected set of specialist commercial suppliers with publicly reviewable evidence.
LEADERBOARD · NOVEMBER 2026
Computer-use training and evaluation providers
7 ranked specialists · 11 tracked (NR). Expand an entry for its role, evidence and sources. Equal assessments share a rank.
=1 Chakra Labs↗Supplies enterprise app replicas and cross-application tasks for GUI and tool-use agents. RL environmentsBenchmarksBrowser GUI MAGI benchmark and Dojo application replicasPublished evidence CommercialBrooklyn, US Documented
Supplies enterprise app replicas and cross-application tasks for GUI and tool-use agents.
MAGI publishes held-out results for seeded workflows across eight enterprise apps, with deterministic checks validated against human reference runs. Dojo supplies runnable app code. Controlled training gains and outside execution remain unverified.
=1 Plato↗Enterprise computer-work environments for testing and training. RL environmentsBenchmarksBrowser GUI Computer Work Val, verifier QA and session resetPublished evidence CommercialSan Francisco, US Documented
Enterprise computer-work environments for testing and training.
Computer Work Val reports results on 300 enterprise tasks. Plato documents simulator resets and reviews tasks for reward hacking and grading mismatches. Training access is available by request.
03 HUD↗Computer-use evaluation and agent training infrastructure with public spreadsheet examples. RL environmentsBenchmarksBrowser GUI SheetBench-50 methods and browser verificationPublished evidence CommercialSan Francisco, US Documented
Computer-use evaluation and agent training infrastructure with public spreadsheet examples.
SheetBench-50 reports outcomes for live spreadsheet workflows, using cell-level checks and blind expert validation with Sepal AI. HUD’s runner captures trajectories and rewards for training. Independent execution was not verified.
04 Collinear AI↗Offers a runnable training corpus and evaluates agents on held-out enterprise GUI workflows. RL environmentsBenchmarksTraining data FrontierCUA seeded workflows and outcome rubricPublished evidence CommercialSan Francisco, US Documented
Offers a runnable training corpus and evaluates agents on held-out enterprise GUI workflows.
FrontierCUA reports 15 seeded evaluation workflows with state-based grading, including a Zendesk-to-Sheets task. A separate corpus of 3,000+ runnable tasks is available on request. Controlled training gains and independent execution remain unverified.
05 Halluminate↗Finance workflows across connected applications with outcome checks. RL environmentsBenchmarksBrowser GUI Westworld Finance Diligence BenchPublished evidence CommercialSan Francisco, US Documented
Finance workflows across connected applications with outcome checks.
The benchmark documents multiple finance task types across distinct apps. Agents interact with stateful apps and produce multistep deliverables. The methodology documents seeded task containers and verifier QA. Repeated reset behavior was not verified. The benchmark documents reproducible evaluation. Usable training assets were not verified.
06 Cua (Cua Bench)↗Desktop sandbox tasks with programmatic scoring and a documented RL integration. RL environmentsBenchmarksDesktop GUI Task lifecycle and GRPO integrationPublished evidence CommercialSan Francisco, US Documented
Desktop sandbox tasks with programmatic scoring and a documented RL integration.
The desktop task framework supports multiple operating systems and a registry of distinct application tasks. Tasks run against desktop software in sandboxes. A representative cross-app training workload was outside this review. Each variant starts a sandbox and runs setup followed by an evaluator. The API supports a reference solution. The documented RL package includes parallel workers and a Tinker GRPO trainer. Controlled held-out gains were not verified.
07 Refresh↗Connected software environments and versioned Blender outcome checks. RL environmentsBenchmarksDesktop GUI BlenderBench methods and commercial training offeringPublished evidence CommercialSan Francisco, US Documented
Connected software environments and versioned Blender outcome checks.
BlenderBench documents workflows in Blender. Tasks involve realistic application artifacts. Versioned checks inspect saved Blender files, protected scene state and renders. BlenderBench documents repeatable evaluation with saved outputs and versioned verifiers.
NR AfterQuery↗Browser and desktop environments paired with expert demonstrations. RL environmentsTraining dataBrowser GUI Commercial CUA offeringEvidence gap Commercial + OSSSan Francisco, US Limited
Browser and desktop environments paired with expert demonstrations.
AfterQuery describes a CUA offering. The reviewed sources did not establish a named GUI training task with sufficiently documented methods and outcomes.
NR Andon Labs↗Builds simulated business evaluations and runs agent-operated retail experiments. Benchmarks Vending-Bench 2 and Project VendEvidence gap CommercialNot listed Limited
Builds simulated business evaluations and runs agent-operated retail experiments.
The published work establishes long-horizon business evaluation and outside collaboration. This review did not establish that the cited results measured GUI-based interaction, so the company is tracked as adjacent evidence.
NR Fleet AI↗Commercial training gyms that simulate software workflows. RL environmentsBrowser GUIDesktop GUI Training gym offeringEvidence gap Commercial + OSSNew York, US Limited
Commercial training gyms that simulate software workflows.
Fleet describes training gyms. The sources reviewed did not establish a completed GUI task with sufficiently documented methods and outcomes.
NR Huzzle Labs↗Desktop and browser environments with end-state grading. RL environmentsDesktop GUIBrowser GUI Named CUA examples and vendor-reported model trainingEvidence gap CommercialLondon, UK Limited
Desktop and browser environments with end-state grading.
The vendor reports model training and end-state rewards. The reviewed sources lack enough detail to connect a named workflow, its outcome checks and its results.
NR Labelbox↗Offers training gyms and evaluations for GUI tasks across enterprise software. RL environmentsBrowser GUI Horizon computer-use offeringTracked incumbent CommercialSan Francisco, US Limited
Offers training gyms and evaluations for GUI tasks across enterprise software.
The commercial offering describes GUI task execution. A named completed CUA evaluation with inspectable methods and outcomes was not verified.
NR Matrices↗Listed as a supplier of browser-native training environments. RL environmentsBrowser GUI Commercial offering listedEvidence gap CommercialNot listed Limited
Listed as a supplier of browser-native training environments.
RL List identifies a computer-use offering. The provider site could not be inspected during this review, and a named artifact with methods and results remains unverified.
NR Mercor↗Combines expert task authoring with simulated training environments. RL environmentsTraining data Deeptune acquisition and environment offeringTracked incumbent CommercialSan Francisco, US Limited
Combines expert task authoring with simulated training environments.
The acquisition announcement describes simulated software, tasks and verifiers. This review did not establish a completed attributable GUI evaluation with detailed methods and outcomes.
NR rlsupply↗Advertises desktop and browser workflows with application-state grading. RL environmentsDesktop GUIBrowser GUI Vendor specificationEvidence gap CommercialNot listed Limited
Advertises desktop and browser workflows with application-state grading.
The supplier describes persistent state and deterministic grading. This review did not inspect a named completed task packet or its reset and verifier.
NR Scale AI↗Commercial desktop and web environments and GUI-based occupational evaluations. RL environmentsBenchmarksDesktop GUI Remote Labor Index methods and outcomesTracked incumbent CommercialSan Francisco, US Documented
Commercial desktop and web environments and GUI-based occupational evaluations.
Remote Labor Index documents evaluated projects across occupational domains. The benchmark uses professional deliverables and a computer-use scaffold for supported models. The paper documents a standardized execution setup and deliverable grading. This review did not establish shortcut controls across the GUI offering. The benchmark documents repeatable evaluation. The product describes training support, with no controlled CUA training gains verified here.
NR Snorkel AI↗Supports long-horizon computer-use evaluation across web and desktop applications. BenchmarksDesktop GUIBrowser GUI OSWorld 2.0 research and data contributionTracked incumbent CommercialRedwood City, US Documented
Supports long-horizon computer-use evaluation across web and desktop applications.
OSWorld 2.0 reports completed runs and state-based grading. XLANG Lab developed the benchmark, with Snorkel credited as a research and data partner.
NR Turing↗Browser and desktop simulations with state-based verification. RL environmentsBenchmarksTraining data Agentic Environments methods and delivered web task suiteTracked incumbent CommercialPalo Alto, US Documented
Browser and desktop simulations with state-based verification.
Turing documents enterprise and consumer applications with measured evaluation outcomes. Documented workflows use seeded app data and browser or desktop interactions. Turing documents state-based checks and dynamic verification. Shortcut controls were not verified across the suite. The case study documents delivered verifier-backed training tasks. Controlled gains on held-out CUA tasks were not verified.
Methodology
The leaderboard covers commercial suppliers of computer-use environments, training data, benchmarks and evaluations for AI developers. A specialist earns a rank when a substantial technical release or completed evaluation provides verifiable methods, outcomes and attribution. The ranks apply to the vendors reviewed here.
We assess each provider’s strongest documented contribution for each factor. Evidence may come from different products or releases. We combine the five factor assessments to determine order. Equal overall assessments share a rank. Broad AI data and services incumbents are tracked as NR regardless of evidence depth. Specialists with unresolved evidence gaps are also tracked as NR.
Open, restricted and confidential work qualify on the same terms. For restricted work, a technical report or customer account can establish methods, outcomes and attribution.
Company age, size, reputation, customer logos and roadmap announcements earn no credit.
Documented means the work clears the technical evidence threshold. Corroborated adds confirmed execution or use by an outside organization. Limited means an eligibility gap remains. The badges summarize the evidence and carry no separate ranking weight.
- 01
Technical substance, depth and scope
Look for completed, attributable work with results across distinct workflows, application states or skill levels. A workflow that checks one outcome counts once, even if it takes many steps.
- 02
Interaction realism
Look for realistic application behavior, persistent state and constraints such as permissions or dependencies between apps. A large task set can still cover a narrow interaction surface.
- 03
Reset and verification
Check how the starting state is restored, what the grader inspects and how the suite handles unintended changes, shortcuts, grader errors and training data leakage.
- 04
Training utility
Evidence ranges from static demonstrations to repeatable evaluation, runnable training tasks or integrations, and controlled gains on held-out tasks. A benchmark result measures performance under that evaluation setup. A training comparison measures improvement.
- 05
External validation
Distinguish provider-run results, confirmed technical collaboration, completed outside use and independent runs under different conditions. Multiple references to the same run count as one result.
Computer-use AI training FAQ
What to look for when comparing computer-use tasks, environments and evidence.
What is a computer-use RL environment?+
A computer-use reinforcement learning environment gives an agent an interactive browser or desktop, a task, a defined starting state and a way to score the outcome. The agent sees the interface and acts through controls such as clicks, typing and scrolling. For example, it might match a purchase order to its invoice and mark the payment approved in an accounting app. The environment lets the agent attempt the workflow repeatedly and returns feedback based on the saved result.
How does a computer-use RL environment differ from a benchmark or a demonstration dataset?+
An RL environment lets an agent act repeatedly and receive rewards from the resulting state. A demonstration dataset records how a person or agent completed a task and can support imitation training. A benchmark fixes the tasks and grading protocol so teams can compare runs. If one platform serves several purposes, hold out evaluation tasks and, where practical, workflow families or application configurations.
What training tasks help agents learn to work across browser and desktop apps?+
Useful suites span distinct outcomes, from editing a document or cleaning a spreadsheet to reconciling records across email, ticketing and CRM apps. A cross-app task might require finding a vendor in a procurement system, checking its contract in a document store and updating a shared budget. Check which steps require GUI actions, which allow APIs and whether changes made in one app carry through to the others. These details determine which agent capabilities the task exercises.
How do teams reset computer-use environments between agent rollouts?+
A reproducible reset restores the task’s starting data, files, accounts, permissions and relevant application settings. Check for leftover state in shared services as well as in the browser. A practical test is to complete a task, reset it and repeat from the same seed. Compare the initial state and the verifier outcome on each run. For tasks that depend on live websites or external services, record which parts of the starting state the harness controls.
What makes a computer-use task reliably verifiable?+
A verifier checks the result against the brief, typically by inspecting a saved file or application state. For a spreadsheet task, it might check the requested values, their location and whether protected sheets remained intact. Test valid solutions that take different paths, incomplete runs, and shortcuts such as writing the expected value into the wrong sheet. If an AI judge handles open-ended criteria, compare its decisions with expert judgments on a sample of borderline cases.
How can teams prevent reward hacking and evaluation contamination?+
Test whether the agent can earn a reward without completing the intended workflow. For example, a spreadsheet grader should reject an expected value written to the wrong cell or obtained by modifying protected inputs. Keep reference answers out of the agent’s view and split training and evaluation by workflow family, application state or underlying records where practical. Review suspicious passes against the saved state and interaction trace.
How can an AI lab tell whether an environment improves training?+
A runnable task set, reward function or training integration shows what a lab can use. A controlled comparison on held-out tasks shows whether training improved performance. Look for the base model or other baseline, the training tasks, the evaluation split and the conditions used for both runs. If a provider reports gains only on tasks seen during training, the result leaves transfer to unfamiliar workflows open.
When should an AI lab build versus buy computer-use training environments?+
Build when tasks depend on proprietary applications, internal permissions or close integration with the lab’s training stack. Buy when the team needs broader application coverage, specialist task authors or faster access to verified workflows. In a pilot, run representative tasks twice from the same starting state, inspect successful and failed verifier decisions, and check whether rewards and trajectories export into the training stack. Compare what it takes to maintain and refresh the task suite after the pilot.