Evaluations providers

Turns specialist problem-solving into fine-tuning examples, grading rubrics, agent environments, and demonstrations of browser and desktop work.

Reconstructs factory stations as digital twins to test process changes and robotic interventions. The current offering is industrial simulation rather than a general-purpose AI dataset marketplace.

Provides AI red teaming, alignment data, training environments, and runtime guardrails, with a focus on adversarial behavior and model safety.

Builds verification for AI agents using expert judgment, enterprise knowledge, and encoded rules. Its published description emphasizes grounding and checking agent behavior.

Studies autonomous AI through real-world business deployments and simulation benchmarks, including retail operations and spatial or robotics tasks.

Generates training environments, tasks, and verifiers from real-world material for long-horizon agent development. Its website describes a limited partner beta.

Creates specialist training and preference data and runs model or workflow evaluations tailored to a customer’s domain or use case.

Provides human-validated data for model alignment, agent trajectories, speech, physical AI, and model evaluation.

Combines professional specialists, domain datasets, and task environments for AI training and evaluation in regulated knowledge work.

Runs a public model-comparison experience and leaderboard based on user judgments across AI outputs. Confirm commercial evaluation access separately.

Builds long-horizon simulated tasks and benchmarks for agent security, reliability, scientific work, and operational incident handling.

Builds agent-learning environments and benchmarks, including workplace simulations, robotics tasks, and a runtime for executing and verifying agent work.

Develops conversational audio datasets, annotation, voice-actor material, and speech-interaction research for teams building voice AI.

Builds production-style RL environments and infrastructure, with agent evaluation and optimization tools and open research datasets.

Provides crowdsourced security testing, vulnerability programs, and researcher expertise. Its broader security offering should be scoped specifically for an AI evaluation project.

Creates specialist-authored tasks and graders for training and evaluating models on finance, legal, computer-use, and responsible-AI work.

Prepares unstructured data and combines human validation, model oversight, and workflow orchestration to support operational AI systems.

Builds simulated work settings with users, tools, persistent context, and tasks for training and evaluating agents under changing conditions.

Uses creative professionals’ judgments to evaluate AI-generated work across media and creative tools, with benchmarks focused on quality and taste.

Develops specialized reinforcement-learning tasks for scientific reasoning and inference or performance engineering.

Develops Cerebro, a platform for turning operational records into fine-tuning data, reinforcement-learning material, and outcome-graded environments.

Builds data-collection infrastructure and curated datasets informed by human expertise, with research including the DeepSWE software-engineering benchmark.

Develops audio datasets for speech recognition, synthesis, translation, and conversational models, with research-led collection and quality review.

Supplies post-training datasets and environments for coding, multimodal systems, and physical AI, alongside enterprise agent services.

Offers a dataset marketplace plus custom collection, annotation, translation, and model-evaluation services for enterprise AI projects.

Builds reinforcement-learning environments for model-training teams. Its engineering work converts software changes, agent traces, and CI failures into containerized tasks with checkable rewards, difficulty calibration, and held-out testing.

Transforms institutional lending records into training examples, reference trajectories, graders, and financial-workflow environments.

Now operating as TrustScale, e2f provides custom training datasets, expert annotation, and model evaluation. Its current offering covers multimodal data, language quality, factual verification, reasoning benchmarks, and safety review.

Builds market-data-based environments for agents to form hypotheses, run experiments, and improve through increasingly difficult research tasks.

Constructs reproducible enterprise environments and training loops from seed data, including browser-based CRM, ERP, and commerce tasks.

Runs robot policies on real tasks, identifies failure states, and collects targeted demonstration and recovery datasets to address them.

Creates multi-agent security worlds with persistent services and verifiable outcomes for adversarial training and adaptive benchmarking.

Converts expert decisions, corrections, preferences, and verification evidence into datasets, private evaluations, and regression environments.

Uses software-engineering expertise to turn production workflows into tasks, rubrics, evaluations, and training environments with scoring and telemetry.

Turns selected games into interactive learning environments designed to exercise planning, memory, tool use, and collaboration.

Provides adversarial assessments and security controls for AI systems, combining vulnerability testing with filtering and behavioral monitoring.

Offers benchmark licensing, custom evaluations, and training-data services, including a benchmark for applying Indian financial regulations.

Builds finance-oriented training environments and benchmarks using professional workflows and anonymized transaction material.

Runs an AI expert program and research activity, with a separate partnership program for licensing company records for AI development.

Runs automated experiment loops against a customer-defined metric to improve models or algorithms, including generated training environments.

Offers an SDK and platform to build, run, inspect, and grade training environments; its site also connects environment creators to DataVendor.

Builds long-horizon training environments, computer-use models, and enterprise evaluations, with post-training data and infrastructure for AI labs.

Builds environments and graders for learning agents and offers selective technical research engagements from a small London research group.

Develops evaluation tasks, environments, and off-the-shelf datasets for long-horizon software work and other professional capabilities.

Provides expert-led annotation and generative-AI workflows, including feedback, reasoning data, and agent evaluation through its tools and services.

Offers data services for generative, agentic, and physical AI, including custom training environments, collection, and model-safety evaluation.

Combines expert networks, data preparation, evaluation tooling, and enterprise AI implementation services.

Builds environments used to train frontier AI and offers enterprise agent products for engineering and research workflows.

Publishes biology and biosecurity benchmarks for AI agents, with tasks spanning omics analysis, therapeutic research, and related scientific workflows.

Develops simulation-ready assets, first-person human data, and evaluation infrastructure for robotics and physical AI.

Offers expert-produced reasoning annotations, preference examples, and reinforcement-learning evaluations aimed at robot learning and world models. Public service detail is limited to the official site’s description.

Collects, labels, transcribes, and evaluates text, audio, images, and video. Its services include ready-made datasets, custom collection, supervised fine-tuning examples, human preference feedback, and model red-teaming.

Processes robot and first-person recordings into timestamped action segments, reconstructed hand motion, and automated quality checks. Its pipelines assess visual quality and task completion in training and deployment footage.

An industrial-systems research lab focused on benchmarking whether models can operate industrial processes safely. Its public homepage links to research but provides limited detail on commercial deliverables.

Collects first-person activity data and enriches it with motion, geometry, contact, and scene information. Its robotics-focused services include hand and body tracking, sensor capture, and model evaluation.

Builds graded software-engineering environments where coding agents implement features, debug unfamiliar projects, and deploy applications. Task scores serve as reinforcement-learning feedback and evaluation measurements.

Combines human-activity datasets, task-specific collection, evaluation tools, and deployment support for embodied systems. Its Egoverse platform provides first-person interaction data and tools for inspecting robot-learning datasets.

Supplies professional expertise, human evaluations, benchmarks, and environments; it also offers a business-workflow data monetization program.

Develops expert data, realistic training environments, contextual agent evaluations, and demonstration data for robotics.

Provides first-person activity datasets and gripper demonstrations alongside simulation infrastructure. Offerings combine video, inertial signals, pose information, and annotations with tools for testing and training robot policies.

Builds industry-specific benchmarks, develops training data, and runs model post-training and deployment. Projects begin with company workflows, expert practice, and historical cases to define measurable task performance.

Turns organizational workflows, outcomes, and expert judgments into model-training signals. Its platform supports evaluations, fine-tuning, reinforcement learning, and ongoing improvement from production interactions.

Creates simulation assets and digital twins to evaluate robot policies and generate targeted training experience. Its workflow connects failure analysis, domain variation, and reinforcement learning with physical deployments.

Recruits and manages domain specialists for model evaluations, post-training examples, and independent annotation audits. Deliverables include preference comparisons, demonstrations, rubric scores, and quality-assurance reporting.

An applied research lab developing data and evaluations for AI-assisted hardware design, including aerospace, robotics, medical-device, and semiconductor work. Public material emphasizes research and benchmarking rather than a detailed catalog.

Builds reinforcement-learning environments from scientific and technical research problems. Its stated approach uses verifiable tasks and formal methods across machine learning, computational science, robotics, and cybersecurity.

Sources licensed business datasets and connects AI teams with domain experts. Services include identifying data owners, agreeing permitted uses, preparing records, and developing expert tasks and evaluation criteria.

Designs multi-agent simulations for training and assessing AI systems. Environments explore negotiation, coordination, enterprise tasks, and emergent behavior under explicitly chosen incentives and participant roles.

Transforms company documents, communications, and tool records into privacy-processed digital twins. These support multi-step agent tasks and evaluations that preserve relationships across organizational systems.

A talent marketplace and managed-service provider for AI data work. Teams can source specialists for annotation, model evaluation, preference feedback, and safety testing while retaining their existing tools.

Creates custom engineering problems, solution traces, and reasoning annotations with senior practitioners. Datasets are calibrated to model difficulty and can support fine-tuning, preference learning, or held-out evaluation.

Researches model failures and converts them into training tasks, synthetic experiences, and simulated environments. Published work spans software engineering, computer use, knowledge work, alignment, and scientific reasoning.

Runs expert-led data and evaluation programs, supplies specialists, and collects embodied multimodal recordings. Its work includes professional-domain assessment, safety testing, multilingual benchmarks, and labeled robotics sensor data.

Converts real-world data into simulated software environments for training and evaluating agents. Its public example combines multiple business tools with loadable tasks and scored agent rollouts.

Provides expert annotation and evaluation services alongside its Syncronus collaboration platform. Work covers difficult reasoning, tool use, agent behavior, safety testing, and peer-reviewed specialist workflows.

Packages tasks mined from licensed private code changes into runnable environments with graded checks, hidden reference solutions, and solver trajectories. Delivery includes verification evidence and reproducibility materials.

Develops reinforcement-learning environments with an emphasis on robust reward functions and hardened graders. Public task examples include coding kernels, strategic games, mathematical reasoning, and image-data curation.

Offers an integrated stack for model training, evaluation, inference, and compute. Its environment hub and open-source tooling help teams create tasks, execute rollouts, and run reinforcement-learning workflows.

Connects researchers and AI teams with verified human participants. AI use cases include collecting preference judgments, safety feedback, and evaluation data from selected populations rather than relying solely on automated metrics.

Positions its offering around data, evaluation, and deployment support for physical AI. The official homepage confirms this focus but does not yet describe dataset specifications or detailed commercial packages.

Researches data and engineering systems for technically capable AI agents. Public work includes long-horizon coding benchmarks, cybersecurity evaluation analysis, and infrastructure for running agentic reinforcement learning.

Provides coding-agent usage and cost analysis, with public benchmark and evaluation work. Its site also documents prior development of long-duration training environments and detailed analysis of agent trajectories.

Combines domain data and expert review to produce reasoning datasets, sandboxed training tasks, and adaptive evaluations. Coverage includes scientific problems, software engineering, financial analysis, and physical-system sensor data.

Builds simulated software environments and verifiable tasks for agent training and evaluation. Its published coverage includes connected workplace applications, coding tasks, and Blender-based design and animation workflows.

Uses software engineers to develop code-focused datasets, expert demonstrations, evaluation tasks, and training environments, alongside its engineering talent business.

Combines subject-matter experts with managed workflows and automation to deliver human feedback, labeled examples, and model evaluations.

Builds extended reinforcement-learning tasks across software engineering, computer interaction, cybersecurity exercises, and incident response. Environments combine realistic systems with grading of task execution and diagnostic performance.

Provides code sandboxes and evaluation infrastructure for developing and deploying AI agents, including options within a customer’s cloud account.

Delivers human-verified annotation, data validation, and model evaluation for computer vision and multimodal AI.

Reviews AI training data against agreed criteria using AI judges calibrated and checked against human experts.

Provides AI training data and human expertise alongside evaluation, enterprise, and public-sector AI services.

Develops a cloud-based factory for producing and standardizing agent training and evaluation data, according to its company profile.

Offers AI data collection, annotation, and model-evaluation services across speech, healthcare, and other multimodal applications.

Researches data-centric AI and develops expert-data methods, training systems, and benchmarks for advanced model and agent capabilities.

Builds procurement environments and benchmarks that exercise supplier selection, purchasing policies, approvals, and invoice-resolution tasks.

Combines multimodal annotation, data orchestration, human review, and evaluation tooling, including support for agent trajectories and RL environments.

Offers off-the-shelf data, expert workforce services, post-training work, and benchmarks across professional tasks.

Develops post-training datasets and benchmarks and uses enterprise records to model operational processes and deploy AI workflows.

Builds research tools, evaluations, and environments for models working on empirical science and automated discovery.

Provides AI data operations, feedback, evaluation, and safety services alongside broader business-process and autonomous-system operations.

Offers an API that extracts brand systems, retrieves style references, and checks whether agent output follows a brand’s visual direction.

Offers data collection, specialist post-training work, annotation, safety evaluation, and off-the-shelf datasets across language, vision, and physical AI.

Builds private evaluations and training tasks from complete creative workflows, beginning with video editing and the TimelineBench benchmark.

Creates AI training material, evaluations, and environments from industrial companies’ operational records.

Captures production agent traces, recreates hard cases as training environments, and feeds resulting evaluation signals into custom-model improvement.

Creates agent-training data, simulated environments, human feedback, and evaluations covering tool use, coding, and model safety.

Provides AI safety evaluations, red teaming, and training environments through researchers and engineers working on frontier-model risks.

Offers data collection, annotation, testing, and human-in-the-loop validation for generative and physical AI through its operations platform and contributor network.

Develops reasoning datasets, expert-reviewed mathematical work, and private benchmarks that capture partial progress as well as final answers.

Publishes model comparisons and benchmarks based on professional tasks across coding, finance, law, and scientific work.

Creates simulated versions of systems, APIs, data, and users for agent testing, training, and software development before production use.

Connects vetted domain specialists with AI projects involving response review, scenario judgments, and practitioner-written problem approaches. Expert matching considers relevant experience and demonstrated depth in the required specialty.

Designs model-learning tasks, rubrics, verifiers, feedback, and evaluations across professional and multimodal work.