Optima tackles AI benchmarking's biggest flaw by letting users test models against their own data

Optima lets users build custom benchmarks and compare AI models on quality, cost, and speed for their specific use cases.

Artificial Analysis, known for its independent LLM evaluations and benchmark implementations like GDPval-AA and AA-Briefcase, has launched a new platform called Optima. The premise is simple: public benchmarks compare models on predefined tasks and criteria, but they don't necessarily reveal which model works best for a particular use case. Optima aims to close that gap with comparisons tailored to individual workflows.

Users can build their own benchmarks using their own data, workflows, or descriptions of a use case, then run them across leading current models and compare results on quality, cost per task, and time per task, according to Artificial Analysis. Optima is available now.

Three paths to a custom benchmark

Optima accepts several types of source material. Users can upload existing evaluation datasets from their own files or from Hugging Face, as well as AI agent traces from platforms like Arize, Braintrust, or Langfuse. Developers can also install a skill that gathers information from their coding environment and past sessions, Artificial Analysis writes.

Users who don't have that kind of data can instead describe their intended use case and provide sample inputs and outputs. Optima then generates suggested test inputs, evaluation criteria, and example tasks. Users can review and refine these through feedback before running the actual benchmark.

Two scoring approaches are available: rubric-based evaluation against objective criteria, or a pairwise comparison method that Artificial Analysis also uses for benchmarks like GDPval-AA and AA-Briefcase. In the pairwise approach, users first evaluate a sample of response pairs and indicate which answer they prefer. Optima then derives the full ranking across the test dataset from those preferences.

Cost and speed become first-class comparison metrics

Beyond raw model quality, Optima tracks cost per task and time per task as standalone comparison dimensions. This makes it possible to check whether a performance gain actually justifies the higher cost or longer processing time of a given model.

For agentic applications, raw token price alone tells you very little. A cheaper model can end up costing more overall if it needs more attempts, fails more often, or requires extra cleanup work. Cost per completed task is often the more meaningful number.

Early testers built benchmarks for finance and accounting agents to find which model could cut costs by a factor of ten without major quality loss, according to Artificial Analysis. Others tested which model best matched the writing style of lawyers or most accurately identified elements in a proprietary image dataset.

When building and running benchmarks, Optima charges only the actual token costs of the models used, with no markup, according to Artificial Analysis. Rubric-based evaluations cost $0.125 per criterion per model, and pairwise evaluations cost $0.375 per comparison. At the start of benchmark creation, each benchmark run, and each evaluation round, the platform holds a balance based on a cost estimate. Billing is then based on actual usage and evaluation costs incurred.

Why general-purpose benchmarks fall short

Optima tackles a well-known problem with AI benchmarks. An analysis by Epoch AI showed that benchmark results depend on implementation details that are rarely disclosed. Different prompt wording and temperature settings caused the same model to score noticeably differently depending on configuration. For agentic benchmarks like SWE-bench, simply swapping the scaffold, meaning the agent's control software and tool environment, accounted for up to 15 percentage points of difference.

An even broader study examining 445 benchmark papers from leading AI conferences found more systemic issues. Nearly all had methodological weaknesses in at least one area, including unclear definitions, unrepresentative samples, and missing statistical validation. Only about 10 percent of the benchmarks studied used complete real-world tasks that reflected actual application scenarios. Key concepts like reasoning or alignment were often poorly defined, limiting the reliability of any conclusions drawn from them.

Optima can address the problem of general benchmarks failing to capture a specific use case. But the deeper methodological challenges of benchmarking don't go away. Even with a benchmark tailored to your own tasks, its usefulness depends on how precisely the target capabilities are defined, how representative the test cases are, and how the evaluation is implemented and documented.

There's another limit worth keeping in mind. Even cost and time per task don't tell you what the output is actually worth to the business. A cheap and fast AI workflow can still be inefficient if its results need heavy rework or add little value to the process they're part of.

AI News Without the Hype – Curated by Humans

Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.