Part I: Let`s make GDPval explained: how AI performs across real-world jobs. What is GDPval?
Let`s make GDPval explained: how AI performs across real-world jobs and occupations. The rapid development of artificial intelligence has created a central question for businesses, workers, and policymakers: how capable are current AI systems when they are asked to perform work that professionals actually do?
GDPval, introduced by OpenAI, approaches this question by evaluating AI models on real-world, economically valuable tasks rather than relying primarily on academic examinations, synthetic reasoning questions, or measures of AI adoption. The benchmark covers 44 occupations across nine major sectors of the U.S. economy, with tasks designed around professional work products such as legal documents, engineering designs, financial analyses, presentations, spreadsheets, customer-service responses, medical and nursing materials, and multimedia content. Let`s analyze the latest OpenAI research “GDPVAL: EVALUATING AI MODEL PERFORMANCE ON REAL-WORLD ECONOMICALLY VALUABLE TASKS”.
The full benchmark contains 1,320 tasks, with at least 30 tasks for each occupation. A separate gold subset contains 220 tasks. The tasks were created and reviewed by experienced professionals, whose average professional experience was approximately 14 years. The benchmark therefore attempts to measure not merely whether an AI can produce a plausible answer, but whether it can produce a deliverable that an experienced professional would consider useful.
GDPval’s broader objective is to provide a more direct measure of AI’s potential economic relevance. Its results suggest that frontier models are increasingly capable of completing substantial professional tasks, while also revealing important weaknesses involving accuracy, formatting, instructions, context, and the need for human review.
Part II: What GDPval Measures?
GDPval was designed to check real-life scenarios and address several weaknesses in traditional AI benchmarks.
First, it emphasizes realism. Instead of asking models questions resembling academic tests, the benchmark uses tasks derived from actual professional work. The tasks were reviewed repeatedly and assessed against real-world standards.
Second, it emphasizes occupational breadth. The full benchmark contains 1,320 tasks spanning 44 occupations. These occupations were selected from nine sectors representing major contributors to U.S. GDP. The tasks were designed to cover a substantial portion of the work activities associated with the selected occupations.
Third, GDPval tests multimodal and computer-based work. Tasks may involve spreadsheets, documents, presentations, images, audio, video, diagrams, CAD files, social-media material, and customer-support conversations. In the gold subset, tasks can require processing as many as 17 reference files, while the full set can involve up to 38. This makes the evaluation substantially different from a conventional text-in/text-out language-model test.
Fourth, the benchmark recognizes that professional quality is not always purely objective. Experts may judge work according to accuracy, relevance, structure, formatting, style, aesthetics, and suitability for its intended audience. GDPval therefore relies primarily on human expert comparison rather than a single automated correctness score.
Finally, the tasks are deliberately long-horizon. The benchmark reports an average expert completion time of roughly seven hours in its headline description, while the gold-set statistics show considerable variation, with some tasks taking substantially longer.
Part III: Selection of Occupations and Experts
The benchmark begins by identifying economically significant sectors and occupations. The researchers selected nine U.S. economic sectors that each contributed more than 5% to GDP. Within those sectors, they selected occupations that were both highly compensated and predominantly digital.
To determine whether an occupation was sufficiently digital, the researchers analyzed occupational tasks and classified them as digital or non-digital. An occupation was considered predominantly digital when at least 60% of its component tasks met the digital classification.
The resulting benchmark covers 44 occupations whose combined annual wages and compensation were estimated at approximately $3 trillion, namely:
Real Estate: Concierges, Property, Real Estate, and Community Association Managers, Real Estate Sales Agents, Real Estate Brokers.
Governance: Recreation Workers, Compliance Officers, First-Line Supervisors of Police and Detectives, Administrative Services Managers, Child, Family, and School Social Workers.
Manufacturing: Mechanical Engineers, Industrial Engineers, Buyers and Purchasing Agents, Shipping, Receiving, and Inventory Clerks, First-Line Supervisors of Production and Operating Workers.
IT and Technical Services: Software Developers, Lawyers, Accountants and Auditors, Computer and Information Systems Managers, Project Management Specialists.
Health Care and Social Assistance: Registered Nurses, Nurse Practitioners, Medical and Health Services Managers, First-Line Supervisors of Office and Administrative Support Workers, Medical Secretaries and Administrative Assistants.
Finance and Insurance: Customer Service Representatives, Financial and Investment Analysts, Financial Managers, Personal Financial Advisors, Securities, Commodities, and Financial Services Sales Agents.
Retail Trade: Pharmacists, First-Line Supervisors of Retail Sales Workers, General and Operations Managers.
Wholesale Trade: Sales Managers, Order Clerks, First-Line Supervisors of Non-Retail Sales Workers, Sales Representatives, Wholesale and Manufacturing, Except Technical and Scientific Products Sales Representatives, Wholesale and Manufacturing, Technical and Scientific Products.
Information: Audio and Video Technicians, Producers and Directors, News Analysts, Reporters, and Journalists, Film and Video Editors, Editors
A major component of the methodology was the recruitment of professional experts to create the tasks. Participants needed at least four years of professional experience, along with evidence of professional achievement, recognition, advancement, or management responsibility. The average expert had approximately 14 years of experience (!). Experts were also required to pass an interview, background check, training, and quiz.
Part IV: Quality Control and Human Evaluation
GDPval’s task-generation process involved several levels of quality control. All 1,320 tasks in the full benchmark underwent iterative review involving automated model-based screening and human professional review. Each task received an average of five human reviews, with at least three reviews per task. Tasks were revised in response to reviewer feedback.
The benchmark’s open gold subset consists of 220 tasks. These tasks were evaluated using blinded pairwise comparisons. Professional graders were given the task request and relevant reference files and asked to compare unlabeled human and AI-produced deliverables.
This methodology attempts to answer a practical question: Which piece of work would a relevant professional prefer?
The researchers also developed an experimental automated grader. On the gold subset, it achieved approximately 66% agreement with human graders, compared with approximately 71% agreement among human graders themselves. Thus, the automated system was considerably faster and cheaper, but human expert assessment remained the preferred evaluation method.
Part V: Human specialists vs AI Performance
GDPval evaluated several frontier AI models, including GPT-4o, o4-mini, o3, GPT-5, Claude Opus 4.1, Gemini 2.5 Pro, and Grok 4.
The results show that frontier models are approaching human expert performance on some economically valuable tasks, although performance varies substantially between models and types of deliverables.
In the gold subset, Claude Opus 4.1 had the highest reported performance in the study. Its strengths were particularly apparent in areas involving aesthetics, document formatting, and slide layout. GPT-5, by contrast, was reported to perform particularly strongly on accuracy and instruction-following, including calculations and adherence to specified requirements.
The paper reports that 47.6% of Claude Opus 4.1 deliverables were judged either better than or as good as the human deliverable. In other words, on the measured gold-set tasks, the model’s output matched or exceeded the expert’s output in just under half (!) of the comparisons.
The researchers also found systematic differences in failure modes. Claude, Gemini, and Grok frequently lost comparisons because of instruction-following problems. GPT-5 had fewer instruction-following failures but more formatting problems. GPT-5 and Grok showed relatively few accuracy errors, although all models sometimes hallucinated information or made calculation mistakes.
The results therefore suggest that model capability is multidimensional. A model can be strong at factual accuracy while being weaker at presentation, or strong at visual formatting while failing to follow specific instructions.
Part VI: Speed, Cost, and the Importance of Human Oversight
One of GDPval’s most consequential questions concerns productivity. If an AI model can complete a seven-hour professional task in a fraction of the time, its economic significance might appear obvious.
However, the researchers show why raw generation speed is not equivalent to economic productivity.
In one modeled example involving GPT-5, the study calculated a very large apparent speed advantage when comparing the model’s generation time with the expert’s completion time. But after accounting for the time required for an expert to review and correct the AI’s work, the effective productivity improvement was much smaller.
The study’s modeling therefore suggests that the economic value of AI may frequently come from reducing the amount of professional time required, rather than eliminating the professional altogether.
This distinction is central to interpreting GDPval. The benchmark demonstrates that AI can produce substantial portions of professional work, but the economic outcome depends on how AI output is incorporated into an actual workflow.
Part VII — Correct Prompting and AI Scaffolding
GDPval also demonstrates that model performance depends substantially on how the AI system is used.
Increasing reasoning effort improved performance. More detailed prompting and additional scaffolding also produced measurable improvements.
For example, the researchers instructed models to check their work, visually inspect rendered files, pay attention to formatting, and avoid certain common output problems. Such interventions reduced serious formatting failures and increased the rate at which AI outputs were preferred by human evaluators.
The researchers also experimented with strategies such as generating multiple candidate answers and using a judging process to select among them.
This finding has an important implication: the performance of an AI model should not necessarily be treated as a fixed number independent of its workflow. The model, prompt, reasoning effort, tools, verification procedures, and proper human oversight will definitely influence the final result.
Part VIII: Context Is a Major Part of Professional Work
Another GDPval experiment reduced the amount of contextual information provided to the model. When important information about the location of data, expected approach, and formatting requirements was removed, performance declined.
This highlights a fundamental difference between benchmark problems and real professional work.
Professionals frequently spend substantial time determining:
- what information is relevant;
- which documents should be consulted;
- what the client or organization actually wants;
- what standards apply;
- which tools should be used;
- how the final product should be formatted; and
- what assumptions need to be verified.
An AI system may perform very well when the task and context are clearly specified while performing less effectively when it has to determine those requirements itself.
GDPval therefore suggests that context management is itself an important component of economically valuable intelligence.
Despite its breadth, GDPval should not be interpreted as a complete representation of the labor market.
The benchmark covers only 44 occupations and approximately 30 tasks per occupation. The authors explicitly describe the first version as an initial evaluation rather than a comprehensive examination of all occupational activities.
Furthermore, GDPval tasks are generally constructed as relatively self-contained assignments. Real professional work can involve weeks or months of interaction with clients, colleagues, managers, regulators, patients, customers, and other stakeholders. Consequently, success on GDPval should be understood as evidence about AI capability on a defined set of professional tasks, rather than a direct measure of whether entire occupations can be automated.
GDPval represents a shift in how AI capability can be evaluated. Rather than asking whether a model can answer an examination question, the benchmark asks whether it can produce something resembling the work that professionals are paid to produce.
Its 44 occupations span a remarkably broad range of economic activity, from software developers and lawyers to registered nurses, mechanical engineers, financial analysts, real-estate professionals, sales representatives, journalists, editors, film and video editors, and administrative workers.
The benchmark nevertheless shows that capability and reliability are not synonymous. Frontier models can produce work that experts sometimes prefer to human-created work, but they can also make mistakes that require professional intervention.
The most significant implication may therefore be neither that AI has already replaced professional workers nor that it remains merely experimental. Instead, GDPval provides evidence that AI systems are becoming capable of performing increasingly substantial components of economically valuable professional work, while the practical economic impact depends heavily on workflow design, verification, context, and human oversight.
Conclusions:
- GDPval introduces a task-based approach to measuring AI’s economic relevance. Rather than relying primarily on AI adoption or macroeconomic indicators, it evaluates what frontier models can actually produce on economically valuable professional tasks.
- The benchmark is grounded in real professional work. Its tasks were created from the experience of industry professionals, with an average expert experience level of about 14 years, and underwent multiple rounds of expert review.
- GDPval has broad occupational coverage but remains limited in scope. The full benchmark contains 1,320 tasks across 44 occupations and nine major economic sectors, but it does not represent the entire labor market.
- Frontier models are approaching human expert performance on some professional deliverables. The results show that AI outputs can sometimes match or exceed expert outputs, although performance varies significantly by model, occupation, and type of deliverable.
- Different models demonstrate different strengths. The study reports particular strengths for Claude Opus 4.1 in aesthetics and formatting and for GPT-5 in accuracy and instruction-following, illustrating that AI capability cannot be reduced to a single dimension.
- Human oversight remains important. Raw AI generation can be extremely fast, but reviewing, correcting, and sometimes redoing AI-generated work substantially reduces the apparent speed and cost advantage.
- AI assistance may nevertheless improve professional productivity. Under the workflows modeled by the researchers, repeated AI attempts combined with expert review could produce measurable time and cost savings compared with unaided expert completion.
- Reasoning effort, prompting, tools, and scaffolding materially affect results. The study found that additional reasoning and improved self-checking and multimodal inspection could reduce errors and increase human-preference win rates.
- Context remains a major challenge. When professional tasks were deliberately underspecified, GPT-5’s performance deteriorated, suggesting that figuring out what to do and locating the relevant context remain important components of professional expertise.
- GDPval provides evidence of rapidly increasing AI capability, but not a complete prediction of labor-market replacement. Its findings are most directly applicable to the specific digital knowledge-work tasks it evaluates. The paper’s broader contribution is to provide a framework for tracking AI progress on realistic economic work while identifying where human expertise, judgment, context, oversight, and accountability remain important.
Amazon staff and AI tools: Goodhart`s law & cobra effect in real life
As detailed in a new report by the Financial Times, Amazon employees are reportedly using the company’s new internal AI tool, MeshClaw, to create extraneous AI agents — not to increase productivity, but just to drive up AI activity. That`s a real life case named “Amazon vs AI tools: Goodhart`s law & cobra effect in real […]
Why Goodhart`s law, Campbell`s law and reward hacking crucial for AI agents stress…
Why Goodhart`s – Campbell`s laws and reward hacking matter? People often distort metrics because complex outcomes are hard to measure and achieve. Once numeric targets are linked to rewards, human behavior naturally shifts to optimize the number itself. As a result, people game the system — sometimes intentionally, sometimes not. There are two well-known […]
Impressive role of weight coefficients in internet search makes SEO keywords absolutely unreliable
Is it true that impressive role of weight coefficients in internet search makes SEO keywords absolutely unreliable? Let`s analyze it. The role of of weight coefficients is substantially increasing. The development of internet search has undergone a fundamental transformation over the past year. Early search engines relied heavily on keyword matching as the principal method […]