{"id":20879,"date":"2024-07-08T07:38:31","date_gmt":"2024-07-08T02:08:31","guid":{"rendered":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/"},"modified":"2026-03-22T21:40:38","modified_gmt":"2026-03-22T16:10:38","slug":"ai-agent-benchmarks-are-misleading-study-warns","status":"publish","type":"post","link":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/","title":{"rendered":"AI agent benchmarks are misleading, study warns\u00a0"},"content":{"rendered":"\r\n<p>AI agents are becoming a promising new research direction with potential applications in the real world. These agents use foundation models such as large language models (LLMs) and vision language models (VLMs) to take natural language instructions and pursue complex goals autonomously or semi-autonomously. AI agents can use various tools such as browsers, search engines and code compilers to verify their actions and reason about their goals.\u00a0\u00a0<\/p>\r\n\r\n\r\n\r\n<p>However, a\u00a0<a href=\"https:\/\/arxiv.org\/abs\/2407.01502\" target=\"_blank\" rel=\"noreferrer noopener\">recent analysis<\/a>\u00a0by researchers at\u00a0<a href=\"https:\/\/www.princeton.edu\/\" target=\"_blank\" rel=\"noreferrer noopener\">Princeton University<\/a>\u00a0has revealed several shortcomings in current agent benchmarks and evaluation practices that hinder their usefulness in real-world applications.\u00a0<\/p>\r\n\r\n\r\n\r\n<p>Their findings highlight that agent benchmarking comes with distinct challenges, and we can\u2019t evaluate agents in the same way that we benchmark foundation models.\u00a0<\/p>\r\n\r\n\r\n\r\n<p><strong>Cost vs accuracy trade-off<\/strong>\u00a0<\/p>\r\n\r\n\r\n\r\n<p>One major issue the researchers highlight in their study is the lack of cost control in agent evaluations. AI agents can be much more expensive to run than a single model call, as they often rely on stochastic language models that can produce different results when given the same query multiple times.\u00a0<\/p>\r\n\r\n\r\n\r\n<p>To increase accuracy, some agentic systems generate several responses and use mechanisms like voting or external verification tools to choose the best answer. Sometimes sampling hundreds or thousands of responses can increase the agent\u2019s accuracy. While this approach can improve performance, it comes at a significant computational cost. Inference costs are not always a problem in research settings, where the goal is to maximize accuracy.\u00a0<\/p>\r\n\r\n\r\n\r\n<p>However, in practical applications, there is a limit to the budget available for each query, making it crucial for agent evaluations to be cost-controlled. Failing to do so may encourage researchers to develop extremely costly agents simply to top the leaderboard. The Princeton researchers propose visualizing evaluation results as a Pareto curve of accuracy and inference cost and using techniques that jointly optimize the agent for these two metrics.\u00a0<\/p>\r\n\r\n\r\n\r\n<p>The researchers evaluated accuracy-cost tradeoffs of different prompting techniques and agentic patterns introduced in different papers.\u00a0<\/p>\r\n\r\n\r\n\r\n<p>\u201cFor substantially similar accuracy, the cost can differ by almost two orders of magnitude,\u201d the researchers write. \u201cYet, the cost of running these agents isn\u2019t a top-line metric reported in any of these papers.\u201d\u00a0<\/p>\r\n\r\n\r\n\r\n<p>The researchers argue that optimizing for both metrics can lead to \u201cagents that cost less while maintaining accuracy.\u201d Joint optimization can also enable researchers and developers to trade off the fixed and variable costs of running an agent. For example, they can spend more on optimizing the agent\u2019s design but reduce the variable cost by using fewer\u00a0<a href=\"https:\/\/venturebeat.com\/ai\/deepmind-researchers-discover-impressive-learning-capabilities-in-long-context-llms\/\" target=\"_blank\" rel=\"noreferrer noopener\">in-context learning examples<\/a>\u00a0in the agent\u2019s prompt.\u00a0<\/p>\r\n\r\n\r\n\r\n<p>The researchers tested joint optimization on\u00a0<a href=\"https:\/\/hotpotqa.github.io\/\" target=\"_blank\" rel=\"noreferrer noopener\">HotpotQA<\/a>, a popular question-answering benchmark. Their results show that joint optimization formulation provides a way to strike an optimal balance between accuracy and inference costs.\u00a0<\/p>\r\n\r\n\r\n\r\n<p>\u201cUseful agent evaluations must control for cost\u2014even if we ultimately don\u2019t care about cost and only about identifying innovative agent designs,\u201d the researchers write. \u201cAccuracy alone cannot identify progress because it can be improved by scientifically meaningless methods such as retrying.\u201d\u00a0<\/p>\r\n\r\n\r\n\r\n<p><strong>Model development vs downstream applications<\/strong>\u00a0<\/p>\r\n\r\n\r\n\r\n<p>Another issue the researchers highlight is the difference between evaluating models for research purposes and developing downstream applications. In research, accuracy is often the primary focus, with inference costs being largely ignored. However, when developing real-world applications on AI agents, inference costs play a crucial role in deciding which model and technique to use.\u00a0<\/p>\r\n\r\n\r\n\r\n<p>Evaluating inference costs for AI agents is challenging. For example, different model providers can charge different amounts for the same model. Meanwhile, the costs of API calls are regularly changing and might vary based on developers\u2019 decisions. For example, on some platforms, bulk API calls are charged differently.\u00a0\u00a0<\/p>\r\n\r\n\r\n\r\n<p>The researchers created a\u00a0<a href=\"https:\/\/benediktstroebl.github.io\/agent-eval-webapp\/\" target=\"_blank\" rel=\"noreferrer noopener\">website<\/a>\u00a0that adjusts model comparisons based on token pricing to address this issue.\u00a0\u00a0<\/p>\r\n\r\n\r\n\r\n<p>They also conducted a case study on\u00a0<a href=\"https:\/\/arxiv.org\/abs\/2403.12766\" target=\"_blank\" rel=\"noreferrer noopener\">NovelQA<\/a>, a benchmark for question-answering tasks on very long texts. They found that benchmarks meant for model evaluation can be misleading when used for downstream evaluation. For example, the original NovelQA study makes\u00a0<a href=\"https:\/\/venturebeat.com\/ai\/from-gen-ai-1-5-to-2-0-moving-from-rag-to-agent-systems\/\" target=\"_blank\" rel=\"noreferrer noopener\">retrieval-augmented generation<\/a>\u00a0(RAG) look much worse than long-context models than it is in a real-world scenario. Their findings show that RAG and\u00a0<a href=\"https:\/\/venturebeat.com\/ai\/how-gradient-created-an-open-llm-with-a-million-token-context-window\/\" target=\"_blank\" rel=\"noreferrer noopener\">long-context models<\/a>\u00a0were roughly equally accurate, while long-context models are 20 times more expensive.\u00a0<\/p>\r\n\r\n\r\n\r\n<p><strong>Overfitting is a problem<\/strong>\u00a0<\/p>\r\n\r\n\r\n\r\n<p>In learning new tasks, machine learning (ML) models often find shortcuts that allow them to score well on benchmarks. One prominent type of shortcut is \u201coverfitting,\u201d where the model finds ways to cheat on the benchmark tests and provides results that do not translate to the real world. The researchers found that overfitting is a serious problem for agent benchmarks, as they tend to be small, typically consisting of only a few hundred samples. This issue is more severe than\u00a0<a href=\"https:\/\/bdtechtalks.com\/2023\/07\/17\/llm-data-contamination\/\" target=\"_blank\" rel=\"noreferrer noopener\">data contamination<\/a>\u00a0in training foundation models, as knowledge of test samples can be directly programmed into the agent.\u00a0<\/p>\r\n\r\n\r\n\r\n<p>To address this problem, the researchers suggest that benchmark developers should create and keep holdout test sets that are composed of examples that can\u2019t be memorized during training and can only be solved through a proper understanding of the target task. In their analysis of 17 benchmarks, the researchers found that many lacked proper holdout datasets, allowing agents to take shortcuts, even unintentionally.\u00a0\u00a0<\/p>\r\n\r\n\r\n\r\n<p>\u201cSurprisingly, we find that many agent benchmarks do not include held-out test sets,\u201d the researchers write. \u201cIn addition to creating a test set, benchmark developers should consider keeping it secret to prevent LLM contamination or agent overfitting.\u201d\u00a0<\/p>\r\n\r\n\r\n\r\n<p>They also that different types of holdout samples are needed based on the desired level of generality of the task that the agent accomplishes.\u00a0<\/p>\r\n\r\n\r\n\r\n<p>\u201cBenchmark developers must do their best to ensure that shortcuts are impossible,\u201d the researchers write. \u201cWe view this as the responsibility of benchmark developers rather than agent developers, because designing benchmarks that don\u2019t allow shortcuts is much easier than checking every single agent to see if it takes shortcuts.\u201d\u00a0<\/p>\r\n\r\n\r\n\r\n<p>The researchers tested\u00a0<a href=\"https:\/\/webarena.dev\/\" target=\"_blank\" rel=\"noreferrer noopener\">WebArena<\/a>, a benchmark that evaluates the performance of AI agents in solving problems with different websites. They found several shortcuts in the training datasets that allowed the agents to overfit to tasks in ways that would easily break with minor changes in the real world. For example, the agent could make assumptions about the structure of web addresses without considering that it might change in the future or that it would not work on different websites.\u00a0<\/p>\r\n\r\n\r\n\r\n<p>These errors inflate accuracy estimates and lead to over-optimism about agent capabilities, the researchers warn.\u00a0<\/p>\r\n\r\n\r\n\r\n<p>With\u00a0<a href=\"https:\/\/venturebeat.com\/ai\/how-ai-agents-are-changing-software-development\/\" target=\"_blank\" rel=\"noreferrer noopener\">AI agents<\/a>\u00a0being a new field, the research and developer communities have yet much to learn about how to test the limits of these new systems that might soon become an important part of everyday applications.\u00a0<\/p>\r\n\r\n\r\n\r\n<p>\u201cAI agent benchmarking is new and best practices haven\u2019t yet been established, making it hard to distinguish genuine advances from hype,\u201d the researchers write. \u201cOur thesis is that agents are sufficiently different from models that benchmarking practices need to be rethought.\u201d\u00a0<\/p>\r\n\r\n\r\n\r\n<p>Content Courtesy &#8211; Venture Beat<\/p>\r\n","protected":false},"excerpt":{"rendered":"<p>AI agents are becoming a promising new research direction with potential applications in the real world. These agents use foundation models such as large language models (LLMs) and vision language models (VLMs) to take natural language instructions and pursue complex goals autonomously or semi-autonomously. AI agents can use various tools such as browsers, search engines [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":737,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-20879","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.3 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>AI agent benchmarks are misleading, study warns\u00a0 - Latest Tech News<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"AI agent benchmarks are misleading, study warns\u00a0 - Latest Tech News\" \/>\n<meta property=\"og:description\" content=\"AI agents are becoming a promising new research direction with potential applications in the real world. These agents use foundation models such as large language models (LLMs) and vision language models (VLMs) to take natural language instructions and pursue complex goals autonomously or semi-autonomously. AI agents can use various tools such as browsers, search engines [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/\" \/>\n<meta property=\"og:site_name\" content=\"Latest Tech News\" \/>\n<meta property=\"article:published_time\" content=\"2024-07-08T02:08:31+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-03-22T16:10:38+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/thecatalystonline.in\/thecatalyst\/wp-content\/uploads\/2024\/07\/AI-agents.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"750\" \/>\n\t<meta property=\"og:image:height\" content=\"421\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"author\" content=\"The Catalyst Editorial\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"The Catalyst Editorial\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/ai-agent-benchmarks-are-misleading-study-warns\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/ai-agent-benchmarks-are-misleading-study-warns\\\/\"},\"author\":{\"name\":\"The Catalyst Editorial\",\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/#\\\/schema\\\/person\\\/9e3e713b2f3c79dfdaf25069eabd6788\"},\"headline\":\"AI agent benchmarks are misleading, study warns\u00a0\",\"datePublished\":\"2024-07-08T02:08:31+00:00\",\"dateModified\":\"2026-03-22T16:10:38+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/ai-agent-benchmarks-are-misleading-study-warns\\\/\"},\"wordCount\":1161,\"publisher\":{\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/ai-agent-benchmarks-are-misleading-study-warns\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/wp-content\\\/uploads\\\/2024\\\/07\\\/AI-agents.webp\",\"articleSection\":[\"Blog\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/ai-agent-benchmarks-are-misleading-study-warns\\\/\",\"url\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/ai-agent-benchmarks-are-misleading-study-warns\\\/\",\"name\":\"AI agent benchmarks are misleading, study warns\u00a0 - Latest Tech News\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/ai-agent-benchmarks-are-misleading-study-warns\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/ai-agent-benchmarks-are-misleading-study-warns\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/wp-content\\\/uploads\\\/2024\\\/07\\\/AI-agents.webp\",\"datePublished\":\"2024-07-08T02:08:31+00:00\",\"dateModified\":\"2026-03-22T16:10:38+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/ai-agent-benchmarks-are-misleading-study-warns\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/ai-agent-benchmarks-are-misleading-study-warns\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/ai-agent-benchmarks-are-misleading-study-warns\\\/#primaryimage\",\"url\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/wp-content\\\/uploads\\\/2024\\\/07\\\/AI-agents.webp\",\"contentUrl\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/wp-content\\\/uploads\\\/2024\\\/07\\\/AI-agents.webp\",\"width\":750,\"height\":421},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/ai-agent-benchmarks-are-misleading-study-warns\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"AI agent benchmarks are misleading, study warns\u00a0\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/#website\",\"url\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/\",\"name\":\"Latest Tech News\",\"description\":\"Catalyst by mavip\",\"publisher\":{\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/#organization\",\"name\":\"Latest Tech News\",\"url\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/wp-content\\\/uploads\\\/2026\\\/02\\\/Catalyst_.png\",\"contentUrl\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/wp-content\\\/uploads\\\/2026\\\/02\\\/Catalyst_.png\",\"width\":1296,\"height\":550,\"caption\":\"Latest Tech News\"},\"image\":{\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/#\\\/schema\\\/person\\\/9e3e713b2f3c79dfdaf25069eabd6788\",\"name\":\"The Catalyst Editorial\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a1f7fd84cf8da56a5d601dfb9fb7995c04d14079e96d58d5845e7271d1935533?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a1f7fd84cf8da56a5d601dfb9fb7995c04d14079e96d58d5845e7271d1935533?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a1f7fd84cf8da56a5d601dfb9fb7995c04d14079e96d58d5845e7271d1935533?s=96&d=mm&r=g\",\"caption\":\"The Catalyst Editorial\"},\"url\":\"https:\\\/\\\/thecatalystonline.in\\\/thecatalyst\\\/author\\\/thecatalystonline\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"AI agent benchmarks are misleading, study warns\u00a0 - Latest Tech News","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/","og_locale":"en_US","og_type":"article","og_title":"AI agent benchmarks are misleading, study warns\u00a0 - Latest Tech News","og_description":"AI agents are becoming a promising new research direction with potential applications in the real world. These agents use foundation models such as large language models (LLMs) and vision language models (VLMs) to take natural language instructions and pursue complex goals autonomously or semi-autonomously. AI agents can use various tools such as browsers, search engines [&hellip;]","og_url":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/","og_site_name":"Latest Tech News","article_published_time":"2024-07-08T02:08:31+00:00","article_modified_time":"2026-03-22T16:10:38+00:00","og_image":[{"width":750,"height":421,"url":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-content\/uploads\/2024\/07\/AI-agents.webp","type":"image\/webp"}],"author":"The Catalyst Editorial","twitter_card":"summary_large_image","twitter_misc":{"Written by":"The Catalyst Editorial","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/#article","isPartOf":{"@id":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/"},"author":{"name":"The Catalyst Editorial","@id":"https:\/\/thecatalystonline.in\/thecatalyst\/#\/schema\/person\/9e3e713b2f3c79dfdaf25069eabd6788"},"headline":"AI agent benchmarks are misleading, study warns\u00a0","datePublished":"2024-07-08T02:08:31+00:00","dateModified":"2026-03-22T16:10:38+00:00","mainEntityOfPage":{"@id":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/"},"wordCount":1161,"publisher":{"@id":"https:\/\/thecatalystonline.in\/thecatalyst\/#organization"},"image":{"@id":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/#primaryimage"},"thumbnailUrl":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-content\/uploads\/2024\/07\/AI-agents.webp","articleSection":["Blog"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/","url":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/","name":"AI agent benchmarks are misleading, study warns\u00a0 - Latest Tech News","isPartOf":{"@id":"https:\/\/thecatalystonline.in\/thecatalyst\/#website"},"primaryImageOfPage":{"@id":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/#primaryimage"},"image":{"@id":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/#primaryimage"},"thumbnailUrl":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-content\/uploads\/2024\/07\/AI-agents.webp","datePublished":"2024-07-08T02:08:31+00:00","dateModified":"2026-03-22T16:10:38+00:00","breadcrumb":{"@id":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/#primaryimage","url":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-content\/uploads\/2024\/07\/AI-agents.webp","contentUrl":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-content\/uploads\/2024\/07\/AI-agents.webp","width":750,"height":421},{"@type":"BreadcrumbList","@id":"https:\/\/thecatalystonline.in\/thecatalyst\/ai-agent-benchmarks-are-misleading-study-warns\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/thecatalystonline.in\/thecatalyst\/"},{"@type":"ListItem","position":2,"name":"AI agent benchmarks are misleading, study warns\u00a0"}]},{"@type":"WebSite","@id":"https:\/\/thecatalystonline.in\/thecatalyst\/#website","url":"https:\/\/thecatalystonline.in\/thecatalyst\/","name":"Latest Tech News","description":"Catalyst by mavip","publisher":{"@id":"https:\/\/thecatalystonline.in\/thecatalyst\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/thecatalystonline.in\/thecatalyst\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/thecatalystonline.in\/thecatalyst\/#organization","name":"Latest Tech News","url":"https:\/\/thecatalystonline.in\/thecatalyst\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/thecatalystonline.in\/thecatalyst\/#\/schema\/logo\/image\/","url":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-content\/uploads\/2026\/02\/Catalyst_.png","contentUrl":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-content\/uploads\/2026\/02\/Catalyst_.png","width":1296,"height":550,"caption":"Latest Tech News"},"image":{"@id":"https:\/\/thecatalystonline.in\/thecatalyst\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/thecatalystonline.in\/thecatalyst\/#\/schema\/person\/9e3e713b2f3c79dfdaf25069eabd6788","name":"The Catalyst Editorial","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/a1f7fd84cf8da56a5d601dfb9fb7995c04d14079e96d58d5845e7271d1935533?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/a1f7fd84cf8da56a5d601dfb9fb7995c04d14079e96d58d5845e7271d1935533?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a1f7fd84cf8da56a5d601dfb9fb7995c04d14079e96d58d5845e7271d1935533?s=96&d=mm&r=g","caption":"The Catalyst Editorial"},"url":"https:\/\/thecatalystonline.in\/thecatalyst\/author\/thecatalystonline\/"}]}},"_links":{"self":[{"href":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-json\/wp\/v2\/posts\/20879","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-json\/wp\/v2\/comments?post=20879"}],"version-history":[{"count":0,"href":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-json\/wp\/v2\/posts\/20879\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-json\/wp\/v2\/media\/737"}],"wp:attachment":[{"href":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-json\/wp\/v2\/media?parent=20879"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-json\/wp\/v2\/categories?post=20879"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/thecatalystonline.in\/thecatalyst\/wp-json\/wp\/v2\/tags?post=20879"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}