MCP-HIVERSE BENCHMARK offers GPT-5. More than half of the tasks of synchronization in the real world

“Two of the largest are: the challenges of the long context, you can lose models that track information or struggle to constantly think when dealing with long or very complex inputs.” “The unknown challenges of the tools, models are often not able to use smoothly unfamiliar tools or systems in the way human beings can adapt to during flying. For this reason it is important to the place not to follow a DIY approach with one model of authority agents alone, but instead, to rely on a platform that combines data context, augmented logic, and guarding confidence to meet the needs of the confusion.”

MCP-inserse joins the other proposed standards that depend on MCPlike MCP-gradar From the University of Massachusetts Amhhest and the University of Xian Jiaotong, as well as Beijing University for Jobs and Telecommunications McPworld. also It builds on McPevalsSlesforce released in July, which focuses mainly on agents. He told me that the biggest difference between McP-insterse and McPevals is that the latter is evaluated with artificial tasks.

How to work

The MCP-HIVERSE evaluates the performance of each series of a series of tasks that mimic the companies do. Salesforce said she designed MCP-inserse to include six basic areas used by companies: site mobility, warehouse management, financial analysis, 3D design, browser automation and web search. I reached 11 MCP servers to a total of 231 tasks.

• Navigation focuses on the site on geographical thinking and caring for spatial tasks. Researchers took advantage of the Google Maps MCP server for this process.
• The warehouse management field is looking at the code base operations and connects to Github McP to expose the release control tools such as searching for ribo, tracking problems and editing software instructions.
• Financial analysis is connected to the McP Finance Yahoo server to evaluate quantitative thinking and make decisions in the financial market.
• The 3D design evaluates the use of design tools with the help of the computer through the Blender MCP.
• The browser automation, connected to the MCP for the theatrical writer, tests the browser interaction.
• Google Search McP and Fetch MCP server are used to check for “open field information” and are organized as a more open task.

Salesforce said it must design new MCP tasks that reflect real use cases. For every field, they created four to five types of tasks that researchers believe that LLMS can easily complete. For example, the researchers assigned the models as a target of road planning, identifying optimal stations and then locating the destination.

Each model is evaluated on how to finish the tasks. Lee and his team chose to follow the implementation evaluation model instead of the most common LLM-AS-A-Judge. The researchers noticed the LLM-AS-A-JUGE model “not good suitable for our McP-Riverse scenario, since some tasks are designed to use data in actual time, while the LLM judge’s knowledge is fixed.”

Salsforce researchers used three types of residents: coordinating residents to see if agents and models track coordination requirements, fixed residents to assess right over time and dynamic residents to obtain volatile answers such as flight prices or GitHub problems.

“The MCP-HIVRSE focuses on creating tasks in the real world with the implementation-based residents, which can test the customer in complex scenarios. Moreover, the MCP-HIVERSE offers an extension/Codes base to build and evaluate factors.”

Even large models face a problem

For the MCP-HIVERSE test, Salesforce has evaluated many famous forms of ownership and source. These include GROK-4 from xiand manS CLAUDE-4 Sonnet and Claude 3.7 Sonnet, Openai’s GPT-5, O4-MINI, O3, GPT-4.1, GPT-4O, GPT -SS, Google‘ Zaiand MonchotKimi-k2, QWEN‘ Dibsic. Each model was tested at least 120B.

In its test, Salesforce GPT-5 found the best success rate, especially for the tasks of financial analysis. Grok-4 followed, overcoming all the models of automation of the browser, and Claude-4.0 Sonnet revolves around the first three, although it has not published any performance numbers higher than any of the models it follows. Among the open source models, GLM-4.5 performed the best.

However, MCP-HIVRSE showed that models have had difficulty dealing with long contexts, especially for site mobility, browser automation and financial analysis, with efficiency dramatically. The moment LLMS faces unknown tools, its performance also decreases. LLMS has shown difficulty in completing more than half of the tasks that institutions usually do.

“These results highlight that the current LLMS is still short in carrying out the tasks reliably through the diverse MCP tasks in the real world. Thus, the MCP-HIVRSE standard provides, therefore, a difficult and necessary test to evaluate LLM’s performance in the areas with the current standards,” the paper said.

Lee Venturebeat told that he hopes that companies will use MCP-inserse to gain a deeper understanding of the place where agents and models fail in tasks so that they can improve their frameworks or carry out their MCP tools.

Daily visions about business use cases with VB daily

If you want to persuade your boss at work, you have covered VB Daily. We give you the internal journalistic precedence over what companies do with obstetric artificial intelligence, from organizational transformations to practical publishing operations, so that you can share visions of the maximum return on investment.

Read with us privacy policy

Thanks for subscribing. Check more VB news bulletins here.

An error occurred.