IBM and Artificial Analysis Release ITBench-AA to Evaluate Agentic AI on Enterprise IT Tasks

IBM and Artificial Analysis have introduced ITBench-AA, a specialized benchmarking framework designed to evaluate the performance of agentic AI in enterprise IT and site reliability engineering tasks. This benchmark moves beyond simple text generation or coding snippets to test how autonomous agents handle end-to-end operational workflows. The framework is now hosted on Hugging Face to provide a standardized environment for assessing agent reliability in technical infrastructure roles. The initial results from ITBench-AA indicate that modern frontier models still struggle with high-level autonomous tasks. Most leading models scored below 50% in the evaluation, highlighting a significant gap between current capabilities and the requirements for fully automated IT operations. These findings suggest that while LLMs excel at individual tasks, maintaining state and executing multi-step logic in live environments remains a major challenge. Researchers can utilize ITBench-AA to identify specific failure points in agentic reasoning, such as tool usage errors or long-horizon planning deficiencies. By basing the benchmark on IBM's established ITBench data, the team ensures that the scenarios reflect realistic enterprise challenges. This release establishes a critical baseline for the development of the next generation of AI agents capable of assisting SRE teams.