KushoAI Launches APIEval-20, a New Benchmark for Evaluating AI Agents in API Testing
KushoAI, a leader in AI-driven API testing solutions, has unveiled its newest tool: APIEval-20. This innovative benchmark focuses on assessing how well artificial intelligence agents can detect functional bugs in APIs when provided only a request schema and sample payload, without access to source code or documentation. The launch is timely, especially given the increasingly frequent reliability issues faced by APIs today. An analysis conducted across more than 1.4 million AI-assisted test executions has revealed alarming statistics; notably, 34% of API outages are attributed to authentication failures, while a staggering 41% experience undocumented schema changes within just a month.
Traditional methods for evaluating the performance of AI in this realm often fail to account for real-world challenges, which is where APIEval-20 stands apart from typical assessment criteria. Unlike existing benchmarks that try to recreate ideal testing conditions, APIEval-20 deliberately incorporates scenarios that mimic the complexities faced in practice, including incomplete context, evolving schemas, and hidden dependencies. This approach encourages AI agents to replicate the problem-solving capabilities of human quality assurance engineers as opposed to mere automated validation tools.
Abhishek Saikia, the Co-Founder and CEO of KushoAI, emphasized this unique approach while discussing the impetus behind the benchmark. He stated, "The discourse surrounding AI in testing has primarily revolved around automation. However, there has been a significant absence of accountability; a clear mechanism to determine whether these systems genuinely perform as expected. APIEval-20 introduces this accountability into the dialogue." He also highlighted the importance of validation over just download statistics, noting, "In the first week following the launch, we received feedback from engineers who had been struggling with this question for months, and they finally have a way to address it. That validation holds greater significance for us than the number of downloads."
The new benchmark encompasses 20 distinct scenarios covering critical domains such as payments, authentication, e-commerce, scheduling, notifications, and user management. Each scenario is designed with 3 to 8 bugs intentionally embedded, presenting challenges that range from straightforward validation failures to complex logic problems that necessitate nuanced, multi-step analysis.
APIEval-20 also introduces a unique scoring model, which is aligned with the practical priorities of real-world API usage. The scoring system allocates:
With APIEval-20, KushoAI aims to set a new standard in API testing, making strides in ensuring that AI tools are held accountable for their performance. KushoAI currently serves over 30,000 engineers within more than 6,000 organizations and is recognized for its commitment to enhancing software reliability through innovative solutions. To explore more about this benchmark, you can access their report and detailed information here.
Traditional methods for evaluating the performance of AI in this realm often fail to account for real-world challenges, which is where APIEval-20 stands apart from typical assessment criteria. Unlike existing benchmarks that try to recreate ideal testing conditions, APIEval-20 deliberately incorporates scenarios that mimic the complexities faced in practice, including incomplete context, evolving schemas, and hidden dependencies. This approach encourages AI agents to replicate the problem-solving capabilities of human quality assurance engineers as opposed to mere automated validation tools.
Abhishek Saikia, the Co-Founder and CEO of KushoAI, emphasized this unique approach while discussing the impetus behind the benchmark. He stated, "The discourse surrounding AI in testing has primarily revolved around automation. However, there has been a significant absence of accountability; a clear mechanism to determine whether these systems genuinely perform as expected. APIEval-20 introduces this accountability into the dialogue." He also highlighted the importance of validation over just download statistics, noting, "In the first week following the launch, we received feedback from engineers who had been struggling with this question for months, and they finally have a way to address it. That validation holds greater significance for us than the number of downloads."
The new benchmark encompasses 20 distinct scenarios covering critical domains such as payments, authentication, e-commerce, scheduling, notifications, and user management. Each scenario is designed with 3 to 8 bugs intentionally embedded, presenting challenges that range from straightforward validation failures to complex logic problems that necessitate nuanced, multi-step analysis.
APIEval-20 also introduces a unique scoring model, which is aligned with the practical priorities of real-world API usage. The scoring system allocates:
- - 70% for bug detection effectiveness, indicating how well an AI tool can uncover defects.
- - 20% for coverage, to evaluate the breadth of API testing it can conduct.
- - 10% to measure efficiency, representing how resourcefully the tool operates during testing.
With APIEval-20, KushoAI aims to set a new standard in API testing, making strides in ensuring that AI tools are held accountable for their performance. KushoAI currently serves over 30,000 engineers within more than 6,000 organizations and is recognized for its commitment to enhancing software reliability through innovative solutions. To explore more about this benchmark, you can access their report and detailed information here.
Topics Business Technology）
【About Using Articles】
You can freely use the title and article content by linking to the page where the article is posted.
※ Images cannot be used.
【About Links】
Links are free to use.