#Japan #Tokyo #AI Collaboration #HireRoo #SEI Index

New Metric for AI Collaboration in Engineering

In a groundbreaking move, HireRoo, a Tokyo-based company, has launched a unique evaluation framework called AI Collaboration Power. This metric aims to assess how effectively engineers work alongside AI agents, using a rich dataset of over 1.16 million evaluations gathered from more than 300 companies, focusing on 5 key areas and 13 specific criteria.

The launch of AI Collaboration Power highlights a significant shift in the way we evaluate AI use in software development. Traditionally, metrics such as usage volume—like token count and frequency—served as key performance indicators (KPIs). However, there’s growing realization that these quantitative metrics do not genuinely reflect productivity or effectiveness. In fact, data from various studies reveal a disconnect between AI consumption and meaningful outcomes.

HireRoo's initiative comes in response to these shortcomings. With AI tools proliferating in development environments, measuring performance purely by usage cannot accommodate the nuanced needs of modern engineering practices. The AI Collaboration Power evaluates qualitative aspects of human-AI interaction during actual development sessions, focusing on how engineers direct AI, verify outputs, and adapt strategies in real time.

Evaluation Framework

The AI Collaboration Power framework consists of five sub-domains:

1. Intent Specification: This area assesses whether the engineer can effectively communicate their goals to the AI, specifying prerequisites, constraints, and acceptance criteria clearly.
2. Environment Engineering: This dimension examines the infrastructure set up for the AI to function correctly, including the necessary documentation and tool connections.
3. Co-Reasoning: Central to human-AI collaboration, this criterion measures the engineer’s ability to engage in dialogue with the AI to solve problems, generate hypotheses, validate them, and iterate on solutions.
4. Execution Control: This evaluates how well the engineer can manage the AI's execution process, including halting actions, approvals, and adjustments.
5. Output Quality Assurance: Here, the focus is on the engineer’s ability to assess the validity of the results produced by the AI, including through testing, reviews, and acceptance decisions.

The hallmark of this evaluation is its focus on the human's role rather than the outcomes produced directly from AI. The underlying principle is that as long as the evaluation targets human behavior, the metric remains pertinent regardless of advancements in AI technology.

Findings on Collaboration

Preliminary data indicates that the weakest area of performance among evaluated engineers is in Co-Reasoning, where scores averaged only 48.6%. This insight sheds light on a critical gap: while engineers can effectively define requirements and control outputs, there is a notable deficiency in collaborative thinking with AI. Many engineers resort to a more traditional model, issuing commands to the AI and manually refining outputs, rather than leveraging the AI's capabilities for iterative problem-solving.

Contrastingly, areas like Execution Control and Intent Specification exhibited higher average scores of 69.7% and 67.3%, indicating that while engineers can articulate their needs and manage outputs, there is substantial room for growth in collaborative reasoning, a skill essential for maximizing AI's potential in real-world applications.

Commitment to Improvement

In conjunction with the release of AI Collaboration Power, HireRoo is providing a comprehensive white paper detailing the evaluation system, promoting an understanding of its sub-domains, criteria, and real-world scoring data. The white paper is available for free and can be downloaded from their website. Additionally, an online seminar titled "AI Collaboration Power: From Quantity to Quality—How to Identify the Quality of AI Utilization" is scheduled for September 3, 2026. This seminar aims to elucidate the challenges in assessing individual performance in the context of AI, utilizing both international study comparisons and HireRoo’s data.

For all industry leaders, particularly engineering managers and HR personnel, this initiative represents a pivotal moment in universally defining and enhancing the collaboration between engineers and AI. HireRoo aims to establish a common language around this essential skill set, facilitating the evolution of engineering organizations in Japan and beyond.

By embracing the AI Collaboration Power framework, the goal is clear: to empower Japanese engineering teams to lead in the digital age, reclaiming their status as the forefront of innovation in manufacturing and technology.