Understanding AI Accountability: The Need for Governance in Automated Decisions

In recent discussions surrounding artificial intelligence (AI), a startling statistic has emerged from Sopra Steria Next's report titled "AI Isn't Effective Without Control and Purpose". According to the findings, less than 20% of automated decisions made by organizations have an identified responsible party. This raises significant concerns about the governance of decisions driven by AI, particularly as these systems evolve from merely providing recommendations to executing actions autonomously.

For the last decade, organizations have focused tirelessly on data governance, ensuring that their data is well-managed, secure, and compliant with regulations. However, the critical aspect of governance concerning decision-making has received little attention. As generative AI and agent-based AI technologies gain traction, the necessity of identifying who is accountable for the decisions these systems make has become paramount. This report by Sopra Steria Next is a call to action for technology leaders (CIOs) to explore nine dimensions of governance that stretch beyond data management into the realm of decision accountability.

Key Findings from the Report

Among the several findings outlined, two stand out:
1. Accountability Lacks: The vast majority of AI-influenced or executed decisions operate in an environment devoid of formal accountability. With fewer than 20% of these decisions being traceable back to a business leader, the risk of poor decision-making increases.
2. Traceability and Reproducibility Issues: Less than 25% of AI-driven decisions that are currently in production can be tracked and reproduced from start to finish. This creates a significant barrier to accountability since understanding how and why a decision was made is essential for rectifying any potential errors.

The implications of an ungoverned AI landscape could be severe. An automated decision failure may affect ten to a hundred times more operations, customers, or revenues than a traditional IT system failure. This emphasizes the urgency of establishing robust governance structures that can withstand scrutiny and provide clarity in decision-making processes.

What Does AI Sovereignty Mean?

The notion of AI sovereignty extends beyond the basic understanding of where data is stored or where AI models are run. It encompasses the critical capability to demonstrate, at any given moment, who controls, clarifies, and arbitrates the automated decisions being made. The shift in dependency from traditional infrastructure to cognitive dimensions makes accountability increasingly difficult to enforce.

To usher in effective governance, Sopra Steria Next urges technology leaders to establish a five-tier sovereignty framework, which includes consideration for data, models, platforms, APIs, and decisions. Each layer requires thoughtful questions: Who operates it? Who has the authority to replace it? Who assumes responsibility for its actions?

Urgent Need for Change

According to Valérie Thiblet, a partner in Data and AI at Sopra Steria Next, a pressing concern emerges - automated systems are already making decisions without any clear assignment of responsibility. "A system that cannot be halted in real-time is not under control; it merely exists passively. What cannot be controlled cannot be effectively industrialized," she asserts.

Moreover, the report highlights critical challenges for IT departments, such as the rise of shadow AI, which undermines traceability and incurs costs often before stakeholders recognize the issue. Additionally, model governance has become an optional layer rather than a prerequisite, greatly increasing the risk of unintentional non-compliance with upcoming regulations.

Conclusion

The insights from Sopra Steria Next's CIO Compass report reveal an urgent need for organizations to re-evaluate how they govern not only their data but also the automated decisions that arise from AI systems. As these technologies continue to shape our world, the foundation of accountability, responsibility, and governance must evolve to ensure that AI serves its intended purpose effectively and ethically. Failure to do so could lead to an era where decisions made by autonomous systems lack clarity, precision, and the necessary accountability to safeguard individuals and organizations alike.

With the growing complexity of AI decision-making, the questions surrounding governance are more critical than ever, urging organizations to act decisively and responsibly.