#United States #New York #IAB Tech Lab #Bot Management #CoMP API

New Bot Management Guidelines from IAB Tech Lab

On May 27, 2026, the IAB Tech Lab, an influential body in digital advertising standards, announced the release of new guidance focused on managing bots and crawlers for public comment until June 26, 2026. This guidance is particularly aimed at content owners and publishers as they navigate the complexities of AI-driven interactions within the digital landscape.

The Purpose of the Guidance

The main goal of the IAB Tech Lab's new guidance is to equip content owners with practical and clear strategies for managing access from non-human user agents—namely, bots and crawlers. Following the introduction of the CoMP API V1, it became apparent that many content stakeholders were lacking a defined strategy to deal with this rapidly evolving aspect of the digital ecosystem. As Anthony Katsur, CEO of IAB Tech Lab, noted, providing straightforward guidance is essential for significant adoption of the API, allowing companies to tailor decisions to their needs while fostering a sustainable environment for content creation and distribution.

Addressing Gaps in Current Strategies

As AI technology continues to expand, the guidance emphasizes the need for effective decision-making frameworks that enable content owners to understand the implications of non-human traffic. The CoMP Working Group aims to clarify the available options and highlight the pros and cons of various management strategies. This empowers stakeholders to better grasp how to control access from AI systems and other bots, aligning these decisions with their overarching business objectives.

Navigating Complexity and Resource Needs

The challenge for many organizations is that managing bots and crawlers can often be resource-intensive and complex. The newly released guidance is directed towards non-technical leaders, who may need to engage with these issues but do not possess detailed technical knowledge. By encouraging these leaders to ask critical questions, IAB Tech Lab seeks to illuminate the value and costs associated with different bot management strategies. This informed approach will enable organizations to mitigate risks, lower operational costs, and operate more efficiently as they interact with AI systems.

Industry Insights and Implications

Participants in the development of the guidance have underscored the critical need for clear frameworks in an evolving digital landscape. Shailley Singh, EVP and COO at IAB Tech Lab, pointed out that content owners must often make swift and significant decisions without the benefit of comprehensive guidelines. The new guidance breaks down available strategies, allowing for choices that align with existing goals.

Scott Messer, Principal and Founder of Messer Media, brought attention to how the surge of AI bots has left many publishers vulnerable to intellectual property theft. He described the guidance as an essential roadmap for owners to navigate the intricate terrain of technology, thereby distinguishing between beneficial and detrimental interactions. The need for robust bot management strategies has transitioned from a technical task to a critical business strategy, especially as content owners aim to protect their assets effectively.

Conversely, Dave Bellous, VP of Strategy at Metal Toad, asserted that blanket bot-blocking measures are becoming obsolete. As AI agents increasingly dominate online interactions, distinguishing between helpful automation and invalid traffic has become a necessity across the advertising landscape. The newly released guidance provides essential tools for creating a more sophisticated web strategy.

Tom Koch, Head of Sales at TwelveLabs, echoed these sentiments, emphasizing the necessitated transparency and control as AI systems take center stage in consumer engagement. The crawler management framework is deemed crucial for empowering content owners with greater awareness of how their work is accessed and utilized.

Public Comment Opportunity

In light of these developments, IAB Tech Lab encourages industry professionals to participate in the public comment period, running until June 25, 2026, providing an opportunity for stakeholder feedback before finalizing the guidelines. The contributions of industry leaders will be pivotal in shaping effective strategies for bot management moving forward.

As digital content evolves in the age of AI, the IAB Tech Lab's guidance illustrates a proactive step toward securing content ownership while fostering collaboration within the burgeoning digital ecosystem. To review the guidance or provide feedback, visit IAB Tech Lab's Bot Management page.