#AI Research #Appier #LLM

Appier's Innovative AI Research

In a bid to enhance AI applications in marketing and advertising technology, Appier Group Inc., headquartered in Minato City, Tokyo, has announced a significant advancement in its AI research. As businesses increasingly incorporate agent-based AI into their core operations, the Appier AI research team has published two pivotal studies addressing critical aspects of AI performance.

The studies delve into how large language models (LLMs) can recognize when there are no correct answers in multiple-choice scenarios, and how the choice of language in multilingual contexts can affect reasoning outcomes, safety, and cultural understanding. These insights aim to reshape how AI's reliability and global applicability are assessed in business environments.

The Growing Importance of AI Self-Awareness

As AI becomes a staple in customer support and marketing strategies, understanding its limitations is crucial. For instance, if an e-commerce AI agent cannot confirm the return policies applicable to a certain product, it may resort to applying rules from similar products erroneously, potentially leading to customer dissatisfaction. Similarly, when expanding games into international markets, the subtleties of language become vital in crafting effective advertising. An AI that lacks cultural and contextual awareness could overlook local preferences and risks, thereby jeopardizing marketing efforts.

To effectively integrate agent-based AI into corporate decision-making, it is essential for AI systems to understand uncertainties and selectively choose appropriate reasoning methods and languages based on the task at hand.

Addressing Information Gaps

For agent-based AI to function autonomously, it needs to gather requisite information from the company’s knowledge base, documents, or external data. One notable architecture beneficial for this is Retrieval-Augmented Generation (RAG). The ability of the model to recognize when it lacks sufficient information directly impacts the reliability of its decisions.

In the research paper titled “None of the Above, Less of the Right: Parallel Patterns between Humans and LLMs on Multi-Choice Questions Answering,” the team evaluated 28 leading LLMs on questions where “None of the Above” is the correct choice. The findings revealed that accuracy drops by 30-50% when faced with such options. This suggests that even when possessing relevant knowledge, models may tend to provide forced answers rather than admitting uncertainty.

Cultivating the ability to recognize one's limitations is particularly essential in tasks requiring comprehensive judgment, such as ethical business decisions, as opposed to directly calculable tasks.

Using supervised fine-tuning (SFT) and direct preference optimization (DPO), the Appier AI research team trained models to identify “None of the Above” scenarios. While SFT focuses on learned correct answers, DPO trains models to distinguish between favorable and unfavorable responses. The results showed that DPO improved the accuracy of identifying “no correct answer” scenarios by nearly 30 points, confirming that this capability can indeed be enhanced through training.

However, the applicability of the “None of the Above” option is limited and can only be used for certain types of clear-cut questions with mutually independent options. This indicates that training AI’s ability to withhold answers must be done on a task-by-task basis to achieve optimal results. In real-world applications, establishing a verification system—ensuring that adequate information is available before acting—would be beneficial. In cases where AI detects insufficient information, it could autonomously re-search or hand over the task to human operators.

Selecting the Right Reasoning Language

In addition to dealing with uncertainties, agent-based AI must also possess the ability to choose the appropriate reasoning language relevant to the task characteristics. The study titled “Language Matters: How Do Multilingual Input and Reasoning Paths Affect Large Reasoning Models?” uncovered that the reasoning language impacts the accuracy of logical reasoning, safety evaluations, and cultural comprehension. When asked questions in different languages, models tend to rely on high-resource languages like English for internal reasoning, often resulting in a mismatch between reasoning and response languages.

Using a technique called text pre-filling, the Appier AI research team forced models to adopt a specific reasoning language from the onset of a task and observed performance impacts. It was found that while high-resource languages generally favored mathematical and knowledge tasks,