Examining Changes in LLM Brand Recall Across Generations: A Comprehensive Study

Examining Changes in LLM Brand Recall Across Generations

Introduction

In a groundbreaking study conducted by Todo Onada, a company based in Taito, Tokyo, the evolution of brand recall in large language models (LLMs) across different generations has been thoroughly analyzed. Specifically, the research sought to understand how the 'recall set' associated with certain questions shifts from one model generation to another. This inquiry examined the responses from Claude Opus 4.8 and Claude Opus 5, each tested with an identical series of questions a hundred times each, allowing for a detailed comparison of their output.

Research Background

The development of LLMs can be broadly categorized into two main phases: pre-training and post-training. The pre-training phase involves feeding massive amounts of text data into the model, which determines how company names and products are remembered in relation to various categories. The subsequent post-training phase focuses on refining the model's output to make it usable and ensuring its safety. This study scrutinized the connections formed during the pre-training phase.

Historically, developing a new model from scratch has been a time-consuming process. Models like Llama 3.1, for instance, required an estimated 80 days for data ingestion and close to seven months from training to public release. This totals approximately ten months before any new information can be expected to reflect in AI responses, underscoring a significant lag time.

However, significant advancements in technology have prompted leading LLM vendors to shorten the timeline between the 'knowledge cutoff' and product release. The knowledge cutoff refers to the point in time until which the model retains widespread and reliable knowledge. Notably, companies such as Anthropic specify two types of cutoff dates: one for training data inclusion and another for reliable knowledge.