HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory
As large language model (LLM) agents increasingly engage in long-term interactions with users, a key challenge emerges: how can these agents infer and apply user habits based on sparse, indirect evidence accumulated over time? Junming Zhang and colleagues from LUMIA Lab and affiliated institutions introduce HABIT-Bench, a novel benchmark designed to rigorously evaluate an agent’s ability to induce user habits from weak, distributed evidence in longitudinal memory and to use these habits appropriately in decision-making.
The Problem: Inferring Habits from Weak, Distributed Evidence
Users reveal their habits through repeated choices and actions across many interactions. However, each individual observation may only partially reflect a habit or its applicable context. Simply retaining past interactions does not guarantee that an agent can reconstruct the underlying behavioral defaults or know when to apply them. For example, a user’s choice in one session might be an exception rather than a rule, or a previously established habit might have changed over time.
Existing benchmarks assess aspects of long-term memory, preference inference, or personalization, but none fully address the joint problem of habit induction from weak evidence combined with the selective and context-sensitive application of those habits. HABIT-Bench fills this gap by focusing on whether agents can infer behavioral defaults from incomplete, longitudinal evidence and apply them correctly considering scope, exceptions, temporal drift, and user endorsement.
HABIT-Bench: Method and Design
HABIT-Bench is constructed from synthetic user histories derived from task-oriented dialogue datasets across four domains: food, finance, software, and travel. These domains provide complementary evidence structures and challenges:
- Food: Cooking dialogues with repeated choices and contextual contrasts.
- Finance: Banking workflows requiring reconstruction of user-confirmed policies and tracking changes over time.
- Software: Technical support interactions with evolving procedures and environment-specific habits.
- Travel: Flight and hotel booking dialogues emphasizing behavioral-rule recovery and application to new requests.
The benchmark includes 114 synthetic users, 37,152 sessions, and 3,328 four-choice probe tasks. Each probe asks the agent to select the correct action based on the user’s history and the current request, testing the agent’s ability to infer habits and apply them appropriately. Probes include challenging cases such as temporary exceptions, habit changes, and unaccepted assistant suggestions.
The evaluation protocol requires methods to process user sessions chronologically, maintaining user-isolated memory. At each probe, the method retrieves relevant memory context and a shared answerer LLM selects one of four candidate actions. The benchmark uses exact-match accuracy to avoid variability from free-form responses.
Evidence and Validation
To ensure the benchmark’s quality, the authors conducted independent human validation on sampled probes. Reviewers assessed whether the annotated evidence uniquely supports a correct action, the sufficiency and scope of evidence, handling of exceptions, and naturalness of dialogues. Agreement between reviewers was very high, supporting the benchmark’s reliability.
Experimental Evaluation and Results
The authors evaluated eight state-of-the-art memory methods alongside retrieval baselines and controls using two large LLMs: Qwen3-235B-A22B and GLM-5.3-Flash. Key findings include:
- Habit induction and action selection remain challenging: The best memory methods achieved only about 33% macro accuracy across four domains, where random guessing yields 25%. This highlights the difficulty of inferring and applying habits from weak, distributed evidence.
- Domain-dependent benefits: Memory methods improved performance notably in Food and Travel domains but struggled in Finance and Software, where reconstructing multiple workflows and temporal states is more complex.
- Memory methods do not consistently outperform retrieval baselines: Simple retrieval approaches like BM25 or dense retrieval performed comparably to or better than complex memory architectures in many cases.
- Oracle diagnostics reveal remaining challenges: Even when supplied with annotated evidence directly, accuracy remained limited (up to ~58%), indicating that evidence alone does not resolve the task. Providing the exact habit state raised accuracy near 99%, showing that the main difficulty lies in evidence selection and correct application.
- Model size effects are modest and non-monotonic: Larger LLMs improved performance somewhat but did not fully overcome the benchmark’s challenges.
Limitations and Diagnostic Insights
HABIT-Bench exposes a persistent gap between having access to past interactions and the ability to infer and appropriately use user habits. The benchmark’s design emphasizes the complexity of habit induction, including the need to:
- Integrate distributed, incomplete evidence spanning many sessions.
- Track the temporal validity and scope of habits, including exceptions and changes over time.
- Distinguish user-endorsed actions from unaccepted assistant suggestions.
Diagnostic analyses show that retrieving relevant evidence does not guarantee correct decisions, as the decisive information may be distributed or require reasoning about context and temporal conditions. The authors also highlight the substantial computational cost of memory construction compared to querying, underscoring practical trade-offs in deploying memory-augmented agents.
Practical Implications and Future Directions
HABIT-Bench provides a controlled, multi-domain environment for studying how agent memory systems can support personalized, context-aware decision-making based on inferred habits. Its findings suggest that current memory architectures and retrieval methods have limited ability to induce and apply habits from weak longitudinal evidence.
For practitioners building personalized AI assistants or agents, this implies that simply storing past interactions is insufficient. Effective personalization requires memory representations that preserve relationships between behavioral defaults, their supporting evidence, and conditions of applicability. Moreover, agents must incorporate mechanisms to handle exceptions, temporal drift, and user endorsement explicitly.
Future research can leverage HABIT-Bench to develop and evaluate memory systems that better capture these complexities, potentially combining structured habit representations, improved evidence retrieval, and reasoning capabilities. The benchmark’s open-source release and detailed validation protocol facilitate reproducibility and extension.
“These findings motivate memory representations that preserve the relationships between a default, its supporting observations, and its conditions of validity.” — Zhang et al.
In summary, HABIT-Bench highlights the nuanced challenges of habit induction in AI agents and provides a rigorous framework to advance memory-augmented personalization research.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.