Who Remembers What? Multi-Granularity Memory Extraction for Multi-User, Multi-Agent Interaction addresses a problem that becomes more difficult as AI agents share experience: useful knowledge should spread, but private or agent-specific context should not.
The problem with one shared memory
Most persistent-memory systems give an agent, a user, or a team a fixed place to store lessons. That design is manageable when one user works with one agent. It becomes less suitable when many users interact with several heterogeneous agents. A global store may transfer a valuable debugging fix, but it can also expose a personal preference or work detail. A private pairwise store protects context, but prevents other agents and users from benefiting from reusable experience.
The authors frame this as cross-learning: independent user-agent pairs learn from one another’s trajectories. They identify three requirements. Knowledge should be transferable when it genuinely generalizes, each lesson should have a selective audience, and disclosure should stop at the boundary of the originating user’s private information.
MIXUA’s four memory scopes
The proposed system, MIXUA, treats memory placement as an audience-routing problem. Each entry is assigned to one of four scopes formed by combining two dimensions: the relevant user and the relevant agent.
- PAIR: readable only by one user working with one agent.
- USER: follows a user across that user’s agents.
- AGENT: follows an agent across its users.
- GLOBAL: available to the full user-agent population.
This structure matches different kinds of knowledge. A newly discovered library defect may belong in GLOBAL. A user’s preferred document format may belong in USER. Advice about a tool-calling quirk can belong in AGENT. A convention that changes depending on which agent is being used requires PAIR.
MIXUA processes each trajectory through separate reusable and protected streams. The reusable stream can produce broadly shareable lessons, while the protected stream is restricted to USER or PAIR destinations. The authors deliberately avoid relying only on a writer to recognize sensitive content: the stream itself constrains where protected evidence can go. A periodic store review can then re-scope entries, merge duplicates, and preserve provenance.
For retrieval, the system selects entries from the current pair’s authorized view. In the main setup, it ranks entries by embedding similarity between each lesson’s situation and the task prompt, subject to a character budget per scope. The paper also tests a file-based variant in which agents search memory files in their workspace.
A benchmark that tests utility and privacy together
The authors introduce CROSSLEARNBENCH, which combines established coding, terminal, spreadsheet, software-testing, and algorithmic tasks with an added user layer. The task environments remain unchanged for verification, but synthetic users receive preferences such as a filename, section order, date format, or banned markup. Those preferences are stated initially and then withdrawn, testing whether memory preserves personalization.
The benchmark also plants synthetic private facts in ordinary-looking workspace files. The study measures whether agents open those files and whether private facts appear in trajectories or in memory visible to another user. This makes privacy a measurable part of the evaluation rather than an assumption about the memory design.
What the experiments show
In the authors’ scheduled experiments, five users interacted with seven agent configurations over repeated rounds. With an LLM-based memory writer and store review, MIXUA reached an 89.8% third-round pass rate, compared with 72.0% for no memory. The reported gain over no memory was 16.3 percentage points, and the gain on tasks previously attempted only by pairs sharing neither user nor agent was 13.9 points.
The comparison with single-scope stores is important. GLOBAL-only memory improved task performance by 11.5 points, but retained only 11% of the tested preferences. PAIR-only memory retained 80% of preferences, but improved performance by only 4.3 points. MIXUA combined the two strengths: its LLM variant retained 90% of preferences, while its agentic writer with review retained 86% and still improved task performance.
The transfer result also appeared when task assignment was randomized. On tasks that no user-agent pair had attempted before, MIXUA improved performance by 10.1 points over no memory, compared with 5.5 points for a single global store. The authors report that memory review generally improved utility, preference retention, and privacy measures together.
Privacy results were favorable within this constructed evaluation. Across the main variants, no more than 0.02% of privacy probes found a planted private fact in a forbidden scope. In a separate preference-leak measure, the reviewed LLM configuration reported a 3.6% leak rate, lower than the corresponding configuration without review. The authors also show why the narrower scopes matter: removing PAIR caused agent-specific user context to leak across the user’s other agents in 95% of targeted probes, compared with 22% when PAIR was available.
Costs and limitations
MIXUA is not free. The full four-scope design delivers more context and requires memory-writing and review calls. In the reported scheduled setup, the reviewed LLM configuration used about 61.4 thousand characters of memory context per task and had a higher estimated cost than single-scope alternatives. Reducing the retrieval budget lowered context and cost while preserving some of the performance gain, but preference retention continued to benefit from more budget.
The authors’ privacy evidence should also be interpreted narrowly. The users and personal facts were synthetic, and the experiment tested a defined set of planted probes rather than real organizational data or adversarial contributors. The study models users and agents as independent entities. It cannot directly represent a team-private lesson, department-wide policy, model-family sharing, or other audiences between one user or agent and everyone.
At larger population sizes, memory stores may grow continuously, retrieval may become crowded, and review may become expensive. The authors report experiments with up to 100 synthetic users, but an open deployment with real users, changing permissions, deletion requests, consent, and malicious memory contributions remains untested.
Practical implications
For developers building multi-user agents, the central lesson is architectural: memory should carry an audience and a disclosure boundary, not just a text embedding. Provenance, explicit protected-data handling, user inspection, deletion, and independent privacy audits would be necessary before treating shared memory as production infrastructure.
MIXUA’s results suggest that “private versus shared” is too coarse for many agent systems. A useful memory layer may need to distinguish what follows the user, what follows the agent, what belongs only to their combination, and what is safe for everyone. That does not solve memory governance, but it provides a concrete framework for deciding who should remember what.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.