Beyond Retrieval: Analytic Memory for Multimodal Agents

Beyond Retrieval: Analytic Memory for Multimodal Agents

Zhoujin Tian 1 , Hao Zhang 2 , Yao Tian 2 , Cheng Chen 2 Yakun Li 2 , Lei Zhang 2 , Xiaofang Zhou 1 1 1 footnotemark: 1 1 HKUST, 2 ByteDance {ztianaf, zxf}@cse.ust.hk {zhanghao.ai, yao.tian, chencheng.sg, liyakun.hit, zhanglei.michael}@bytedance.com Corresponding authors.

Abstract Long-term multimodal memory must support not only retrieving relevant information but also computing over observations accumulated across interactions. Existing systems largely emphasize retrieval memory , organizing interaction histories through summaries and indexes to return query-relevant information at multiple granularities, from high-level abstractions to underlying records. In this paper, we formulate analytic memory as a complementary abstraction that organizes recurring multimodal observations into queryable structures supporting filtering, aggregation, ranking, and temporal comparison. We present AdaMM , a framework that jointly supports retrieval and analytic memory. Rather than relying on application-defined schemas, AdaMM extracts provenance-linked attribute-value observations from dialogue, images, and contextual metadata, discovers recurring field structures, and materializes them for analytical access. At inference time, a memory-aware planner decomposes queries into retrieval and analytic operations and routes each operation to the appropriate tools. Experiments on two long-term multimodal memory benchmarks, MemEye and MemGallery, show that AdaMM improves performance by up to 11.3% and 6.9%, respectively.

Beyond Retrieval: Analytic Memory for Multimodal Agents

Zhoujin Tian 1 , Hao Zhang 2 , Yao Tian 2 † † thanks: Corresponding authors. , Cheng Chen 2 Yakun Li 2 , Lei Zhang 2 , Xiaofang Zhou 1 1 1 footnotemark: 1 1 HKUST, 2 ByteDance {ztianaf, zxf}@cse.ust.hk {zhanghao.ai, yao.tian, chencheng.sg, liyakun.hit, zhanglei.michael}@bytedance.com

1 Introduction

As Large Language Model (LLM) agents interact with users over increasingly long horizons, they accumulate extensive multimodal histories – including dialogues, screenshots, images, documents, and contextual observations – that quickly exceed the finite context windows of their backbone models (He et al. , 2024 ; Koh et al. , 2024 ; Xie et al. , 2024 ; Wang et al. , 2024b ) . Multimodal memory systems, which retain and organize such histories to support future reasoning (Wang and Chen, 2025 ; Feng et al. , 2026 ) , have therefore become essential and are widely adopted in applications such as personal assistance (Li et al. , 2024 ; Feng et al. , 2026 ) , embodied interaction (Driess et al. , 2023 ; Wang et al. , 2024a ) , and tool-augmented workflows (Yao et al. , 2023 ; Schick et al. , 2023 ) .

Figure 1: Long-term multimodal interaction histories contain observations distributed across sessions, modalities, and time. Answering user queries may require either retrieving specific past events or performing analytical operations, such as temporal selection and aggregation, over multiple observations.

Recent multimodal memory systems have explored a broad range of designs, including textual abstraction (Feng et al. , 2026 ) , specialized memory types (Nguyen et al. , 2026 ) , hybrid stores (Wang and Chen, 2025 ) , and cross-modal retrieval (Lu et al. , 2026 ) . Most existing systems follow a retrieve-then-answer paradigm, conditioning an LLM on a bounded set of relevant memories, which we denote as retrieval memory. Such systems are effective at selecting relevant memories from long interaction histories. However, long interaction histories also accumulate recurring observations that collectively form an append-only log. Analytical questions over such histories require complete, correctly scoped records and operations such as filtering, aggregation, ranking, and temporal selection. The relevance-based approach adopted by retrieval memory systems therefore creates a coverage–context trade-off: a small retrieval set may omit required observations, whereas a larger retrieval set consumes the limited context budget and introduces redundant or distracting evidence. For example, as illustrated in Figure 1 , computing a user’s average sleep duration over the past month requires collecting all relevant measurements across multiple screenshots before aggregation; incomplete retrieval may therefore produce a biased estimate. We call this gap between relevance-based retrieval and analytic the retrieval–analysis mismatch .

To bridge this gap, our key insight is that a memory system should couple two complementary subsystems: retrieval memory , which identifies and supplies relevant historical records as context, and analytic memory , which proactively organizes interaction histories into reusable and queryable structures and adaptively executes operations over them. Together, they support both flexible semantic recall and structured analysis, rather than requiring the LLM to repeatedly reconstruct the needed organization from retrieved records. Realizing analytic memory, however, presents two key challenges. First, interaction histories do not come with an explicit organizing schema, making it unclear what information should be preserved and how it should be structured to support future queries. Second, user queries rarely specify memory access strategies; the system therefore must reconcile what each query requires with what the evolving memory can currently support. Even questions over the same underlying records may require fundamentally different operations (e.g., aggregation, ordering, filtering), posing challenges to the design of adaptive execution plans.

To address these challenges, we introduce AdaMM , an Ada ptive M ulti-view M emory framework that couples data-driven structure induction with memory-aware query planning. It maintains two complementary subsystems: retrieval memory for flexible semantic access and analytic memory for executable analysis over recurring observations. AdaMM extracts provenance-linked attribute–value pairs from multimodal interactions (e.g., { Sleep Time: 5.5 hrs }), induces recurrent schemas, and materializes them as structured tables, which are accessible via tools, while preserving unstructured evidence in a hierarchical semantic graph. At query time, a planner adaptively composes different analytic and retrieval tools based on the query and the memory structures currently available. This design enables grounded semantic recall and structured analysis over long-term interaction histories.

Our contributions are summarized as follows:

• We identify a retrieval–analysis mismatch in long-term multimodal memory: retrieval-oriented interfaces support returning query-relevant information at multiple granularities, but cannot reliably answer queries requiring complete-range filtering, aggregation, ranking, or temporal selection over append-only interaction histories.

• We propose AdaMM , a multimodal memory framework that complements retrieval memory with schema-induced analytic memory. AdaMM discovers recurrent structures from multimodal observations, materializes them as queryable tables, and uses a memory-aware planner to select or compose semantic retrieval with native analytical operations.

• We evaluate AdaMM on two multimodal long-term memory benchmarks, MemEye and MemGallery, where AdaMM consistently outperforms strong memory baselines, and improves accuracy by up to 11.3% and 6.9%, respectively.

2 Related Work

Agent memory.

Long-term agent memory preserves information across interactions, allowing agents to reuse preferences, observations, decisions, and task states beyond a finite context window Hatalis et al. ( 2023 ); Li et al. ( 2024 ); Zhang et al. ( 2025 ) . Existing systems explore personalized stores and tiered memory management Zhong et al. ( 2024 ); Packer et al. ( 2023 ) , scalable or agentic memory construction Chhikara et al. ( 2025 ); Xu et al. ( 2025 ) , and temporal or relational organization Rasmussen et al. ( 2025 ); Huang et al. ( 2026 ) . Recent schema-grounded memory moves beyond semantic recall by transforming textual interactions into validated, queryable records, but relies on application-defined schemas Petrov et al. ( 2026 ) . Despite these advances, existing methods remain largely text-centric, leaving underexplored how to induce reusable analytical structures from heterogeneous multimodal histories and adaptively execute retrieval and analytical operations over them.

Multimodal agent memory.

Multimodal agent memory extends text-centric memory to histories containing heterogeneous visual observations. Existing approaches broadly follow two directions: MIRIX and M2A translate visual content into captions, summaries, or semantic abstractions for text-based retrieval Wang and Chen ( 2025 ); Feng et al. ( 2026 ) , whereas MMA, VisualMem, and Omni-SimpleMem retain native visual evidence and retrieve multimodal or image-backed memories Lu et al. ( 2026 ); Nguyen et al. ( 2026 ); Liu et al. ( 2026 ) . Although these methods improve visual evidence preservation, they still primarily organize histories as semantic memories. MemEye further reveals their difficulty in capturing fine-grained details and evolving visual states Guo et al. ( 2026 ) . In contrast, AdaMM adaptively organizes multimodal histories into complementary analytical and semantic memories, enabling both precise analysis and context-aware retrieval for complex queries.

3 Method

Figure 2: Overview of AdaMM . It complements recall-oriented retrieval memory with schema-induced analytic memory over recurrent multimodal observations. At query time, an operation planner jointly considers the query and the instantiated memories to select designed access tools for answer generation.

3.1 Problem Formulation

Let ℋ = { S i } i = 1 M \mathcal{H}={S{i}}{i=1}^{M} denote a multimodal interaction history, where each session S i = { R i , j } j = 1 n i S{i}={R{i,j}}{j=1}^{n{i}} consists of multiple interaction rounds. Each round R i , j = ( d i , j , 𝒱 i , j , τ i , j ) R{i,j}=(d{i,j},\mathcal{V}{i,j},\tau{i,j}) comprises the user–agent dialogue d i , j d{i,j} , visual observations 𝒱 i , j \mathcal{V}{i,j} , and temporal information τ i , j \tau{i,j} . A multimodal agent memory system transforms the interaction history into memory representations, ℳ = F build ​ ( ℋ ) , \mathcal{M}=F{\mathrm{build}}(\mathcal{H}), and aims to generate an accurate answer y ^ = F answer ​ ( ℳ , q ) \hat{y}=F_{\mathrm{answer}}(\mathcal{M},q) for a user query q q based on ℳ \mathcal{M} .

3.2 Overview

Figure 2 presents the overview of the proposed AdaMM framework. Given a long-term multimodal interaction history, AdaMM constructs two complementary memories: analytic memory ℳ ana \mathcal{M}^{\mathrm{ana}} induces recurring patterns as adaptive schemas for analytical queries while retrieval memory ℳ ret \mathcal{M}^{\mathrm{ret}} hierarchically organizes interactions according to their semantics to enable flexible retrieval. At query time, AdaMM jointly reasons over the query and the memories instantiated from the current history, then constructs an execution plan by adaptively selecting and composing access tools to gather sufficient evidence for grounded answer generation.

3.3 Analytic Memory Construction

Analytic memory aims to uncover and organize recurring patterns across fragmented observations in multimodal interaction histories for analytical operations. The key challenge is that the meaningful attributes, which is in the form of key-value pair, and their co-occurrence patterns are unknown in advance, and each round reveals only a sparse fragment of the latent structure. To address this challenge, AdaMM treats ℋ \mathcal{H} as a chronologically ordered interaction rounds { R t } t = 1 N {R{t}}{t=1}^{N} , performs Attribute Extraction to adaptively identify attribute evidence within each round, then conducts Schema Induction to discover stable structures from recurring cross-round patterns, and finally applies Memory Materialization to transform the induced schemas into executable analytic memory.

3.3.1 Attribute Extraction

Recovering recurring structure must begin from the evidence available within individual rounds, yet the attributes that will form a useful schema are unknown at extraction time. Record fragment extraction therefore aims to preserve explicit analytic observations from each round without imposing a predefined and unobserved pattern.

Given R t R_{t} , AdaMM jointly examines its dialogue and visual content to identify every grounded attribute–value correspondence:

𝒪 t = Extractor ⁡ ( R t ) = { ( a t ​ ℓ , x t ​ ℓ , p t ​ ℓ ) } ℓ = 1 m t , \mathcal{O}{t}=\operatorname{Extractor}(R{t})=\left{\left(a{t\ell},x{t\ell},p{t\ell}\right)\right}{\ell=1}^{m_{t}}, (1)

where a t ​ ℓ a{t\ell} and x t ​ ℓ x{t\ell} are the observed attribute and its source-faithful value, and p t ​ ℓ p{t\ell} points to the supporting dialogue span or image. Extractor \operatorname{Extractor} is an LLM-based extractor that aims to identify all possible pairs from each round. The resulting collection { 𝒪 t } t = 1 N {\mathcal{O}{t}}_{t=1}^{N} captures these observations and their within-interaction co-occurrence patterns for subsequent schema induction.

3.3.2 Schema Induction

Given the extracted record fragments, it remains challenging to induce reusable schemas without assuming a predefined schema. New observations may reveal either a previously unseen structure or evolution to an existing one. To address this challenge, AdaMM first mine candidate patterns and then induces schemas through two processes: Schema Discovery creates new schemas, while Schema Evolution extends existing ones with consistently co-occurring attributes.

Candidate Pattern Mining.

For each extracted record fragment 𝒪 t \mathcal{O}_{t} , let

𝒜 t = { a t ​ ℓ ∣ ( a t ​ ℓ , x t ​ ℓ , p t ​ ℓ ) ∈ 𝒪 t } \mathcal{A}{t}=\left{a{t\ell}\mid\left(a{t\ell},x{t\ell},p{t\ell}\right)\in\mathcal{O}{t}\right} (2)

denote the set of attributes contained in 𝒪 t \mathcal{O}_{t} . A reliable candidate pattern C C is a set of attributes that consistently recur sufficient interactions together. Following the Apriori framework for frequent-itemset mining (Agrawal and Srikant, 1994 ) , we use support to quantify information recurrence. Specifically, the support of a candidate pattern 𝒞 \mathcal{C} after processing round t t is

supp t ⁡ ( 𝒞 ) = ∑ s = 1 t 𝕀 ​ [ C ⊆ 𝒜 s ] , \operatorname{supp}{t}(\mathcal{C})=\sum{s=1}^{t}\mathbb{I}!\left[C\subseteq\mathcal{A}_{s}\right], (3)

AdaMM retains 𝒞 \mathcal{C} as a candidate pattern if

supp t ⁡ ( 𝒞 ) ≥ θ s , \operatorname{supp}{t}(\mathcal{C})\geq\theta{s}, (4)

which ensures that 𝒞 \mathcal{C} recurs across sufficient interaction rounds to yield a populated schema.

Schema Discovery.

For each recurrent candidate 𝒞 \mathcal{C} , AdaMM first compares it with the active schemas before round t t . If 𝒞 \mathcal{C} exhibits an attribute structure distinct from all existing schemas, it is evaluated for reliable new-schema discovery. Following a variant of Apriori algorithm (Omiecinski, 2003 ) , we use all-confidence to measure within-pattern co-occurrence consistency, defined as

all − conf t ⁡ ( 𝒞 ) = supp t ⁡ ( 𝒞 ) max a ∈ 𝒞 ⁡ supp t ⁡ ( { a } ) . \operatorname{all-conf}{t}(\mathcal{C})=\frac{\operatorname{supp}{t}(\mathcal{C})}{\max{a\in\mathcal{C}}\operatorname{supp}{t}({a})}. (5)

A candidate is admitted as a new schema if

all − conf t ⁡ ( 𝒞 ) ≥ θ a . \operatorname{all-conf}{t}(\mathcal{C})\geq\theta{a}. (6)

This constraint requires the complete candidate to occur reliably whenever any of its constituent attributes occurs, thereby preventing an ubiquitous attribute from being merged with most patterns. To avoid admitting a qualified candidate together with its qualified subsets, AdaMM retains only inclusion-maximal novel candidates.

Schema Evolution.

For an existing schema 𝒜 \mathcal{A} , if 𝒞 ⊆ 𝒜 \mathcal{C}\subseteq\mathcal{A} , then 𝒞 \mathcal{C} is already subsumed by it. If 𝒞 \mathcal{C} contains 𝒜 \mathcal{A} , AdaMM treats 𝒞 \mathcal{C} as a potential extension of 𝒜 \mathcal{A} . Let Δ = 𝒞 ∖ 𝒜 \Delta=\mathcal{C}\setminus\mathcal{A} be the newly added attributes. We use extension confidence to measure how consistently Δ \Delta accompany occurrences of 𝒜 \mathcal{A} , defined as

ext − conf t ⁡ ( 𝒞 ∣ 𝒜 ) = supp t ⁡ ( 𝒞 ) supp t ⁡ ( 𝒜 ) \operatorname{ext-conf}{t}(\mathcal{C}\mid\mathcal{A})=\frac{\operatorname{supp}{t}(\mathcal{C})}{\operatorname{supp}_{t}(\mathcal{A})} (7)

When ext − conf t ⁡ ( 𝒞 ∣ 𝒜 ) ≥ θ e \operatorname{ext-conf}{t}(\mathcal{C}\mid\mathcal{A})\geq\theta{e} , the candidate provides sufficient evidence that Δ \Delta have become stable components of 𝒜 \mathcal{A} , and we update the schema 𝒜 \mathcal{A} to 𝒞 \mathcal{C} . Otherwise, 𝒜 \mathcal{A} remains unchanged.

3.3.3 Memory Materialization

Considering that each schema specifies a stable set of attributes and each matched interaction provides their co-occurring values, the induced patterns naturally align with relational tables. AdaMM therefore materializes each schema 𝒜 \mathcal{A} into a table T A T_{A} , using its attributes as columns and the matched interactions as rows.

Specifically, for a newly discovered schema 𝒜 \mathcal{A} , its attributes first define the data columns. Given these attributes and their sampled values from the corresponding extracted observations { 𝒪 t } t = 1 N {\mathcal{O}{t}}{t=1}^{N} , a language model generates the table metadata, including its name and description, and infers the type of each column. We then populate the table at the interaction level. For each associated round R t R{t} , every extracted tuple ( a t ​ ℓ , x t ​ ℓ , p t ​ ℓ ) ∈ 𝒪 t (a{t\ell},x{t\ell},p{t\ell})\in\mathcal{O}{t} whose attribute belongs to 𝒜 \mathcal{A} is mapped to the corresponding column, and values extracted from the same round jointly form one row. Two auxiliary columns, order and time , record the position and occurrence time of the source interaction, respectively. Any value not observed in that round is left empty. When a schema evolves from 𝒜 \mathcal{A} to 𝒜 ′ = 𝒜 ∪ Δ \mathcal{A}^{\prime}=\mathcal{A}\cup\Delta , the attributes in Δ \Delta are appended as new columns to the existing table T 𝒜 T{\mathcal{A}} . AdaMM then updates the table information and rematerializes the rows from their associated observations following the same procedure. The resulting analytic memory ℳ ana \mathcal{M}^{\mathrm{ana}} is represented as the collection of tables T 𝒜 T_{\mathcal{A}} .

3.4 Retrieval Memory Construction

Beyond structured analytic queries, many user requests can be addressed through direct semantic matching against past interactions. Retrieval memory is designed for such queries by preserving open-ended events, relations, and visual details that may not exhibit recurring schemas. It complements analytic memory by enabling flexible semantic retrieval and context-aware reasoning over interaction histories.

Following prior work on hierarchical memory organization (Yue et al. , 2026 ) , we organize retrieval memory ℳ ret \mathcal{M}^{\mathrm{ret}} into three levels: topic → episode → event \text{topic}\rightarrow\text{episode}\rightarrow\text{event} . Events preserve fine-grained evidence within individual interactions, episodes group temporally adjacent and semantically coherent events, and topics aggregate related episodes into broader semantic contexts. Each level maintains a concise description, separate textual and visual representations, and temporal information. Appendix A details its incremental construction.

3.5 Adaptive Query Processing

Different queries place different demands on memory: some require semantic retrieval over open-ended events, whereas others require structured computation over analytic memory. No single retrieval primitive can adequately support this full range of needs, calling for an adaptive mechanism that selects and composes operations according to query intent and the structures available in memory. AdaMM therefore abstracts the native capabilities of analytic and retrieval memory as operation-specific tools, and employs a joint query–memory planner to generate an executable procedure for answering each query.

3.5.1 Memory Access Tools

Analytic and retrieval memory differ in their underlying structures and consequently provide distinct native capabilities. To expose these capabilities at query time, AdaMM equips each memory with structure-specific tools.

Specifically, each tool is represented as

ξ i = ( d i , Θ i , Ω i ) , \xi{i}=(d{i},\Theta{i},\Omega{i}), (8)

where d i d{i} textually specifies its tool function, while Θ i \Theta{i} and Ω i \Omega{i} define its typed arguments and return schema. Table 1 summarizes the access tools designed for analytic and retrieval memory. For example, for the analytic tool Compute , d i d{i} specifies a deterministic computation over an induced table, Θ i \Theta{i} includes the target table, computation operator, target column, and optional constraints, and Ω i \Omega{i} returns the computed result and matched records. For the retrieval tool SemanticMatch , d i d{i} specifies relevance-based memory retrieval, Θ i \Theta{i} includes semantic query content and a retrieval budget, and Ω i \Omega_{i} returns ranked memory units with their relevance scores. Detailed specifications are provided in Appendix B .

Table 1: Access tools for analytic and retrieval memory.

Memory

Tools

Capability

Analytic

Lookup , Filter , Compute , Rank

Exact record access and deterministic computation.

Retrieval

SemanticMatch , EventLocate

Relevance-based retrieval and event localization.

3.5.2 Memory-Aware Joint Query Planning

The access tools specify how each memory can be queried. However, users describe desired outcomes rather than access procedures, and a feasible operation also depends on what the current memory contains. To bridge this gap, AdaMM first constructs a query-conditioned planning context that exposes relevant memory structure, and then employs an LLM-based planner to jointly reason over the query and this context, composing an executable procedure for retrieving and analyzing the required information. The resulting evidence is subsequently provided to the answer model for response generation.

Planning Context Construction.

Valid query planning requires knowledge of the current memory structures and their supported operations. Exposing the complete memory contents, however, introduces irrelevant information and expands the planning space. AdaMM therefore abstracts analytic and retrieval memory into compact metadata, retrieves query-relevant entries, and associates them with their available tools to form the effective planning context.

Specifically, we treat each analytic table T ∈ ℳ ana T\in\mathcal{M}^{\mathrm{ana}} and retrieval episode P ∈ ℳ ret P\in\mathcal{M}^{\mathrm{ret}} as an individual planning candidate represented by textual metadata μ T \mu{T} and μ P \mu{P} , respectively. For each table, μ T \mu{T} summarizes its name, description, columns, and sampled values, while each episode directly uses its description as μ P \mu{P} . Together, these candidates form the metadata search space 𝒰 \mathcal{U} . Given a query q q , AdaMM ranks each μ ∈ 𝒰 \mu\in\mathcal{U} using

h λ ​ ( q , μ ) \displaystyle h_{\lambda}(q,\mu) = λ ​ cos ⁡ ( E ​ ( q ) , E ​ ( μ ) ) \displaystyle=\lambda\cos!\left(E(q),E(\mu)\right) (9)

• ( 1 − λ ) ​ TokSim ⁡ ( q , μ ) , \displaystyle\quad+(1-\lambda)\operatorname{TokSim}(q,\mu),

  where E ​ ( ⋅ ) E(\cdot) is a semantic encoder and TokSim ⁡ ( ⋅ ) \operatorname{TokSim}(\cdot) is computed using BM25 over the tokenized query and metadata, capturing lexical matches following prior works Feng et al. ( 2026 ) . After retaining the top-ranked μ \mu , AdaMM augments their metadata with designed access tool descriptions, forming the planning context 𝒟 q \mathcal{D}_{q} to expose relevant memory candidates and available operations to the planner.

  Progressive Tool-Execution Planning.

  Complex queries often require multiple dependent operations, with the arguments of later tool calls determined by earlier outputs. Instantiating all tool calls upfront is therefore unreliable. To address this challenge, AdaMM separates high-level planning from progressive tool instantiation.

  High-level plan generation. Conditioned on the query q q and planning context 𝒟 q \mathcal{D}{q} , AdaMM first employs an LLM-based planner Planner ⁡ ( ⋅ ) \operatorname{Planner}(\cdot) to generate an abstract plan π ¯ ​ ( q , 𝒟 q ) = Planner ⁡ ( q , 𝒟 q ) = [ ( g ℓ , ξ ℓ ) ] ℓ = 1 L \bar{\pi}(q,\mathcal{D}{q})=\operatorname{Planner}(q,\mathcal{D}{q})=\big[(g{\ell},\xi{\ell})\big]{\ell=1}^{L} , which specifies the information goals g ℓ g{\ell} and the selected tool ξ ℓ \xi{\ell} at step ℓ \ell without binding them to concrete arguments.

  Progressive tool instantiation. Guided by the above high-level plan, AdaMM instantiates tool calls sequentially. At step ℓ \ell , it conditions on the preceding results 𝒵 < ℓ \mathcal{Z}_{<\ell} and optimizes:

  Θ ℓ = Planner ⁡ ( q , 𝒟 q , 𝒵 < ℓ , g ℓ , ξ ℓ ) , \displaystyle\Theta{\ell}=\operatorname{Planner}(q,\mathcal{D}{q},\mathcal{Z}{<\ell},g{\ell},\xi_{\ell}), (10)

  𝒵 < ℓ + 1 = 𝒵 < ℓ ∪ Invoke ⁡ ( ξ ℓ , Θ ℓ ) , \displaystyle\mathcal{Z}{<\ell+1}=\mathcal{Z}{<\ell}\cup\operatorname{Invoke}(\xi{\ell},\Theta{\ell}),

  where ξ ℓ \xi{\ell} and Θ ℓ \Theta{\ell} denote the selected tool and its instantiated arguments. This allows later calls to consume intermediate results (e.g., a timestamp returned by EventLocate can serve as a temporal constraint for a subsequent Compute call).

  Question Answering.

  Finally, the answer model generates response conditioned on the query and an evidence context comprising the instantiated plan and its tool outputs.

  4 Experimental Evaluation

  4.1 Experimental Setup

  Benchmarks.

  We evaluate AdaMM on two complex multimodal memory benchmarks. MemEye evaluates visual-memory granularity and reasoning through paired multiple-choice and open-ended queries (Guo et al. , 2026 ) , while MemGallery assesses long-term conversational memory management (Bei et al. , 2026 ) . MemEye uses EM, BLEU-1 and LLM-judge for evaluation, whereas MemGallery reports F1, BLEU-1, and LLM-Judge scores. Dataset statistics are provided in Table 2 .

  Benchmarks Sessions Rounds Images QA pairs

  MemEye 221 848 438 742

  MemGallery 240 3,962 1,003 1,711 Table 2: Benchmark statistics.

  Baselines.

  We compare against unimodal memory agents, including A-Mem and MemoryOS (Xu et al. , 2025 ; Li et al. , 2025 ) , which organize long-term textual interaction histories. Multimodal baselines include dedicated memory agents ( M2A , MMA , and MIRIX ) (Feng et al. , 2026 ; Lu et al. , 2026 ; Wang and Chen, 2025 ) , which construct persistent memories from dialogue and visual observations, and retrieval-based systems ( MM-RAG and UniversalRAG ), which directly retrieve relevant multimodal evidence for answer generation.

  Implementation details.

  We use GPT-4.1-nano and GPT-5.4-mini as answer and memory construction backbones for all methods. Text and image representations are produced by MiniLM-L6-v2 (Wang et al. , 2020 ) and siglip2-base-patch16-384 (Tschannen et al. , 2025 ) , respectively. We use the top-10 retrieved memory units for retrieval memory baselines, and constrain our planner to generate at most three execution steps under a shared budget of 10 evidence units. Results are averaged over 3 runs. Complete parameter and prompt details are provided in Appendix C and F .

  4.2 Main Results

  Backbone Method MemEye MemGallery

  EM BLEU-1 LLM-Judge F1 BLEU-1 LLM-Judge

  GPT-4.1-nano Unimodal A-Mem 39.1 17.1 32.0 53.2 47.2 67.4

  MemoryOS 42.4 15.8 27.6 53.4 47.1 65.9

  Multimodal M2A 34.1 5.8 13.8 50.2 44.3 61.5

  MMA 39.0 10.6 32.0 55.9 50.7 64.1

  MIRIX 40.9 5.7 13.6 56.3 51.1 66.3

  MM-RAG 42.7 16.6 42.2 58.3 52.4 67.3

  UniversalRAG 43.1 16.5 40.2 57.7 52.4 67.0

  AdaMM (Ours)

  50.4 (+7.3)

  21.2 (+4.1)

  48.0 (+5.8)

  62.6 (+4.3)

  56.7 (+4.3)

  74.3 (+6.9)

  GPT-5.4-mini Unimodal A-Mem 48.0 22.4 35.2 63.0 58.6 74.4

  MemoryOS 48.7 21.8 33.3 64.5 59.0 75.5

  Multimodal M2A 40.1 12.3 33.5 60.8 56.0 71.7

  MMA 53.9 25.7 43.3 64.0 59.6 73.3

  MIRIX 46.7 18.0 33.2 65.7 60.9 78.2

  MM-RAG 61.8 29.7 49.4 66.5 61.9 78.7

  UniversalRAG 62.4 30.4 48.6 65.6 60.8 77.4

  AdaMM (Ours)

  65.5 (+3.1)

  35.9 (+5.5)

  60.7 (+11.3)

  69.1 (+2.6)

  64.1 (+2.2)

  83.9 (+5.2) Table 3: Main results on MemEye and MemGallery. Higher is better for all metrics. Bold values denote the best result within a backbone and red values denote improvements over the strongest baseline.

  Overall performance.

  Table 3 presents a comprehensive comparison of AdaMM with representative unimodal and multimodal memory frameworks across two benchmarks and answer backbones. AdaMM consistently achieves the best result on every metric. With GPT-4.1-nano, it surpasses the strongest competing result on MemEye by 7.3% and 5.8% percentage points for MCQ and open-ended questions, respectively, and improves MemGallery F1, BLEU-1, and LLM-Judge by 4.3%, 4.3%, and 6.9%. The advantage persists with GPT-5.4-mini, yielding gains of 3.1% and 11.3% points on MemEye and 2.6%, 2.2%, and 5.2% on the three MemGallery metrics. The consistent improvements on different settings demonstrate that AdaMM generalizes across answer formats, benchmark settings, and backbone capacities.

  Fine-grained task analysis.

  To further examine where the improvement arises, Figure 3 reports the LLM-Judge breakdown using GPT-5.4-mini. On MemEye, the largest gains over the strongest baseline occur on Card Playlog and Personal Health, with improvements of 18.8% and 16.7%, respectively. Both tasks require exact operations over recurring records, such as filtering observations, comparing values, or tracking changes, highlighting the benefit of analytic memory. AdaMM also achieves substantial gains on Multi-scene (11.1%), Brand Memory (9.0%), Outdoor Navigation (8.6%), and Social Chat (8.2%), where relevant evidence must be retrieved and connected across interactions. The smaller gains on Cartoon Entertainment (6.1%) and Home Renovation (5.2%) are consistent with their greater reliance on direct visual-semantic recall. On MemGallery, the largest margins appear in Conflict Detection (10.6%), Knowledge Resolution (10.5%), and Factual Retrieval (8.1%), demonstrating the value of structured access for reconciling and precisely locating information. AdaMM further improves Multi-entity Reasoning, Visual-centric Reasoning, and Visual-centric Search, while remaining comparable to the strongest baselines on Answer Refusal, Temporal Reasoning, and Test-time Learning. Together, these results suggest that combining retrieval with operation-specific analytic access is particularly beneficial when answering requires more than semantic relevance alone. Further analysis of cost and case study are provided in Appendix D and E .

  Figure 3: LLM-as-Judge performance across fine-grained tasks with GPT-5.4-mini.

  4.3 Ablation Study

  Figure 4: Ablation results on MemEye. Overall and task-level LLM-as-Judge accuracy using GPT-5.4-mini.

  We evaluate four ablated variants on MemEye to examine the contributions of the designed memories and the memory-aware planning mechanism. The Health task emphasizes numerical and temporal analysis over multiple personal-health records, whereas the Brand task primarily evaluates visually grounded recall of brand-related information. In w/o Analytic and w/o Retrieval , we disable the analytic memory and retrieval memory, respectively. In w/o Planning Context , the planner receives the entire memory metadata. In w/o Progressive Execution , all tool calls and their arguments are instantiated before execution.

  Contributions of Complementary Memories.

  As shown in Figure 4 , removing either memory component consistently degrades performance. Without analytic memory, overall accuracy decreases by 4.6%, with a substantially larger 14.9% drop on Health. This result highlights the importance of organizing recurring records into executable structures for analytical queries. Removing retrieval memory instead causes a 7.5% overall drop and is particularly detrimental to Brand, where performance decreases by 9.9%. The distinct degradation patterns confirm that analytic and retrieval memory provide complementary capabilities.

  Contributions of Memory-Aware Planning.

  Removing the planning context reduces performance by 3.3% overall and by 6.7% and 6.3% on Health and Brand, respectively. This demonstrates that effective tool selection requires knowledge of both the query intent and the memory structures currently available. Instantiating all calls before execution also produces consistent degradation, including drops of 4.2% on Health and 3.2% on Brand. These results indicate that progressive execution is useful when later operations depend on information obtained from preceding calls.

  5 Conclusion

  In this work, we identify a retrieval–analysis mismatch in long-term multimodal agent memory, highlighting the need for executable operations over accumulated experience beyond retrieval. To address this issue, we introduced AdaMM , which couples hierarchical retrieval memory with schema-induced analytic memory and exposes their distinct capabilities through operation-specific tools. A memory-aware planner further grounds tool selection in the current memory state and progressively composes retrieval and analytic operations. Experiments on MemEye and MemGallery across two answer backbones demonstrate consistent improvements from combining flexible recall with executable analysis.

  Limitations

  While AdaMM demonstrates promising results in long-term multimodal agent memory, two limitations remain. First, analytic memory depends on record fragments extracted from multimodal interactions. Incorrect or missing fields may propagate to schema induction, table construction, and downstream computations. Future work could incorporate confidence-aware extraction and cross-round consistency checks to improve robustness. Second, AdaMM uses a predefined set of access tools and thus requires manual extension for unseen, domain-specific operations. Future work could develop a self-evolving tool framework that identifies emerging capability gaps and safely synthesizes, validates, and integrates new tools.