What happened
Analysis and impact
该研究提出了名为 MerchantBench 的评测基准，其核心是一个为期一年的订单级电商运营模拟器。这个环境并非凭空设计，而是基于 98,843 条真实的商品记录构建，并为智能体提供了涵盖产品采购、定价、订单追踪和现金流管理等环节的 26 种工具 [来源：huggingface.co]。与以往评测不同，MerchantBench 的关键设计在于引入了延迟反馈机制：智能体不仅要应对即时的供应商事件，还必须处理退款、差评和罚款等需要追溯历史决策的滞后结果，这迫使智能体必须具备长期记忆和策略连贯性 [来源：hyper.ai]。
研究团队在两种智能体框架下，对 8 款大语言模型进行了共计 48 次、每次长达 365 个模拟日的运行测试 [来源：arxiv.org]。结果暴露了当前智能体的一个核心短板：长期运营的连贯性严重不足。在衡量最终盈利能力的净收益指标上，表现最佳的智能体配置也仅达到人类参与者水平的 27.3% [来源：arxiv.org]。研究发现，智能体在长期任务中普遍出现活动质量下降、过早放弃原始目标以及策略切换失当等问题，这凸显了从短期任务成功到长期商业部署之间的巨大鸿沟 [来源：hyper.ai]。
该论文已于近期在预印本平台 arXiv 发布，并引起了社区的关注。在一段研究解读视频中，播主 Alex 指出，这项发表于 2026 年 7 月底的研究表明，顶级自主智能体在长期运营中仅能达到人类净收益的 27% [来源：youtube.com]。对于希望深入研究实现细节的开发者，作者已在 GitHub 上公开了相关代码 [来源：youtube.com]。这项研究为评估和提升大模型智能体在持续性商业场景中的可靠性，提供了一个更为严苛和贴近现实的测试平台。
References
Original source text
MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations
Qiming Shi 2 , Yulong Tao 1 , Linbo Jin 1 1 1 footnotemark: 1 , Zhaolu Kang 4 , Yibo Dou 4 , Jiawen Zhu 3 , Tianjun Pan 5 , Shaokang Fu 1 , Chengyu Wang 1 , Siyue Li 1 , Yaping Cheng 1 , Di Weng 3 , Chengfu Huo 1 Corresponding authors.
Abstract Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3% of the mean final net assets achieved by human participants. Our code is available at https://github.com/KhanCold/merchantbench .
Introduction
Large language models have become a foundation for autonomous agents that plan, call tools, interact with external systems, and make decisions over multiple turns. General agent benchmarks now measure tool use, web interaction, application control, and state-changing workflows (Liu et al. 2024 ; Zhou et al. 2024 ; Yao et al. 2025 ; Trivedi et al. 2024 ) . Yet many real-world deployments are not bounded tasks with immediate and unambiguous completion criteria. They require agents to operate over extended horizons in environments whose state persists, where earlier actions constrain later options and relevant consequences may emerge only after many intervening decisions. Evidence from recent benchmarks indicates that even state-of-the-art agents often fail to maintain coherent performance over long horizons (Luo et al. 2025 ; Wang et al. 2025c ; Yuan et al. 2026 ) . These results motivate extending agent evaluation beyond isolated task completion to examine sustained objective pursuit and decision consistency in realistic long-horizon environments.
Figure 1: Order-level dynamics couple immediate liquidity pressure with delayed feedback. New orders commit available cash before settlement, whereas abnormal outcomes surface later and affect store rating.
Seller-side e-commerce provides a suitable setting for evaluating Long-Term Coherence. Unlike bounded tasks with explicit completion criteria, online store operation requires continued intervention throughout the operating horizon. The agent must repeatedly select products, control listings and prices, manage limited cash, and revise earlier decisions as market conditions, supplier states, and order outcomes evolve. Seller-side e-commerce therefore tests whether an agent can sustain and revise a merchant policy over time, rather than complete an isolated action.
Vending-Bench and RetailBench have taken important steps toward realistic long-horizon business evaluation (Backlund and Petersson 2025 ; Zhang et al. 2026 ) . Vending-Bench evaluates vending-machine operation, while RetailBench models supermarket management over a fixed catalog of 96 grocery products. Seller-side e-commerce, however, introduces two challenges that are not represented together in these environments.
First, e-commerce feedback is generated through individual order lifecycles. A listing or pricing decision may create new orders and commit available cash immediately, whereas fulfillment failures and after-sales outcomes become observable only later. Figure 1 visualizes this temporal asymmetry between immediate order-driven cash commitments and delayed abnormal outcomes. As delayed evidence accumulates, the agent must associate later outcomes with earlier decisions and determine whether its current merchant policy should be maintained or revised.
Second, a long episode alone does not meaningfully test Long-Term Coherence if the available products and their demand remain static. A large, data-grounded Product Catalog with full-year demand trajectories creates a changing opportunity set in which promising products emerge and existing choices lose value over time. The agent must therefore continually identify new opportunities and revise its portfolio using market signals and realized order outcomes.
To address this gap, we introduce MerchantBench, which evaluates Long-Term Coherence through persistent seller-side e-commerce operation. MerchantBench formulates e-commerce operation as a partially observable decision-making problem over 365 simulated days. The environment grounds a Product Catalog in 98,843 real e-commerce product records and converts demand into individual orders that progress through fulfillment and after-sales stages. Through merchant-visible tools, the agent performs Product Sourcing, controls listings and prices, manages cash flow, and monitors supplier and order states. Upstream Supplier Events and Downstream Order Outcomes become observable at different times, requiring the agent to use later evidence to maintain or revise earlier decisions. Across the operating horizon, the simulator updates demand, supplier states, order lifecycles, cash flow, penalties, and store reputation.
Our contributions are as follows:
• To our knowledge, we introduce the first benchmark for evaluating Long-Term Coherence through persistent seller-side e-commerce operation, structured around four interdependent decision components.
• We develop an order-level simulation environment grounded in 98,843 real e-commerce product records, with partial observability, cash constraints, Upstream Supplier Events, and delayed Downstream Order Outcomes.
• We conduct 48 runs of 365 simulated days across eight LLMs and two agent frameworks and analyze business outcomes and decision traces to characterize their Long-Term Coherence.
Related Work
Agent Evaluation in Commerce.
Existing benchmarks cover shopping and storefront interaction (Yao et al. 2022 ; Wang et al. 2026a ; Zhang et al. 2025 ; Wang et al. 2026b ; Savadikar et al. 2026 ; Du et al. 2026 ) , customer support (Wang et al. 2025a ) , and merchant workflows and negotiation (Zhao et al. 2026 ; Wang et al. 2025b ) . Market-Bench and Magentic Marketplace examine market competition and transactions among economic agents (Zheng et al. 2026 ; Bansal et al. 2025 ) . Across these categories, evaluation centers on tasks, dialogues, transactions, or competitive episodes rather than continuous operation of the same online store.
Long-Horizon Agent Evaluation.
Recent long-horizon benchmarks assess sustained reasoning and action across workplace workflows, open-ended exploration, virtual-world planning, web navigation, computer use, inventory control, order fulfillment, and interactive economies (Luo et al. 2025 ; Wang et al. 2025c ; Anokhin et al. 2025 ; Jang et al. 2026 ; Ding et al. 2026 ; Yuan et al. 2026 ; Baek et al. 2026 ; Zhu et al. 2023 ; Hu et al. 2026 ; Sugiura et al. 2026 ) . Vending-Bench and Vending-Bench 2 emphasize sustained business operation, while RetailBench evaluates evidence acquisition, action conversion, and temporal follow-up in supermarket management (Backlund and Petersson 2025 ; Andon Labs 2025 ; Zhang et al. 2026 ) . MerchantBench extends this line by evaluating how agents adapt to delayed order-level feedback and nonstationary demand derived from real e-commerce data.
MerchantBench
Figure 2: Overview of MerchantBench. The merchant agent coordinates four decision components through shorter supplier feedback and delayed order outcomes over 365 days. The environment combines an upstream supplier simulation, a merchant store, and a downstream order level simulation to evaluate Long-Term Coherence and business outcomes.
Task Formulation
We formulate store operation as a finite horizon partially observable Markov decision process (POMDP) (Kaelbling et al. 1998 )
ℳ = ⟨ 𝒮 , 𝒜 , P , 𝒪 , Z , R , μ 0 , H c ⟩ . \mathcal{M}=\langle\mathcal{S},\mathcal{A},P,\mathcal{O},Z,R,\mu{0},H{c}\rangle. (1)
The simulator advances hourly over a 365 day control horizon, giving H c = 8 , 760 H{c}=8{,}760 steps indexed by t ∈ { 0 , … , H c − 1 } t\in{0,\ldots,H{c}-1} . Demand, supplier states, and order lifecycles evolve at every step, while the agent receives a decision window once every 12 steps. The latent state s t ∈ 𝒮 s{t}\in\mathcal{S} contains the simulation clock, product demand profiles, supplier conditions, store listings and finances, active orders, and pending events. The initial state follows μ 0 \mu{0} and includes the cash balance, security deposit, and listing capacity. The cash balance funds procurement and realized losses, unpaid fines draw from the security deposit, and operation terminates when the deposit is exhausted. At activation steps, a t ∈ 𝒜 a{t}\in\mathcal{A} denotes the sequence of merchant tool invocations within the decision window, while other steps use a fixed null action. The transition kernel P ( s t + 1 ∣ s t , a t ) P(s{t+1}\mid s{t},a{t}) combines tool induced store changes with autonomous demand, supplier, and order evolution, with listing and pricing changes affecting demand from the next step. The observation kernel Z ( o t + 1 ∣ s t + 1 , a t ) Z(o{t+1}\mid s{t+1},a{t}) exposes only merchant visible information, while demand profiles, risk parameters, pending outcomes, and future event times remain latent until their effects become observable. The policy therefore conditions on observation and tool result history rather than the full state. At t = H c t=H{c} , new demand and agent activations stop while active orders continue until terminal settlement at a terminal step T ≥ H c T\geq H_{c} . Intermediate rewards are zero, and the objective is expected terminal net assets
J ( π ) = 𝔼 π [ R ( s T ) ] , R ( s T ) = B T + D T + I T + Q T . \begin{array}[]{rcl}J(\pi)&=&\mathbb{E}{\pi}[R(s{T})],\ R(s{T})&=&B{T}+D{T}+I{T}+Q_{T}.\end{array} (2)
where B T B{T} , D T D{T} , I T I{T} , and Q T Q{T} denote the terminal cash balance, security deposit, funds in transit, and receivables. Thus, R ( s T ) R(s_{T}) is the realized net asset value of one run and J ( π ) J(\pi) is its expectation across stochastic trajectories.
Real-World Data Grounding
MerchantBench is grounded in real-world e-commerce data from 1688, the largest integrated domestic wholesale marketplace in China (Alibaba Group n.d. ) . The data contain product and supplier attributes, 365-day product-level demand histories, and platform quality and fulfillment signals. The data cover 365 days from June 1, 2025 through May 31, 2026. Alongside the product data, MerchantBench incorporates 365 daily market reports from 1688 as date-aligned signals for product sourcing. We select 10 first-level product categories spanning apparel, household and office goods, appliances, pet and gardening products, toys, bags, and sports and outdoor products. After excluding records with missing identifiers, unmapped categories, nonpositive prices, or incomplete demand histories, the dataset contains 98,843 products from 36,576 suppliers. Through catalog and supplier tools, the agent accesses only public product and supplier attributes. Figure 3 captures aggregate demand peaks at 618 and during the first wave and final day of 11.11, together with a trough during the Spring Festival. The lower panels show the distributions of effective Upstream Supplier Event and Downstream Order Outcome probabilities.
Upstream Supplier Simulation
The supply pool maintains time varying procurement prices, available inventory, availability, and supplier shipment times, while inventory replenishes over time. At each simulator step, the simulator samples the three Upstream Supplier Events, namely Price Change, Product Delisting, and Shipment Delay, using product level probabilities calibrated from real platform fulfillment signals. These events alter the upstream procurement price, suspend product procurement, and extend supplier dispatch time, respectively. Inventory stockouts arise endogenously when incoming orders deplete stock faster than it replenishes. To prevent persistent environment drift, each triggered abnormality receives a sampled recovery time at which the affected supplier attributes return to their base states. The agent can observe realized changes to price, availability, quantity, and shipment time through catalog and supplier queries, but it cannot access the underlying abnormality flag, trigger probability, or recovery schedule.
Figure 3: Real-world demand patterns and calibrated risk profiles across 98,843 products. Whiskers span the 10th to 90th percentiles, and boxes show interquartile ranges.
Downstream Order Simulation
Order Level Simulation.
The downstream simulation converts product level daily demand traces into individual orders. For product i i listed by merchant m m at time t t , the hourly arrival intensity is
λ m , i , t = D i , d ( t ) w c ( i ) , h ( t ) r m , t ℓ m , i , t ( p m , i , t / p i ref ) − ϵ i . \lambda{m,i,t}=D{i,d(t)}w{c(i),h(t)}r{m,t}\ell{m,i,t}\left(p{m,i,t}/p{i}^{\mathrm{ref}}\right)^{-\epsilon{i}}. (3)
The indices d ( t ) = ⌊ t / 24 ⌋ d(t)=\lfloor t/24\rfloor and h ( t ) = t mod 24 h(t)=t\bmod 24 denote the data day and hour of day associated with step t t . The quantity D i , d ( t ) D{i,d(t)} is the linked daily demand from the real-world data, and w c ( i ) , h ( t ) w{c(i),h(t)} distributes category demand across hours. The price term uses product elasticity ϵ i \epsilon{i} , while r m , t r{m,t} and ℓ m , i , t \ell{m,i,t} capture store rating and listing exposure. Realized order outcomes update the published store rating after each completed day, and its discrete star level determines r m , t r{m,t} . The factor ℓ m , i , t \ell{m,i,t} increases during a listing’s cold start and then decays with age. The appendix provides precise definitions of both factors in the section on demand and rating dynamics. The environment samples N m , i , t ∼ Poisson ( λ m , i , t ) N{m,i,t}\sim\mathrm{Poisson}(\lambda_{m,i,t}) and instantiates each arrival as an order candidate. At creation, each candidate receives one latent customer outcome from normal fulfillment, Cancellation, Returnless Refund, Return and Refund, or Bad Review according to its product specific risk profile. Stockout and Late Shipment instead arise from procurement and fulfillment dynamics. The selected outcome and its realization time remain hidden until the corresponding lifecycle transition occurs.
Model or Operator Business Performance Store Reliability Long-Horizon Activity
Net Assets
GMV
Profit Margin
Orders
Fines
Avg. Store Rating
Anomaly Rate
Avg. Active Listings
SWR
Tool Calls
ReAct
GPT-5.6 Sol
40.89
74.19
51.3
996
499
4.04
10.7
50.0
99.4
7,257
Claude Opus 4.8
31.89
69.10
44.4
1,214
796
4.05
12.1
24.0
45.0
1,139
Qwen3.7-Max
20.66
39.73
44.5
925
672
3.90
16.1
39.6
11.1
815
Qwen3.7-Plus
20.74
40.85
45.6
1,056
705
3.99
13.1
49.9
52.2
1,221
GLM-5.2
25.73
60.90
37.3
2,158
1,422
3.93
14.9
26.0
53.3
2,045
DeepSeek-V4-Pro
6.56
8.40
41.9
450
245
4.01
14.4
23.5
30.6
660
DeepSeek-V4-Flash
14.47
28.78
39.6
985
517
4.04
14.1
19.3
40.6
960
Kimi K2.6
24.99
63.69
32.9
2,230
1,474
3.89
15.3
47.3
10.6
1,228
Hermes
GPT-5.6 Sol
52.93
133.07
40.2
3,251
1,096
4.09
9.2
50.0
66.1
4,831
Claude Opus 4.8
35.56
83.23
39.9
1,808
1,089
4.02
11.8
22.1
31.7
1,138
Qwen3.7-Max
59.46
116.76
46.9
1,929
1,295
3.90
15.7
49.6
22.2
1,366
Qwen3.7-Plus
29.42
53.69
48.9
981
642
3.95
13.8
49.9
19.4
820
GLM-5.2
42.32
103.06
36.9
2,731
1,454
4.05
11.3
49.6
62.8
1,792
DeepSeek-V4-Pro
16.71
31.95
43.4
1,062
665
3.98
14.5
33.0
33.3
942
DeepSeek-V4-Flash
24.69
64.52
37.6
1,989
1,774
3.93
16.0
48.8
62.2
1,259
Kimi K2.6
23.96
75.06
26.8
3,398
2,671
3.73
19.1
48.3
17.8
969
Others
Human
217.61
608.06
35.3
9,442
5,622
3.98
12.5
49.1
100.0
8,311
Rule-based
24.48
53.37
40.3
1,605
1,374
3.76
18.0
50.0
100.0
3,236
Table 1: Business performance, store reliability, and long-horizon activity after 365 simulated days. Values are means over three runs. Final net assets and GMV are reported in thousands of RMB, total fines in RMB, and rate metrics in percent. SWR denotes Sustained Window Rate. The best result within each framework is shown in bold.
Order Lifecycle.
MerchantBench follows the single item drop shipping model supported by 1688, in which merchants hold no inventory in advance. Each Order Placed triggers immediate procurement at the current supplier price, and successful procurement deducts the cash balance, decreases supplier inventory, and moves the order to Procured. Following supplier and logistics delays, the order advances through Shipped and Delivered, at which point the sale price becomes a receivable. A normal order becomes Settled after a sampled delay and credits the receivable to the cash balance.
MerchantBench models Normal Fulfillment together with six abnormal Downstream Order Outcomes, namely Cancellation, Stockout, Late Shipment, Returnless Refund, Return and Refund, and Bad Review. Cancellation restores the procurement cost, while Stockout prevents procurement. Late Shipment marks a missed dispatch deadline, after which the order continues through fulfillment and settlement. Both refund outcomes remove the receivable, but only Return and Refund restores the procurement cost, whereas Bad Review preserves the sales revenue. Stockout, Late Shipment, Return and Refund, and Bad Review incur platform fines. All abnormal outcomes except Cancellation also contribute adverse evidence to the store rating through outcome-specific experience scores and evidence weights.
Agent Interface
MerchantBench exposes a shared observation protocol and 26 merchant tools, allowing different agent frameworks to interact with identical environment dynamics and observable state. At each decision window, the agent receives a summary of simulated time, store status, and recent supplier and order changes, then acts until it ends the window or reaches the time limit. The appendix lists the complete MerchantBench tool interface.
Product Sourcing.
Daily market reports, catalog search, product details, and public supplier profiles support product selection, while demand, product risk rates, future order outcomes, and future supplier events remain latent.
Listing and Pricing Control.
Listing, delisting, repricing, and performance views allow agents to construct the store portfolio and revise its products and prices.
Cash-Flow Management.
Finance and store views expose the cash balance, security deposit, committed funds, expected settlements, fines, and closure conditions, which agents manage through subsequent sourcing, listing, delisting, and pricing decisions.
Mixed-Latency Feedback Adaptation.
Supplier and order tools reveal Upstream Supplier Events and Downstream Order Outcomes as they unfold, allowing agents to revise the other three decision components in response to mixed-latency feedback.
Experiments
Experimental Setup
Agent Configurations and Baselines.
We evaluate eight LLMs under ReAct (Yao et al. 2023 ) and Hermes (Nous Research 2026 ) with three runs for each pairing of an LLM and a framework. The evaluated models are GPT-5.6 Sol (OpenAI 2026 ) , Claude Opus 4.8 (Anthropic 2026 ) , Qwen3.7-Max and Qwen3.7-Plus (Qwen Team 2026 ) , GLM-5.2 (Z.ai 2026 ) , DeepSeek-V4-Pro and DeepSeek-V4-Flash (DeepSeek-AI 2026 ) , and Kimi K2.6 (Moonshot AI 2026 ) . ReAct pairs each model with a minimal controller over the 26 MerchantBench tools to assess core planning, reasoning, and tool use. Hermes uses its default configuration, combining the 26 MerchantBench tools with built in capabilities for code execution, planning, memory, and skill management. Both frameworks compress long interaction histories, with each evaluated model also serving as its own summarizer. Each run starts with RMB 2,000 in cash, a RMB 1,000 security deposit, and capacity for 50 active listings. We additionally compare with a Rule-based baseline and three Human participants without prior e-commerce operating experience. The Rule-based baseline performs daily checks, removes inactive or supplier-affected products, and fills open listing slots using the daily market report. The appendix provides the detailed experimental settings.
Evaluation Metrics.
We evaluate Business Performance using Final Net Assets, GMV, Net Profit Margin, and Orders; Store Reliability using Total Fines, Average Store Rating, and Order Anomaly Rate; and Long-Horizon Activity using Average Active Listings, Sustained Window Rate (SWR), and Total Tool Calls. SWR is the minimum share of scheduled decision windows containing at least one environment tool call across all rolling 30 day periods.
Main Results
Overall Performance.
Table 1 reports the final performance of all evaluated configurations and baselines. GPT-5.6 Sol records the highest final net assets under ReAct, whereas Qwen3.7-Max ranks first under Hermes. Qwen3.7-Max with Hermes achieves the highest final net assets among all 16 configurations. When results are aggregated by model across the two frameworks, GPT-5.6 Sol has the highest average final net assets.
Performance Variability.
Figure 4 shows substantial differences in stability across configurations. GPT-5.6 Sol under ReAct and Claude Opus 4.8 under Hermes have the lowest coefficients of variation within their respective frameworks at 3.3% and 10.0%. Despite achieving the highest mean final net assets, Qwen3.7-Max under Hermes is considerably less stable, with a coefficient of variation of 55.1%.
Figure 4: Final net asset distributions across three repeated runs for each configuration.
Framework Analysis.
Averaged across the eight models, Hermes produces 53.3% higher final net assets, 71.5% higher GMV, and 71.2% more orders than ReAct. Mean final net assets are higher under Hermes for seven of the eight models, with gains ranging from 11.5% for Claude Opus 4.8 to 187.8% for Qwen3.7-Max. Kimi K2.6 is the sole exception, with mean final net assets 4.1% lower under Hermes than under ReAct, showing that framework benefits depend strongly on the underlying model.
Order-Level Risk Propagation
Figure 1 illustrates the mixed-latency evidence generated by MerchantBench’s order-level simulation. Agents observe prior ratings of upstream catalog products during sourcing and receive prompt demand signals from realized sales, whereas product quality is revealed only through delayed order outcomes that may require product-level risk response or store-level rating adaptation.
Product-Level Risk Response.
Models differed in whether they converted delayed order outcomes into product-specific interventions. In representative runs, GPT-5.6 Sol and Kimi K2.6 attributed adverse outcomes to the responsible listing and replaced it, whereas Qwen3.7-Plus retained a risky product for further observation and DeepSeek-V4-Pro did not revise affected listings after refunds. Human participants described a more complete response for popular but risky products, in which they delisted the product and searched similar keywords for a replacement that could preserve the underlying demand. These behaviors distinguish simple anomaly detection from the full chain of product attribution, risk removal, and demand-preserving replacement.
Store-Level Rating Adaptation.
Order-level anomalies also reduce the store rating, allowing product-specific failures to affect demand across the portfolio. In a GPT-5.6 Sol trajectory, the agent lowered prices on proven products without adverse outcomes to increase normal settlements and recover the rating threshold. Qwen3.7-Max applied the same mechanism more aggressively by repricing 40 listings after an early bad review reduced the store to three stars.
Long-Term Coherence Analysis
The aggregate results reveal a substantial gap between the evaluated LLM agents and Human operators. Our trace evidence suggests that two forms of Long-Term Coherence failure developing over extended operation may contribute to this performance gap. Some agents progressively reduce store intervention and fail to follow up on prior decisions or delayed outcomes, indicating a loss of Operational Coherence. Others remain active but drift from the terminal net assets objective or fail to revise ineffective policies as evidence accumulates, indicating a loss of Strategic Coherence.
Figure 5: Monthly Operational Coherence profiles for the Human baseline and all eight Hermes models. Panels report effective window rate, environment tool calls, and monthly net profit.
Operational Coherence.
Figure 5 reveals substantial differences in whether merchant activity persists across the operating horizon. Table 1 shows that Human operators retain an SWR of 100%, while LLM configurations range from 10.6% to 99.4% under ReAct and from 17.8% to 66.1% under Hermes. Qwen3.7-Max provides the clearest contrast, with its quarterly Effective Window Rate falling from 62% to 37% under Hermes and more sharply from 68% to 23% under ReAct, alongside substantial reductions in environment tool calls. Similar patterns of Activity Decay appear across several other models under both frameworks. Monthly net profit shows how business performance evolves alongside these activity patterns. Such operational decline often originates in strategic drift.
Strategic Coherence.
Strategic Coherence comprises two complementary dimensions. Goal Consistency concerns whether decisions across time continue to serve the long-term business objective, whereas Evidence-Calibrated Adaptation concerns how an agent maintains or revises its policy as feedback arrives at different latencies and evidence accumulates over time.
Goal Consistency. Goal Consistency fails when agents lose the autonomy to keep pursuing terminal net assets. Under Control-Loop Narrowing, the sourcing and operating loop gradually collapses into reactive handling of Upstream Supplier Events, with little self-initiated sourcing, repricing, replacement, or diagnosis. For ReAct Qwen3.7-Max, an SWR of 11.1% coincides with supply chain checks rising from 14% to 34% of its remaining tool calls. Related activity decay also appears in Claude Opus 4.8, GLM-5.2, DeepSeek-V4-Pro, and Qwen3.7-Plus under Hermes. At the extreme, Premature Abandonment occurs when an agent concludes that the store cannot recover although feasible actions remain. In one Hermes Kimi K2.6 run, the agent made this judgment on Day 104 and then took no environment action in 355 of the remaining 523 decision windows. In both cases, the agent waits for external events or time to change the store instead of operating it autonomously.
Evidence-Calibrated Adaptation. Evidence-Calibrated Adaptation examines whether agents revise their policies as liquidity, seasonal demand, and accumulated experience change.
Under initial liquidity constraints, Human and agents operate within similar low price ranges. As liquidity increases, Human operators broaden the procurement price range and selectively return to lower price and higher throughput products when higher value experiments underperform. Their mean active listing procurement prices increase from between RMB 43.4 and RMB 53.1 in the first three months to between RMB 58.7 and RMB 90.8 in the last three months, whereas GLM-5.2, DeepSeek-V4-Flash, and Kimi K2.6 retain comparatively flat listing price trajectories.
Dynamic market demand makes Product Sourcing a continual portfolio allocation problem rather than a one-time selection decision. Figure 6 first measures monthly portfolio alignment, with Human rising from 56.1 in June to above 80 in December and January while Rule-based remains near the catalog median and LLM improvements are weaker or less consistent. Figure 5 (c) then reports realized profit, with Human peaking in winter while the winter profit gains of Hermes Qwen3.7-Max and GPT-5.6 Sol fade in spring. Claude Opus 4.8 further shows that demand alignment alone is insufficient, since its stronger alignment in later months coincides with a shelf contraction from 37.3 to 12.0 products and no corresponding profit improvement. Together, the figures show that effective long horizon operation requires alignment with changing demand, sufficient portfolio breadth, and conversion of that alignment into realized returns.
Figure 6: Monthly Product Sourcing across four representative models. Scores are listing-hour-weighted catalog demand percentiles of active products in each month. Lines and bands show repeat means and standard deviations.
Memory traces further show how local errors become persistent policies. In one Hermes Claude Opus 4.8 run, the agent falsely inferred that removing weak listings would concentrate traffic on the remaining products, while its shelf contracted from 47 active listings on Day 54 to three on Day 322 despite independent demand opportunities for every listing. In one Hermes Qwen3.7-Max run, the agent misremembered Day 285 as the endpoint on Day 282 and stopped filling vacant slots with 83 days remaining, correcting the error only after simulated time advanced beyond the assumed endpoint.
Conclusion
We introduced MerchantBench to evaluate Long-Term Coherence through persistent seller-side e-commerce operation in a 365-day order-level environment grounded in 98,843 real e-commerce product records. Across 48 runs involving eight LLMs and two agent frameworks, LLM agents exhibit a substantial gap from the Human baseline in system-level performance. Trace analyses further show that weaker outcomes accompany declining operational activity, premature goal abandonment, and strategy changes that are not calibrated to accumulated evidence.