Harnessing the Wisdom of LLM Crowds through Complementarity-Driven Iterative Collaboration

Yanbin Fang Business School Chinese University of Hong Kong, Shanghai, China yibofang@link.cuhk.edu.hk &Xuan Wei Antai College of Economics and Management Shanghai Jiao Tong University, Shanghai, China weix@sjtu.edu.cn &Wei Chen School of Business University of Connecticut, Stamford, CT, US weichen@uconn.edu

Abstract Large language models (LLMs) are increasingly deployed in enterprise contexts to support complex problem-solving tasks. Yet individual LLMs remain bounded by model-specific capability limitations. These heterogeneous capability boundaries pose a deployment challenge, but they also create an opportunity: strategically coordinating multiple LLMs may unlock collective intelligence that exceeds the performance of any single model. Existing approaches fix how models are combined in advance, overlooking the dynamic and state-dependent role of complementarity in complex LLM problem solving. Drawing on the wisdom-of-crowds paradigm, we reconceptualize collective LLM intelligence as relay-style complementarity: a sequential coordination process in which each successor model is selected to address the specific bottleneck identified in its predecessor’s output. To operationalize this relay-style complementarity, we propose WILC (Wisdom Integration of LLM Crowds), a framework grounded in two design principles. First, iterative reflection-and-refinement establishes a state-preserving workflow through which models diagnose and refine prior outputs. Second, complementarity-driven model selection governs model transitions through a dual-gate mechanism: prospective complementarity fit (PCF) identifies the worker most suited to the current bottleneck, while posterior complementarity gain (PCG) evaluates whether the selected transition improves the evolving solution. Together, these design principles support strategic model transitions that refine answers through capability complementarity. Extensive experiments across four diverse benchmarks show that WILC achieves superior performance compared to existing approaches, including single-model self-refinement, ensemble methods, and dedicated query-routing methods. Under standardized pricing assumptions, WILC achieves comparable average benchmark performance to GPT-5.2 at approximately 7 × \times lower estimated per-query cost, while facilitating data sovereignty through self-hosted deployment. This study contributes to IS research by extending wisdom-of-crowds theory from static aggregation to sequential AI complementarity and by providing transferable design principles for multi-AI coordination.

K eywords Large Language Models ⋅ \cdot Wisdom of Crowds ⋅ \cdot Capability Complementarity ⋅ \cdot Iterative Refinement ⋅ \cdot Computational Design Science ⋅ \cdot Contextual Multi-Armed Bandit

1 Introduction

Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks and have been increasingly adopted in enterprise contexts. For instance, firms leverage LLMs internally to support employees in resolving operational or analytical challenges, and deploy them externally to deliver LLM-driven services to users through conversational or interactive applications (Haki et al. 2025 , Raza et al. 2025 ) . Such deployments, however, require organizations to choose among many available models that differ markedly in capability. This heterogeneity stems from variations in training data, model architectures, and training techniques (Shnitzer et al. 2023 , Lu et al. 2024 ) . Even as frontier models continue to advance, their relative strengths remain uneven across task types, and the model best suited to a given problem is rarely known in advance. This issue is particularly salient among open-source models—a preferred choice for private enterprise deployment given strategic considerations such as data privacy, cost efficiency, and customization flexibility (AIMultiple 2024 ) . Yet this very heterogeneity also presents a promising opportunity to enhance LLM deployments without escalating model parameter scales. Specifically, integrating the complementary advantages of multiple models may unlock collective wisdom and improve problem-solving performance beyond what individual models achieve alone.

The aim of integrating multiple LLMs (hereafter, the wisdom of LLM crowds ) aligns with the established “wisdom of crowds” paradigm, which focuses on aggregating the judgments of large groups to achieve superior collective performance compared to any single member and even expert individuals (Surowiecki 2005 , Wei et al. 2022a ) . However, the nature of the problem differs fundamentally when the crowd members shift from humans to LLMs. First , regarding the outputs, LLMs produce unstructured outputs, such as free-form text, code, and reasoning chains, rather than the commensurable judgments commonly studied in traditional crowd settings. This limits the direct application of conventional statistical aggregation (Atanasov et al. 2017 , Wei et al. 2025 ) , although voting remains possible when outputs can be reduced to standardized final answers. Second , the inference paradigm differs. While traditional wisdom of crowds relies on aggregating independent and single-shot judgments (Becker et al. 2022 , Thomas et al. 2021 ) , LLMs typically operate via autoregressive decoding, transforming problem solving into a dynamic process of inference-time computation (Wei et al. 2022b ) . This sequential generation makes it natural to allocate additional inference-time resources to revise an initial draft; outputs can often be improved through iterative refinement mechanisms, including self-correction and reflective critique (Madaan et al. 2023 , Shinn et al. 2023 ) . Consequently, LLM crowds create an opportunity to harness collective intelligence not only by aggregating completed outputs, but also by deliberately coordinating models during the evolving problem-solving process.

Current approaches for the wisdom of LLM crowds can be broadly divided into two categories: (a) LLM ensemble and (b) multi-agent systems (MASs). LLM ensembles typically operate by either routing queries to the most suitable model based on models’ capabilities (Shnitzer et al. 2023 , Lu et al. 2024 ) or fusing outputs from multiple models (Wang et al. 2022 , Jiang et al. 2023 ) . Conversely, MAS frameworks orchestrate collaboration through the role-playing of agents and the use of either predefined or automatically generated team workflows (Li et al. 2024 , Guo et al. 2024 ) . Despite leveraging multiple, often heterogeneous, models, existing designs in both categories still fall short of achieving deep and synergistic capability complementarity. In ensembles, routing and fusion usually determine the collaboration point either before generation through model selection or after generation through output fusion, giving them limited access to evolving intermediate states. MASs support richer interactions among models, but their roles and workflows are typically predefined or reused across queries within the same task family, rather than tailored to the individual query. Neither line of work adjusts its coordination as an individual solution actually develops, even though reflective mechanisms (Shinn et al. 2023 , Madaan et al. 2023 ) can already expose where a solution falls short.

To transcend these capability boundaries and unlock the latent potential of heterogeneous LLM crowds, coordination must move within the generation process, which requires clarifying how complementarity is realized once problem solving unfolds as a sequence rather than a set of parallel contributions. Complementarity is generally defined as “the quality of being different but useful” (Cambridge Dictionary 2025 ) , yet its realization mechanism differs between static judgment tasks and dynamic sequential reasoning tasks. Traditional wisdom-of-crowds settings commonly combine parallel judgments through static aggregation (Surowiecki 2005 , Yin et al. 2021 , Wei et al. 2025 ) . In contrast, complex LLM problem solving is path-dependent: an impasse reached by a model reflects a specific capability bottleneck in the evolving solution. We conceptualize the use of such intermediate bottlenecks to coordinate models during generation as relay-style complementarity . Rooted in capability asymmetries across heterogeneous models (Hemmer et al. 2025 ) , relay-style complementarity arises when a successor model can specifically address the predecessor’s bottleneck and deliver non-negative marginal improvement (i.e., complementary team performance) (Bansal et al. 2021 ) to the evolving solution. This distinction between static aggregation and relay-style complementarity is illustrated through the example in Figure 1 , which contrasts how each mechanism addresses a hybrid task: analyzing a patient’s clinical notes to calculate a personalized drug dosage using Python code.

Figure 1: Motivating Example: Static Aggregation vs. Relay-Style Complementarity.

Note. The task requires both medical knowledge and coding skills. Left: Static aggregation combines completed responses after generation, which can retain component-specific errors when no individual response resolves all task requirements. Right: WILC achieves relay-style complementarity by first assigning the task to model A and then using the bottleneck identified in its intermediate solution (the syntax error) to guide the transition to model B, which corrects the solution.

To operationalize this relay-style complementarity, we follow the design science paradigm (Hevner et al. 2004 , Abbasi et al. 2024 , Fang et al. 2025 ) and propose WILC (Wisdom Integration of LLM Crowds), a novel collaborative framework that conceptualizes problem solving as a dynamic, relay-style iterative reflection-and-refinement process. Drawing on the insight that capability heterogeneity allows one model to compensate for the capability limitations of another (Kamoi et al. 2024 , He et al. 2025 ) , WILC introduces a strategic model switching mechanism. When an incumbent model falters, the system identifies the specific nature of the issues (hereafter, bottlenecks ) and transfers the evolving problem-solving context to a successor model. The core rationale is to maximize model-bottleneck fitness , selecting a successor not merely because it is better on average, but because its specific capability spectrum aligns with the current bottlenecks. In this way, each model transition is conditioned on the current solution state rather than determined solely by the initial query or a predefined workflow.

To promote the realization of such complementarity at each relay step, WILC employs a complementarity verification mechanism to govern the iteration. It operates along two dimensions: (a) prospective complementarity fit (PCF), which identifies the model with the highest potential to address the current bottleneck by adapting a contextual multi-armed bandit algorithm (Li et al. 2010 ) , and (b) posterior complementarity gain (PCG), which evaluates whether the selected model preserves or improves the evolving solution. By combining prospective selection with posterior verification, WILC constrains unproductive model transitions and sustains collaborative refinement until the crowd’s complementary potential is exhausted or a satisfactory answer is attained.

To validate the effectiveness of WILC, we conduct extensive experiments with diverse LLM crowds across four benchmarks: code generation, mathematical reasoning, general knowledge reasoning, and data visualization. Our results demonstrate that WILC effectively coordinates the complementary capabilities of the LLM crowd and outperforms existing methods. Importantly, WILC mitigates the performance degradation risks that organizations face when making suboptimal model selection decisions amid multiple available models. Ablation studies validate the contributions of key components within the WILC framework, notably, WILC outperforms single-model self-reflection across every benchmark-crowd combination, indicating that its gains cannot be reduced to iterative refinement alone. Furthermore, we analyze the LLM invocation overhead of WILC to evaluate its cost-effectiveness, demonstrating that, under the standardized pricing assumptions used in our analysis, WILC achieves performance comparable to GPT-5.2 at roughly 7 × \times lower estimated per-query cost by coordinating moderate-scale open-source models, while simultaneously facilitating data sovereignty through self-hosted deployment.

The rest of this paper is organized as follows: The following section reviews the relevant literature. Then, we detail the proposed WILC framework in Section 3 and evaluate the proposed framework by describing the experimental design and showing the empirical results in Section 4 . Finally, we conclude the paper by presenting the contributions, implications, limitations, and future research directions in Section 5 .

2 Related Work

We review three streams of work: capability complementarity (Section 2.1 ); approaches that leverage the wisdom of multiple LLMs through ensembles and multi-agent systems (Sections 2.2 and 2.3 ); and the contextual multi-armed bandit problem (Section 2.4 ), which provides the technical foundation of our design.

2.1 Complementarity in Collaboration

Complementarity, generally defined as “quality of being different but useful when combined” (Cambridge Dictionary 2025 ) , serves as a cornerstone for understanding how diverse entities can outperform any single one acting alone (Hemmer et al. 2025 ) . Existing literature has explored complementarity through the lens of human-algorithm collaboration, systematically investigating the conceptualization and methodologies to leverage the complementarity (Bansal et al. 2021 , Fügener et al. 2022 , Donahue et al. 2022 ) .

For the conceptualization, existing literature consistently posits that complementarity is realized when the combined system yields higher performance (or equivalently, lower loss) than either the human or the algorithm acting in isolation (Donahue et al. 2022 ) . It is theoretically impossible if the performances of the human and the algorithm are constant across different contexts, or if one entity consistently dominates the other (Bansal et al. 2021 ) . This implies that for complementarity to exist, the capabilities of the collaborators must be heterogeneous across different contexts, such that neither entity is universally superior. Drawing on the conceptualization, researchers have designed approaches to realize complementary team performance in tasks like image classification and crowdsourcing. Specifically, Steyvers et al. ( 2022 ) proposed a Bayesian modeling approach to combine the classifications and confidence scores from humans and machine classifiers in the image classification task. Extending this Bayesian perspective, Wei et al. ( 2025 ) proposed a human-algorithm collaborative framework for crowdsourcing tasks, which leverages a hybrid complementarity score to dynamically adjust the integration of human labels and algorithmic predictions.

In this study, we draw on the insights from existing studies to design a collaborative framework for LLM crowds from the perspective of complementarity. In the context of LLM crowds, inherent variations in training data distributions, model architectures, and training techniques engender capability heterogeneity across different LLMs, thereby underpinning the existence of complementarities. Building upon prior research, we adapt the conceptualization of complementarity to the proposed relay-style collaboration, thus constructing a novel artifact for achieving complementarity in LLM crowds.

2.2 Large Language Model Ensemble

Large language model (LLM) ensemble, which stems from ensemble learning (Dietterich 2000 , Sagi and Rokach 2018 ) , has emerged as a promising research direction that aims to harness the collective wisdom of LLMs to achieve superior performance. Depending on whether the models used for ensemble are identical, current LLM ensemble methods can be categorized into self-ensemble and heterogeneous ensemble.

Self-ensemble involves generating multiple responses to a single query using the same LLM and aggregating them to derive a final solution. A representative work is Chain-of-Thought with Self-Consistency (CoT-SC) (Wang et al. 2022 ) , which utilizes diverse chain-of-thought prompts to elicit multiple reasoning paths from the model, ultimately selecting the final output via majority voting. Building on CoT-SC, subsequent studies have introduced various enhancements, such as incorporating more complex reasoning chains (Fu et al. 2023 ) , implementing answer verification mechanisms to screen out implausible results (Li et al. 2023 ) , and adopting self-agreement strategies to improve answer reliability (Lin et al. 2024a ) .

Heterogeneous ensemble methods leverage the heterogeneous strengths of different models primarily through two approaches: (1) routing queries to the LLM that is best suited for a particular task based on its capabilities; and (2) fusing outputs from multiple models to synthesize a superior response. It is predicated on the assumption that different LLMs possess heterogeneous capabilities, and therefore it is possible to leverage their complementary strengths and mitigate individual weaknesses (Shnitzer et al. 2023 , Jiang et al. 2023 , Lu et al. 2024 ) . For model routing methods, an external model router is typically trained to assign each query to the most suitable LLM (Shnitzer et al. 2023 , Lu et al. 2024 ) . For instance, Shnitzer et al. ( 2023 ) learn a routing function for each model based on their performance on benchmark datasets by supervised learning. Given a new query, the best model is selected by comparing the output of each routing function, which represents the probability of the model providing the correct answer. For answer fusion methods, studies typically learn a fusion model to synthesize different models’ responses. For instance, the LLM-Blender framework first evaluates the responses of N N LLMs and then employs a fusion model to merge the outputs of the top K K . This fusion model is fine-tuned from a pre-trained LLM that takes the input query and K K responses and produces an improved output as the final response (Jiang et al. 2023 ) .

In summary, self-ensemble improves the reliability of a single LLM but cannot draw on the strengths of other models. Heterogeneous ensembles do combine multiple models, yet they fix the collaboration point either before or after generation: routing remains bounded by the capability of the model it selects, while fusion depends on the fusion model and can be misled when weaker members produce correlated errors. In both cases, the models never interact while the solution is still taking shape.

2.3 Multi-Agent Systems (MASs)

Inspired by human teamwork, multi-agent systems (MASs) have been developed to foster the collaboration between LLM-based agents with diverse expertise (Li et al. 2024 ) . Compared to a single agent, MASs provide enhanced capabilities by two core design elements: (a) role-playing, i.e., assigning diverse roles to agents, and (b) team workflow, i.e., facilitating interactions among agents to harness the power of teamwork (Guo et al. 2024 ) . Based on the designer’s identity, existing works can be divided into two categories: (a) manually designed MAS by human experts and (b) automatically designed MAS by LLMs.

For manually designed MAS, human experts predefine clear role divisions and collaboration workflows based on specific task properties and domain knowledge (Xiong et al. 2023 , Hong et al. 2024 , Qian et al. 2024 ) . Such systems are typically tailored for specific tasks (like software development (Xiong et al. 2023 ) , scientific experiments (Zheng et al. 2023 ) , scientific debates (Tang et al. 2024 , Du et al. 2023 ) , social simulations (Park et al. 2022 ) , etc.) and remain static once established. However, manually designed MASs have clear limitations: (1) they require significant expert effort and domain knowledge to establish, resulting in high design costs; (2) their fixed roles and workflows lack flexibility and are difficult to adapt to changing tasks; (3) scalability is poor, as adding new roles or restructuring requires manual intervention.

To address the limitations of manual design in terms of cost, flexibility, and scalability, recent research has proposed a new paradigm of LLM-designed MAS. Building on the remarkable understanding and planning capabilities of LLMs, these systems can automatically infer and generate adaptive agent roles and collaboration workflows based on input queries (Chen et al. 2023 ) . Although LLM-designed MAS offers adaptability and flexibility, existing approaches still face key issues: (1) they often design architectures based on task types rather than specific queries, resulting in similar structures that fail to optimize performance for individual queries; (2) the automatically generated roles and workflows can be redundant or imprecise, lacking rigorous control and potentially undermining effectiveness in complex situations.

In our research, we propose a framework that synthesizes the strengths of both paradigms while mitigating their respective drawbacks. First, we establish a clear collaboration structure that applies to arbitrary queries, ensuring process rigor and avoiding the redundancy often found in automatically generated workflows. Second, within this structure, collaboration is tailored to each specific query, so that capability complementarity is realized at a fine-grained level rather than at the level of task types. WILC is itself a multi-agent system, but its agents are distinct LLMs with intrinsically different capability profiles rather than one backbone model differentiated by role prompts.

2.4 Contextual Multi-Armed Bandit

The multi-armed bandit (MAB) problem models sequential decision-making under uncertainty (Robbins 1952 , Sutton et al. 1998 ) . At each step, the decision-maker selects one arm and observes a reward, facing the classic exploration–exploitation trade-off: exploit the arm with the best observed return or explore uncertain arms that may perform better. A classic solution is the upper confidence bound (UCB) algorithm (Auer et al. 2002 ) , which scores each arm a a at time t t by U ​ C ​ B a ​ ( t ) = μ ^ a ​ ( t ) + α ​ 2 ​ ln ⁡ t n a ​ ( t ) , UCB{a}(t)=\hat{\mu}{a}(t)+\alpha\sqrt{\frac{2\ln t}{n{a}(t)}}, where μ ^ a ​ ( t ) \hat{\mu}{a}(t) is the empirical mean reward, n a ​ ( t ) n_{a}(t) is the number of times arm a a has been selected, the second term is an exploration bonus, which decreases as the arm is chosen more frequently. Here, α \alpha is a hyperparameter that controls the degree of exploration. MAB assumes a context-free scenario where the decision remains identical regardless of who the decision-maker is. This limitation becomes apparent in real-world applications such as recommendation systems, where the best option depends on user-specific context.

To address this limitation, researchers have proposed a variety of effective algorithms for the contextual multi-armed bandit (CMAB) problem, among which the LinUCB algorithm (Li et al. 2010 ) is one of the most classic. LinUCB extends UCB to handle contextual information by modeling the expected reward of arm a a as a linear function of its d d -dimensional context 𝐱 t , a \mathbf{x}{t,a} with a parameter vector 𝜽 a \boldsymbol{\theta}{a} . Namely, at time t t , the expected reward of arm a a is given by: 𝔼 ​ [ R t , a | 𝐱 t , a ] = 𝐱 t , a ⊤ ​ 𝜽 a . \mathbb{E}[R{t,a}|\mathbf{x}{t,a}]=\mathbf{x}{t,a}^{\top}\boldsymbol{\theta}{a}. Each time an arm is chosen and its reward observed, the parameter vector 𝜽 a \boldsymbol{\theta}_{a} is updated for the next arm-selection.

For each arm a a , LinUCB maintains 𝑨 a = 𝑫 a ⊤ ​ 𝑫 a + 𝑰 \boldsymbol{A}{a}=\boldsymbol{D}{a}^{\top}\boldsymbol{D}{a}+\boldsymbol{I} and 𝒃 a = 𝑫 a ⊤ ​ 𝒄 a \boldsymbol{b}{a}=\boldsymbol{D}{a}^{\top}\boldsymbol{c}{a} , where 𝑫 a ∈ ℝ m × d \boldsymbol{D}{a}\in\mathbb{R}^{m\times d} is the design matrix whose rows are the m m contexts previously observed for arm a a , and 𝒄 a ∈ ℝ m \boldsymbol{c}{a}\in\mathbb{R}^{m} is the corresponding reward vector. Ridge regression then yields the estimated parameter vector 𝜽 ^ a = 𝑨 a − 1 ​ 𝒃 a \boldsymbol{\hat{\theta}}{a}=\boldsymbol{A}{a}^{-1}\boldsymbol{b}_{a} , and each arm is scored by

U ​ C ​ B a ​ ( t ) = 𝐱 t , a ⊤ ​ 𝜽 ^ a + α ​ 𝐱 t , a ⊤ ​ 𝑨 a − 1 ​ 𝐱 t , a , UCB{a}(t)=\mathbf{x}{t,a}^{\top}\boldsymbol{\hat{\theta}}{a}+\alpha\sqrt{\mathbf{x}{t,a}^{\top}\boldsymbol{A}{a}^{-1}\mathbf{x}{t,a}}, (1)

where the second term is a context-dependent exploration bonus and α \alpha controls its weight. A self-contained introduction is provided in Appendix B . After choosing an arm a t a{t} for the new context 𝐱 t , a t \mathbf{x}{t,a{t}} , the reward R t , a t R{t,a{t}} of this choice is received and thus a new observation ( a t , 𝐱 t , a t , R t , a t a{t},\mathbf{x}{t,a{t}},R{t,a{t}} ) is obtained. The algorithm then updates 𝑨 a t \boldsymbol{A}{a{t}} , 𝒃 a t \boldsymbol{b}{a{t}} , and the capability vector 𝜽 ^ a t \boldsymbol{\hat{\theta}}{a{t}} for the next arm-selection by:

𝑨 a t ← 𝑨 a t + 𝐱 t , a t ​ 𝐱 t , a t ⊤ , 𝒃 a t ← 𝒃 a t + R t , a t ​ 𝐱 t , a t , 𝜽 ^ a t ← 𝑨 a t − 1 ​ 𝒃 a t . \boldsymbol{A}{a{t}}\leftarrow\boldsymbol{A}{a{t}}+\mathbf{x}{t,a{t}}\mathbf{x}{t,a{t}}^{\top},\quad\boldsymbol{b}{a{t}}\leftarrow\boldsymbol{b}{a{t}}+R{t,a{t}}\mathbf{x}{t,a{t}},\quad\boldsymbol{\hat{\theta}}{a{t}}\leftarrow\boldsymbol{A}{a{t}}^{-1}\boldsymbol{b}{a{t}}. (2)

In summary, the contextual bandit formulation aligns well with our setting: we select the most suitable model (i.e., arm) for an evolving query that encodes the latest problem-solving state (i.e., context). As an online learning algorithm, it further allows the capability estimates to be updated instantly from feedback obtained during problem solving, which reduces reliance on pre-collected training data.

3 WILC: A Complementarity-Driven Framework

This section presents our proposed framework for the wisdom of LLM crowds, Wisdom Integration of LLM Crowds (WILC) . Given a crowd of K K LLMs, 𝒲 = { w 1 , w 2 , … , w K } \mathcal{W}={w{1},w{2},...,w{K}} , where each w k w{k} represents a unique LLM, our goal is to harness their collective intelligence such that the collaboration among these models achieves superior problem-solving performance compared to any single LLM operating alone. In the following, we begin by outlining the key design principles that facilitate complementarity in Section 3.1 , followed by a detailed introduction of WILC. For convenience, a summary of major notation is provided in Appendix A .

3.1 From Complementarity to Design Principles

To translate the concept of relay-style complementarity into a practical and actionable artifact, we distill two core design principles (DPs) that serve as the guiding tenets for the development of WILC.

DP1: Iterative Reflection-and-Refinement. Since complementarity is realized through a “relay” process, the system must support a multi-round and state-preserving workflow in which models reflect on and refine prior drafts. As illustrated in the right panel of Figure 1 , each handover requires a reflection that identifies the limitations of the current answer, together with a context that carries the initial task, the answer of the current model, and the identified bottleneck forward to the successor model.

DP2: Complementarity-Driven Model Selection. The effectiveness of relay-style collaboration hinges on the successor model, at each relay step, possessing capabilities that complement the bottleneck of the current model. Following prior studies on complementarity, we adapt its realization condition to the sequential relay process. Specifically, the successor model must appear well-suited to the identified bottleneck and, crucially, its contribution must be verified to yield a non-negative improvement to the existing answer. Hence, a complementarity verification mechanism is necessary to ensure the collaborative process continues refining the answer until complementarity is exhausted or a satisfactory result is attained.

Together, these two principles translate relay-style complementarity into an operational design: DP1 establishes the state-preserving collaboration process, while DP2 governs model transitions by requiring successor workers to both fit the identified bottleneck and improve the evolving solution.

3.2 Framework Formulation

Following DP1, our framework WILC conceptualizes the problem-solving process of LLM collaboration as an iterative reflection-and-refinement process. This process is designed to leverage the inherent reflective capability of LLMs, and iteratively refine the answer in a collaborative manner by combining the complementary advantages of different models. Figure 2 provides a visual walkthrough of our framework, illustrating the system roles, role responsibilities, and the iterative reflection-and-refinement process.

Figure 2: Illustration of the Iterative Reflection-and-Refinement Process.

3.2.1 System Roles and Their Responsibilities

Our framework establishes two distinct roles in the problem-solving process: (a) a coordinator and (b) a pool of workers . Serving as executors, the worker pool comprises a heterogeneous set of LLMs responsible for generating solutions, leveraging their specialized capabilities to produce initial answers or refine prior drafts based on specific query contexts. Complementing this, the coordinator, typically designated from the worker pool, acts as the orchestrator tasked with reflective supervision. By critically evaluating current solutions against the initial query, the coordinator diagnoses reasoning bottlenecks and generates diagnostic insights that serve as the “baton” for the relay, i.e., providing explicit directives that guide the selection of the next worker and the focus of subsequent refinement. In practice, coordinator selection proceeds as follows: for an initial small batch of queries from the target task (we use 10 in our experiments), an arbitrary worker serves as the interim coordinator to generate reflections and compute proxy rewards. Based on the burn-in results from this batch, the worker that most frequently achieves the highest proxy reward is then designated as the permanent coordinator for subsequent queries. 1 1 1 This selection relies solely on the proxy reward derived from model-generated reflections (i.e., the count of identified issues), without access to ground-truth labels, and therefore introduces no data leakage from the test set. Section 4.5 empirically validates that WILC’s performance is insensitive to the choice of coordinator (strongest vs. weakest differs by only 0.66 pp), confirming that this arbitrary initialization does not degrade results. This hierarchical division between execution and coordination ensures that the diverse generative strengths of the worker pool are effectively channeled through the coordinator’s diagnostic oversight, thereby establishing a rigorous feedback loop for iterative refinement.

3.2.2 Iterative Reflection-and-Refinement Process

The iterative reflection-and-refinement process is initiated when a user submits an initial query, and proceeds through multiple rounds in which the answer is continually refined via collaborative efforts. As illustrated in the lower panel of Figure 2 , we take three rounds as an example to demonstrate this multi-round process.

In round 1, the query is the initial query, and we select the most suitable worker (i.e., Worker 02 in the figure) to provide an initial answer to the initial query. The coordinator then reflects on this answer, rigorously evaluating its correctness and identifying any potential weaknesses (i.e., the bottleneck in the current round). Based on this reflection, the query for the next round is constructed as the concatenation of the initial query, the current answer, and the reflection. Formally, the queries in different rounds are given by:

q r = { q 0 , r = 1 q r − 1 + a r − 1 + f r − 1 , r ≥ 2 , q{r}=\begin{cases}q{0},&r=1\ q{r-1}+a{r-1}+f_{r-1},&r\geq 2,\end{cases} (3)

where r r denotes the round number; q r q{r} is the query for round r r ; q 0 q{0} is the initial query; q r − 1 q{r-1} , a r − 1 a{r-1} , and f r − 1 f_{r-1} refer to the query, answer, and reflection for round r − 1 r-1 , respectively.

In round 2, the updated query, enriched with the previous answer and reflection, is now used to guide the worker selection, aiming to identify the worker that is most suitable to address the bottleneck identified in the previous reflection. This iterative process is repeated, with each cycle involving worker selection, answer generation, and reflection generation. The worker selection step is guided by the complementarity verification mechanism, which will be detailed in Section 3.3 . In this three-round example, the resulting relay-style collaboration follows the sequence Worker 02 → \rightarrow Worker 04 → \rightarrow Worker 01 , as illustrated by the “Switch Line” in Figure 2 . Through this iterative reflection-and-refinement process, our framework enables the sequential collaboration among heterogeneous workers, whereby each worker is strategically selected to contribute its capabilities to address the bottleneck identified in preceding iterations. This dynamic allocation of workers’ capabilities ensures that the collective wisdom of the LLM crowd is effectively harnessed, leading to increasingly refined answers in a principled and systematic manner.

3.3 Complementarity Verification Mechanism

The effectiveness of the iterative process described in Section 3.2 hinges on worker selection at rounds r ≥ 2 r\geq 2 : the selected worker must be able to address the bottleneck identified in the preceding answer. Following DP2, we therefore design a complementarity verification mechanism, evaluating whether a candidate worker can complement the current problem-solving state and deliver non-negative value in the next round.

3.3.1 Definition: Complementarity of Workers

Building upon the established definitions of complementarity in the literature (Bansal et al. 2021 , Donahue et al. 2022 ) , we define the complementarity of workers in our sequential collaboration scenario. After completing round r − 1 r-1 , let worker A A be the generator of answer a r − 1 a{r-1} , and let worker B B be a candidate for round r r . We regard worker B B as complementary to worker A A under the current state if: (a) worker B B appears well-suited to address the specific bottleneck f r − 1 f{r-1} based on its capabilities, and (b) after worker B B contributes its answer, there is a non-negative improvement to the quality of the answer relative to a r − 1 a_{r-1} . Note that worker A A and worker B B can be the same model in situations where a model possesses diverse abilities to address the bottleneck identified in its previous answer. This generalizes our framework to allow for both inter-model and intra-model complementarity, enabling repeated self-improvement by a single strong worker as well as collaboration among different models.

Aligned with the two facets of this definition, we design a dual-gate complementarity verification mechanism. The prospective complementarity fit (PCF) gate models the fitness between each worker and the query containing the current problem-solving state (i.e., initial query, current answer, and identified bottleneck), and selects the most promising worker (Section 3.3.2 ). The posterior complementarity gain (PCG) gate then checks whether the selected worker degrades the answer relative to the previous round, as assessed by the coordinator rather than against ground truth (Section 3.3.3 ).

3.3.2 Prospective Complementarity Fit (PCF)

Our scenario for worker selection aligns naturally with the contextual multi-armed bandit (CMAB) problem (as introduced in Section 2.4 ). Formally, let 𝒲 = { w 1 , … , w K } \mathcal{W}={w{1},\dots,w{K}} denote the set of workers (arms). We model the problem-solving state at round r r —whether an initial query 2 2 2 In the initial round where no prior answer exists, the initial query itself is conceptualized as the bottleneck state requiring resolution. or a bottleneck identified after reflection—as a context vector 𝐱 q r ∈ ℝ d \mathbf{x}{q{r}}\in\mathbb{R}^{d} , derived from the text embedding of the query. We assume the expected reward (i.e., answer quality) for choosing worker w k w{k} is linear with respect to the context: 𝔼 ​ [ R r , w k | 𝐱 q r ] = 𝐱 q r ⊤ ​ 𝜽 w k \mathbb{E}[R{r,w{k}}|\mathbf{x}{q{r}}]=\mathbf{x}{q{r}}^{\top}\boldsymbol{\theta}{w{k}} , where 𝜽 w k ∈ ℝ d \boldsymbol{\theta}{w{k}}\in\mathbb{R}^{d} is an unknown parameter vector representing the worker’s capability profile. Building on this formulation, we adapt the LinUCB to learn 𝜽 w k \boldsymbol{\theta}{w_{k}} and capture the model-bottleneck fitness (Li et al. 2010 ) . Our adaptation proceeds in two distinct phases:

Cold-Start Phase. The few rounds of worker selection within a single problem-solving process yield too few observations to estimate worker capabilities reliably. We therefore introduce a cold-start phase that establishes a preliminary capability estimate before the framework is deployed. Specifically, we construct an auxiliary dataset 𝒬 \mathcal{Q} covering queries from diverse problem domains. To mirror the real context, the dataset contains two types of queries: (a) initial queries and (b) bottleneck-contextualized queries (i.e., queries containing the current bottleneck to be resolved, analogous to the inputs in rounds r ≥ 2 r\geq 2 ). We deliberately assign a larger proportion to the latter type, as our primary objective is to model each worker’s capability in addressing specific bottlenecks (i.e., model-bottleneck fitness ). Each worker is tasked with responding to the queries in this auxiliary dataset, and their answers are evaluated against the corresponding ground-truth answers. This evaluation produces a reward for each pair of worker and query, thereby providing the data for the initial estimation of every worker’s capabilities.

Following the LinUCB formulation, the learning process of the parameter vector 𝜽 w k \boldsymbol{\theta}{w{k}} of each worker w k w{k} proceeds as follows. Let 𝑫 w k \boldsymbol{D}{w{k}} denote the design matrix with dimension m × d m\times d , where each row corresponds to a context vector previously observed for worker w k w{k} across m m historical trials. Here, m m equals the number of queries in the auxiliary dataset. Let 𝒄 w k ∈ ℝ m \boldsymbol{c}{w{k}}\in\mathbb{R}^{m} denote the corresponding reward vector. Based on LinUCB, the estimated parameter vector 𝜽 ^ w k cold \boldsymbol{\hat{\theta}}{w{k}}^{\text{cold}} is given by:

𝜽 ^ w k cold = ( 𝑫 w k ⊤ ​ 𝑫 w k + 𝑰 ) − 1 ​ 𝑫 w k ⊤ ​ 𝒄 w k = 𝑨 w k − 1 ​ 𝒃 w k , \boldsymbol{\hat{\theta}}{w{k}}^{\text{cold}}=(\boldsymbol{D}{w{k}}^{\top}\boldsymbol{D}{w{k}}+\boldsymbol{I})^{-1}\boldsymbol{D}{w{k}}^{\top}\boldsymbol{c}{w{k}}=\boldsymbol{A}{w{k}}^{-1}\boldsymbol{b}{w{k}}, (4)

where 𝑰 \boldsymbol{I} denotes the d × d d\times d identity matrix, and for simplicity, let 𝑨 w k = 𝑫 w k ⊤ ​ 𝑫 w k + 𝑰 \boldsymbol{A}{w{k}}=\boldsymbol{D}{w{k}}^{\top}\boldsymbol{D}{w{k}}+\boldsymbol{I} and 𝒃 w k = 𝑫 w k ⊤ ​ 𝒄 w k \boldsymbol{b}{w{k}}=\boldsymbol{D}{w{k}}^{\top}\boldsymbol{c}{w{k}} .

Problem-Solving Phase. During the problem-solving phase, the LinUCB algorithm leverages the learned capability parameters to select the worker with the highest fitness for the query at each round. Given the query q r q{r} at round r r , we first transform it into a d d -dimensional context vector 𝐱 q r ∈ ℝ d \mathbf{x}{q_{r}}\in\mathbb{R}^{d} (utilizing an embedding model), and then select the worker with the highest upper confidence bound (UCB) score by the following equation, passing it to the PCG gate for further verification:

w r = arg ⁡ max w k ∈ 𝒲 ⁡ UCB w k ​ ( r ) = arg ⁡ max w k ∈ 𝒲 ⁡ ( 𝐱 q r ⊤ ​ 𝜽 ^ w k + α ​ 𝐱 q r ⊤ ​ 𝑨 w k − 1 ​ 𝐱 q r ) , w{r}=\arg\max{w{k}\in\mathcal{W}}\mathrm{UCB}{w{k}}(r)=\arg\max{w{k}\in\mathcal{W}}\left(\mathbf{x}{q{r}}^{\top}\boldsymbol{\hat{\theta}}{w{k}}+\alpha\sqrt{\mathbf{x}{q{r}}^{\top}\boldsymbol{A}{w{k}}^{-1}\mathbf{x}{q_{r}}}\right), (5)

where α \alpha is a hyperparameter controlling the exploration-exploitation trade-off.

After a worker is finally chosen and its reward observed, LinUCB further refines its capability estimates by incorporating this new observation, continually updating its belief to inform future selections. However, during the multi-round process of solving an initial query, the true reward for each intermediate answer is unavailable, as the ground-truth answer is unknown. To address this challenge, we introduce two key domain adaptations (Abbasi et al. 2024 ) in the implementation of LinUCB during this phase.

First, we introduce a proxy reward mechanism that converts the coordinator’s reflection into a scalar reward signal. During reflection generation, the coordinator is required to enumerate distinct issues in a structured format, and we assign a penalty value of γ \gamma (e.g., 0.1) to each identified issue. Given a reflection with s s issues, the proxy reward R r , w k R{r,w{k}} is computed as:

R r , w k = max ⁡ { 1 − s ​ γ , 0 } R{r,w{k}}=\max{1-s\gamma,0} (6)

which yields a reward in the interval [ 0 , 1 ] [0,1] . The proxy reward thus supplies a reward signal at every round without access to ground truth. When there is no issue identified (i.e., R r , w k = 1 R{r,w{k}}=1 ), we consider the answer to be satisfactory and the iteration process can be terminated.

Second, we introduce a weighted updating mechanism that balances the general capability estimates learned from true rewards in the cold-start phase against the more immediate but potentially biased proxy feedback from the current query. Formally, at round r r , after selecting worker w k w{k} and observing reward R r , w k R{r,w{k}} , we obtain a new observation tuple ( w k , 𝐱 q r , R r , w k ) (w{k},\mathbf{x}{q{r}},R{r,w{k}}) . We adapt the standard LinUCB update in Equation ( 2 ) by incorporating a weighting parameter β ∈ [ 0 , 1 ] \beta\in[0,1] :

𝜽 ^ w k ← β ​ 𝜽 ^ w k cold + ( 1 − β ) ​ 𝑨 w k − 1 ​ 𝒃 w k ​ , \hat{\boldsymbol{\theta}}{w{k}}\leftarrow\beta\boldsymbol{\hat{\theta}}{w{k}}^{\text{cold}}+(1-\beta)\boldsymbol{A}{w{k}}^{-1}\boldsymbol{b}{w{k}}\text{,} (7)

where 𝜽 ^ w k cold \boldsymbol{\hat{\theta}}{w{k}}^{\text{cold}} denotes the capability estimate of worker w k w_{k} learned in the cold-start phase. A larger β \beta places greater weight on the more reliable and broadly learned prior estimate, whereas a smaller β \beta allows the update to respond more strongly to query-specific feedback, which is more responsive to the focal query but also noisier. Thus, β \beta governs a trade-off between exploiting generalizable prior knowledge and adapting to local query-specific signals, thereby mitigating proxy-reward bias and improving the robustness of worker selection. The adaptive selection of β \beta will be discussed in Section 3.4.1 .

Through iteratively selecting promising workers and continually refining capability estimates, PCF ensures a rigorous and adaptive approach to select the most promising worker who fits best with the current query.

3.3.3 Posterior Complementarity Gain (PCG)

The PCG gate serves as an additional safeguard mechanism for validating the worker selected by the PCF gate, checking that its contribution does not degrade the assessed quality of the answer. Its verification procedure is straightforward: PCG first records the proxy reward of the previous answer, then evaluates the proxy reward for the current worker’s submission. If the current worker’s proxy reward is no lower than that of the preceding answer (i.e., R r ≥ R r − 1 R{r}\geq R{r-1} ), the worker is deemed to have provided a non-negative improvement and the iteration continues. In contrast, if the current worker’s proxy reward is strictly lower ( R r < R r − 1 R{r}<R{r-1} ), the worker fails to pass the complementarity verification, and the previous answer is retained. If even the most promising worker is unable to maintain or improve the current proxy reward, we take this as evidence that the remaining complementarity in the worker pool is unlikely to be realized under the current assessment, and the iteration terminates. It is worth noting that this termination mechanism endows WILC with an appealing adaptive property: when the PCG gate halts the collaboration after the burn-in round (i.e., a single round), WILC operates as an evaluate-then-select mechanism—all workers produce responses during the burn-in, and the best-performing worker’s output is directly returned based on proxy evaluation—investing additional rounds only when iterative refinement is expected to yield meaningful gains. This adaptive depth control also brings a practical cost advantage: simpler queries that are resolved in a single round incur substantially fewer API calls, while the full multi-round overhead is reserved for genuinely difficult queries where the additional investment is justified by measurable performance gains. We provide a detailed cost-effectiveness analysis in Section 4.6.2 .

3.4 Designs for Robustness Enhancement

Although the core WILC framework enables principled complementarity-driven collaboration, its performance may still be challenged by query-level distributional shifts and selection uncertainty in practice. To address these issues, we further introduce two robustness-oriented designs: a burn-in initialization strategy and a one-step forward search (OSFS) mechanism.

3.4.1 Burn-In Initialization Strategy

To mitigate potential distributional shifts between the cold-start training data and the novel queries encountered during actual problem solving, we introduce a burn-in initialization strategy at the outset of the process. Operationally, at the first round for a given query, all workers are required to generate an answer to the initial query. The corresponding proxy rewards for each worker’s answer (i.e., R 1 , w k R{1,w{k}} for worker w k w_{k} ) are then calculated, and the adapted LinUCB algorithm for the problem-solving phase is applied to update the capability parameters of all workers using these newly obtained observations. At the same time, for the first round of worker selection, the worker with the highest proxy reward is directly chosen to proceed as the initial executor for the problem-solving process.

The burn-in strategy improves the effectiveness and robustness of WILC through two aspects. First , it enables the framework to quickly adapt its capability estimation to novel queries that may deviate significantly from the cold-start training data, which matters in real-world deployments where incoming queries span a wide variety of types and complexities. This immediate recalibration updates worker parameters to better match the current query, rather than relying only on less relevant prior knowledge. Second , it enables more informed initial worker selection. By collecting actual response and performance on the query at hand before making any exclusionary decisions, WILC reduces the risk of sub-optimal initial selections, thereby providing a stronger starting point that facilitates a smoother refinement process in subsequent rounds.

Furthermore, the proxy-reward signal in the burn-in phase provides a principled way for dynamically setting the weighting parameter β \beta in Equation ( 7 ). The central idea is to compare the ranking of workers’ UCB scores, derived from LinUCB parameters trained in the cold-start phase, with the proxy reward rankings for new queries. If the two rankings are highly consistent, the cold-start estimates remain credible for the current query and we assign a larger β \beta to rely more heavily on the cold-start parameters; conversely, if the rankings diverge, we reduce β \beta and place more weight on the newly observed feedback. Concretely, we compute the Spearman’s rank correlation coefficient (Sedgwick 2014 ) , denoted as ρ \rho , between the UCB score list and the proxy reward list, and set β = ( 1 + ρ ) / 2 \beta=(1+\rho)/2 . Because ρ ∈ [ − 1 , 1 ] \rho\in[-1,1] , this transformation ensures β ∈ [ 0 , 1 ] \beta\in[0,1] .

3.4.2 One-Step Forward Search (OSFS)

To further enhance the robustness of worker selection within the framework, we introduce a one-step forward search design. Specifically, at round r r , after the worker w r w{r} is selected based on PCF and PCG, and their answer a r a{r} , reflection f r f{r} , and proxy reward R r , w r R{r,w{r}} are obtained, we add an additional forward simulation if the current selected worker differs from the previous one at round r − 1 r-1 (i.e., w r ≠ w r − 1 w{r}\neq w{r-1} ). In this simulation, the previous worker w r − 1 w{r-1} is also tasked with generating an answer to the current query q r q{r} , yielding a simulated answer a r ′ a^{\prime}{r} , reflection f r ′ f^{\prime}{r} , and proxy reward R r , w r − 1 ′ R^{\prime}{r,w{r-1}} . If the simulated proxy reward exceeds that of the current worker, i.e., R r , w r − 1 ′ > R r , w r R^{\prime}{r,w{r-1}}>R{r,w_{r}} , we thus use the simulated results to overwrite the current round’s outputs, i.e.,

( a r , f r , R r , w r ) ← ( a r ′ , f r ′ , R r , w r − 1 ′ ) . (a{r},f{r},R{r,w{r}})\leftarrow(a^{\prime}{r},f^{\prime}{r},R^{\prime}{r,w{r-1}}). (8)

This design addresses two distinct sources of risk that arise when the framework switches workers. First , it mitigates selection uncertainty. The worker selection within the PCF gate is based on the LinUCB algorithm, whose capability estimates are themselves updated from proxy rewards rather than true rewards, so a recommended switch may turn out to be suboptimal. The one-step forward search introduces a fallback mechanism that enables the system to revert to the previous worker when empirical evidence suggests superior performance, thereby reducing the risk of acting on a misleading capability estimate. Second , it promotes reflection-response alignment. The reflection f r − 1 f{r-1} that drives round r r is generated specifically for the answer produced by worker w r − 1 w{r-1} , and is subsequently incorporated into the query q r q{r} . There are therefore cases where, despite the LinUCB algorithm recommending a new worker based on updated capability estimates, w r − 1 w{r-1} possesses a more concrete understanding of the bottleneck raised in its own reflection and can address it more effectively. By evaluating both candidates on the current query, the forward search allows the framework to retain whichever contribution the proxy reward actually favors, rather than committing to the switch in advance.

3.5 WILC Framework Overview

Algorithm 1 provides a unified, end-to-end specification of the WILC framework, integrating all components introduced in Sections 3.2 – 3.4 . Notably, the operational ordering in Algorithm 1 differs from the conceptual organization of the preceding subsections, and we clarify this distinction here. Sections 3.3 and 3.4 are organized by design purpose : the complementarity verification mechanism (PCF and PCG gates) and the robustness enhancement designs (burn-in and one-step forward search) are treated as separate, self-contained components, each elaborated in full before the next is introduced. Algorithm 1 , in contrast, presents these components in their execution order . In the problem-solving phase, the burn-in initialization strategy (Section 3.4.1 ) governs round r = 1 r=1 , during which all workers respond to the initial query and capability estimates are updated for all. From round r ≥ 2 r\geq 2 onward, the PCF gate (Section 3.3.2 ), one-step forward search (Section 3.4 ), PCG gate (Section 3.3.3 ), and weighted capability update are applied in sequence within each iteration. This interleaving of components across rounds reflects how WILC operates in practice, and Algorithm 1 serves as the definitive reference for implementing the full framework.

Algorithm 1 Wisdom Integration of LLM Crowds (WILC)

0: Worker pool 𝒲 = { w 1 , … , w K } \mathcal{W}={w{1},\dots,w{K}} ; coordinator; auxiliary dataset 𝒬 \mathcal{Q} ; exploration parameter α \alpha ; issue penalty γ \gamma ; maximum rounds r max r_{\max}

0: Final answer a ∗ a^{*} for each query q 0 q_{0} (i.e., the initial query)

1: — Cold-Start —

2: for each worker w k ∈ 𝒲 w_{k}\in\mathcal{W} do

3: Let w k w_{k} answer all queries in 𝒬 \mathcal{Q} ; evaluate against ground truth to obtain rewards

4: Estimate capability vector: 𝜽 ^ w k cold \boldsymbol{\hat{\theta}}{w{k}}^{\text{cold}} with Eq. ( 4 )

5: end for

6: — Problem-Solving —

7: // Round r = 1 r=1 : Burn-In Initialization

8: Construct query: q 1 ← q 0 q{1}\leftarrow q{0} ; Embed q 1 q{1} to obtain context vector 𝐱 q 1 ∈ ℝ d \mathbf{x}{q_{1}}\in\mathbb{R}^{d}

9: for each worker w k ∈ 𝒲 w_{k}\in\mathcal{W} do

10: Obtain ( a 1 , f 1 , R 1 , w k ) ← (a{1},f{1},R{1,w{k}})\leftarrow worker w k w{k} generates answer a 1 a{1} to q 1 q{1} ; coordinator reflects to obtain f 1 f{1} enumerating s s issues; compute proxy reward R 1 , w k ← max ⁡ { 1 − s ​ γ , 0 } R{1,w{k}}\leftarrow\max{1-s\gamma,;0}

11: end for

12: Compute Spearman’s ρ \rho between UCB scores and { R 1 , w k } {R{1,w{k}}} ; set β ← ( 1 + ρ ) / 2 \beta\leftarrow(1+\rho)/2

13: for each worker w k ∈ 𝒲 w_{k}\in\mathcal{W} do

14: Update: 𝜽 ^ w k \boldsymbol{\hat{\theta}}{w{k}} with Eq. ( 7 )

15: end for

16: Select initial worker: w 1 ← arg ⁡ max w k ∈ 𝒲 ⁡ R 1 , w k w{1}\leftarrow\arg\max{w{k}\in\mathcal{W}}R{1,w{k}} ; record ( a 1 , f 1 , R 1 , w 1 ) (a{1},f{1},R{1,w_{1}})

17: if R 1 , w 1 = 1 R{1,w{1}}=1 then

18: return a ∗ ← a 1 a^{*}\leftarrow a_{1} { Satisfactory at round 1 }

19: end if

20: // Rounds r ≥ 2 r\geq 2 : Iterative Reflection-and-Refinement

21: for r = 2 , … , r max r=2,\dots,r_{\max} do

22: Construct query: q r ← { q r − 1 , a r − 1 , f r − 1 } q{r}\leftarrow{q{r-1},a{r-1},f{r-1}} ; Embed q r q{r} to obtain context vector 𝐱 q r ∈ ℝ d \mathbf{x}{q_{r}}\in\mathbb{R}^{d}

23: Select worker: w r w_{r} with Eq. ( 5 ) { Prospective Complementarity Fit (PCF) }

24: Obtain ( a r , f r , R r , w r ) ← (a{r},f{r},R{r,w{r}})\leftarrow worker w r w{r} generates answer a r a{r} to q r q{r} ; coordinator reflects to obtain f r f{r} enumerating s s issues; compute proxy reward: R r , w r ← max ⁡ { 1 − s ​ γ , 0 } R{r,w{r}}\leftarrow\max{1-s\gamma,;0}

25: // One-step Forward Search

26: if w r ≠ w r − 1 w{r}\neq w{r-1} then

27: Obtain ( a r ′ , f r ′ , R r , w r − 1 ′ ) ← (a^{\prime}{r},f^{\prime}{r},R^{\prime}{r,w{r-1}})\leftarrow worker w r − 1 w{r-1} generates answer a r ′ a^{\prime}{r} to q r q{r} ; coordinator reflects to obtain f r ′ f^{\prime}{r} enumerating s s issues; compute proxy reward: R r , w r − 1 ′ ← max ⁡ { 1 − s ​ γ , 0 } R^{\prime}{r,w{r-1}}\leftarrow\max{1-s\gamma,;0}

28: if R r , w r − 1 ′ > R r , w r R^{\prime}{r,w{r-1}}>R{r,w{r}} then

29: ( a r , f r , R r , w r ) ← ( a r ′ , f r ′ , R r , w r − 1 ′ ) (a{r},f{r},R{r,w{r}})\leftarrow(a^{\prime}{r},f^{\prime}{r},R^{\prime}{r,w{r-1}})

30: w r ← w r − 1 w{r}\leftarrow w{r-1}

31: end if

32: end if

33: if R r , w r < R r − 1 , w r − 1 R{r,w{r}}<R{r-1,w{r-1}} then

34: return a r − 1 a_{r-1} (complementarity exhausted) { Posterior Complementarity Gain (PCG) }

35: end if

36: Update: 𝜽 ^ w r \boldsymbol{\hat{\theta}}{w{r}} with Eq. ( 7 )

37: if R r , w r = 1 R{r,w{r}}=1 then

38: return a ∗ ← a r a^{*}\leftarrow a_{r} { Satisfactory, no issues identified }

39: end if

40: end for

41: return a ∗ ← a r a^{*}\leftarrow a_{r}

4 Empirical Evaluations

In this section, we comprehensively evaluate the proposed WILC framework for different LLM crowds across diverse tasks. First, we describe the experimental setup (including the evaluation tasks, LLM crowds, baseline methods, and other implementation details). Then, we report and discuss the experimental results.

4.1 Experimental Setup

4.1.1 Evaluation Tasks

We evaluate the proposed WILC framework across four representative task domains that capture core forms of enterprise knowledge work: code generation, mathematical reasoning, general knowledge reasoning, and data visualization. For each domain, we select a corresponding typical benchmark dataset: HumanEval for code generation, MATH-500 for mathematical reasoning, MMLU for general knowledge reasoning, and VisEval for data visualization. The diversity of these datasets enables us to evaluate the performance of the proposed framework in a comprehensive manner. The descriptions of these benchmarks are as follows:

• HumanEval (Chen 2021 ) : HumanEval tests the ability of LLMs to generate correct code from natural language prompts. Developed by OpenAI, it consists of 164 hand-written Python programming problems, each paired with a reference answer and test cases. The evaluation metric is Pass@k, which measures the probability that at least one of the top-k generated answers will pass the tests. Following common practice, we report Pass@1, i.e., the percentage of problems where the model generates a correct answer on the first attempt.

• MATH-500 (Hendrycks et al. 2021a ) : MATH-500 evaluates the mathematical reasoning and problem-solving proficiency of LLMs. It contains 500 challenging problems spanning five core mathematical domains (algebra, combinatorics, geometry, number theory, and precalculus), derived from high-level high school competitions such as the American Mathematics Competitions (AMC) and the American Invitational Mathematics Examination (AIME). Rather than testing simple arithmetic, these problems require complex, multi-step reasoning and abstract problem-solving skills. The benchmark is evaluated using exact match on the final numerical answer, demanding precision in both the reasoning process and the final calculation.

• MMLU (Hendrycks et al. 2021b ) : Massive multitask language understanding (MMLU) is a comprehensive benchmark for evaluating models’ multitask understanding across 57 diverse subjects, ranging from elementary mathematics and U.S. history to computer science, and spanning difficulty levels from high school to professional domains. It comprises 15,908 multiple-choice questions and is designed to measure a model’s ability to generalize knowledge without extensive task-specific training. Given the extensive scale of the dataset and the computational costs associated with LLM inference, we randomly selected 50 questions from each subject, resulting in a total of 2,850 questions. The evaluation metric is accuracy , i.e., the percentage of questions answered correctly.

• VisEval (Chen et al. 2024a ) : VisEval is a large-scale benchmark for natural language-to-visualization generation (NL2VIS), containing 1,150 unique visualizations and 2,524 ⟨ \langle NL, VIS ⟩ \rangle pairs drawn from 146 databases. It evaluates LLMs’ ability to generate accurate Python visualization code from natural-language instructions given one or more data files, and assesses each generated visualization along three dimensions: validity , i.e., whether the code successfully renders a visualization; legality , i.e., whether the visualization satisfies the query requirements; and readability , i.e., how effectively it presents the underlying information. The benchmark’s original pass rate aggregates results by visualization; we instead evaluate performance at the level of individual queries and define accuracy as the proportion of queries whose generated result passes both the validity and legality checks.

4.1.2 LLMs

We evaluate WILC on two distinct LLM crowds, using open-source models with parameter scales of approximately 14 billion (14B) and 30 billion (30B), as detailed in Table 1 . Building on the deployment considerations noted in Section 1 , we situate our evaluation in a setting where organizations privately deploy open-source models, motivated by data privacy, cost efficiency, and customization flexibility. This setting instantiates the capability heterogeneity discussed earlier in a concrete form: the models we deploy come from different developers and training pipelines (Shnitzer et al. 2023 , Lu et al. 2024 ) , so their relative strengths vary across task types and no single member of the crowd dominates the others. This provides a natural foundation for the relay-style complementarity that WILC is designed to exploit. Our selection of the 14B and 30B parameter scales is further motivated by practical hardware constraints. Models in this range can typically fit within the memory envelope of widely available commercial GPUs (e.g., 24GB or 48GB VRAM), allowing a crowd of such models to be served via horizontal scaling with multiple standard cards in parallel. In contrast, deploying ultra-large models (e.g., 70B or larger) often necessitates vertical scaling with specialized high-memory hardware or complex multi-GPU tensor parallelism, creating a significantly higher barrier to entry. 3 3 3 In addition to private enterprise deployment, on-device and edge deployment is increasingly viewed as a promising application scenario for LLMs, where resource constraints make medium-sized or smaller models particularly relevant. Our setup therefore simulates a pragmatic enterprise scenario built on accessible, heterogeneous models rather than a single costly hardware configuration.

Table 1: Overview of LLMs Used in Our Experiments

14B Scale 30B Scale

Model Name

Params

Affiliation Model Name

Params

Affiliation

Qwen2.5: 14B 14.8B Alibaba GLM4: 32B 32B Zhipu AI

Phi4: 14B 14.7B Microsoft Qwen2.5: 32B 32.8B Alibaba

DeepSeek-R1: 14B 14.8B DeepSeek Gemma2: 27B 27.2B Google

4.1.3 Baseline Methods

To comprehensively evaluate the effectiveness of WILC, we compare it with several carefully selected baselines that represent different approaches to improving LLM performance.

• Single Execution : As a fundamental baseline, we first evaluate each worker model individually through single execution, which establishes the performance benchmark of standalone models and provides insights into their inherent capabilities. See Appendix C.1 for the single execution prompts for each task.

• ReAct (Yao et al. 2022 ) : ReAct is a framework for task-solving agents that combines reactive reasoning with action execution. It allows the model to reason about the current state of the task and take actions to progress towards the goal. Since ReAct is originally designed for LLM-based agents, we modify it for our setting by introducing a tailored ReAct prompt template, as detailed in Appendix C.2 .

• Reflexion (Shinn et al. 2023 ) : Reflexion is a framework for language-based agents that enables them to improve through “verbal reinforcement”: after an agent produces a result, it generates a linguistic self-reflection on its performance, and uses it as context to guide future attempts. This baseline is particularly relevant to our research as it shares our framework’s emphasis on iterative reflection-and-refinement, while relying on a single model rather than leveraging the collective wisdom of multiple LLMs.

• Self-ensemble : We implement a self-ensemble baseline in the spirit of universal self-consistency (USC) (Chen et al. 2024b ) . Specifically, for each query, we invoke the same model three times to generate multiple candidate responses, and then use an additional synthesis prompt to identify the most consistent answer among them. This implementation is closely aligned with USC when the final answer is selected from the candidate set, while also allowing a lightweight synthesis-based extension in our experimental setting. See Appendix C.3 for the general response-synthesis prompt used across tasks.

• Heterogeneous Ensemble : Heterogeneous ensemble aims to combine the wisdom from heterogeneous models. We design this baseline in the spirit of LLM-Blender (Jiang et al. 2023 ) , which uses a fusion model to aggregate the responses from multiple models. However, LLM-Blender needs to fine-tune a pretrained LLM as the fusion model, which is computationally expensive. For the consideration of computational cost, we adopt a prompt-based fusion strategy to merge the responses into a final answer. Specifically, the prompt for aggregation is the same as that used in the self-ensemble baseline. Also, to ensure a fair comparison with WILC, the fusion is performed by the coordinator LLM used in WILC.

4.1.4 Implementation Details

Unless otherwise noted, all reported results are averaged over five runs. Experiments were conducted on a high-performance GPU cluster primarily composed of NVIDIA A100 GPUs, with inference backend llama.cpp . 4 4 4 https://github.com/ggml-org/llama.cpp All models are quantized to 4-bit precision (i.e., Q4KM scheme) to improve memory efficiency and support practical multi-model deployment (Lin et al. 2024b ) . This setup enables the parallel execution of multiple LLM workers at both the 14B and 30B scales. The temperature is set to 0.8 for all models, following common practice in prior work.

The cold-start phase relies on a set of diverse queries 𝒬 \mathcal{Q} with standard answers, through which each worker w k w_{k} answers these queries and evaluates its performance (i.e., real reward), thereby obtaining an initial understanding of the worker’s capabilities. For the initial queries in 𝒬 \mathcal{Q} , we use an uncontaminated benchmark dataset called LiveBench (White et al. 2024 ) , the data of which is carefully filtered with the aim of avoiding test contamination, namely, queries that appear in the pre-training data typically used by LLMs are excluded. Specifically, the queries in LiveBench are from recently released math competitions, arXiv papers, news articles, and datasets, and include the clean versions of Big-Bench Hard, AMPS, and IFEval benchmarks. It contains 1436 queries from six distinct domains with the following numbers of questions: 200 for reasoning, 150 for data analysis, 128 for coding, 400 for instruction following, 368 for math, and 190 for language. These diverse and comprehensive queries provide a solid foundation for capturing the initial capabilities of each worker in the cold-start phase. Next, we further construct queries with bottlenecks by prompting each worker to self-reflect on these initial queries for two rounds, ultimately generating an additional set of queries with bottlenecks that is twice the number of the initial queries.

For the embedding model that converts textual queries into embeddings, we use a high-performing open model called nomic-embed-text:v1.5 (Nussbaum et al. 2025 ) . The default embedding dimension of the model is 768 and its key advantage over others is its large context window of up to 8,192 tokens, thereby enabling it to handle extended content. After embedding the queries, we use the principal component analysis (PCA) to reduce the dimensionality of the embeddings to 32. Finally, for the hyperparameters of the LinUCB algorithm, we set the exploration parameter α = 0.1 \alpha=0.1 for both the cold-start phase and the problem-solving phase, and dynamically set the weighting parameter β \beta based on Spearman’s rank correlation coefficient between the UCB score list and the proxy reward list.

4.2 Performance: WILC vs. Baselines

In this section, we report the experimental results of WILC and the baselines on the four benchmarks, as shown in Tables 2 and 3 . 5 5 5 Notably, the 14B crowd achieves higher absolute performance than the 30B crowd on HumanEval, MATH-500, and MMLU. This is primarily because the 14B crowd includes DeepSeek-R1: 14B, a reasoning-specialized model distilled from a much larger teacher, which exhibits exceptionally strong performance on these well-established reasoning and knowledge benchmarks. On VisEval, however, the task requires generating complete visualization code from natural language specifications (NL2VIS), a less standard task type that demands broad code generation capabilities (data manipulation, visualization library APIs, layout design) beyond chain-of-thought reasoning. Here, the distillation advantage of DeepSeek-R1: 14B diminishes, and the 30B crowd’s larger model capacity leads to substantially better performance. Our empirical evaluation reveals several key findings that demonstrate the effectiveness of the proposed approach. First , WILC delivers the strongest overall performance across both the 14B and 30B model scales, with a clear advantage over competing methods. Across the eight experimental configurations (four evaluation benchmarks × \times two model scales), WILC achieves the best performance in seven cases, generally by a meaningful margin. For example, in the 14B-scale setting, WILC achieves the highest scores on HumanEval (95.48%), MATH-500 (89.98%), MMLU (88.92%), and VisEval (76.72%). This consistent superiority supports the effectiveness of our complementarity-based collaboration mechanism in harnessing collective intelligence from heterogeneous LLM crowds.

Second , our method consistently surpasses both self-improvement baselines (ReAct and Reflexion) and ensemble approaches (self-ensemble and heterogeneous ensemble), exhibiting greater reliability and stability across diverse experimental configurations. Although a few baselines achieve competitive results in certain experiments, their performance may fall short when evaluated on other models or datasets. For example, while ReAct using GLM4: 32B achieves competitive performance on HumanEval at the 30B scale (92.10%), it falls short on MATH-500 (83.96% vs. 88.56%) and significantly underperforms on VisEval (73.38% vs. 84.06%). Similarly, the heterogeneous ensemble performs comparably to WILC on HumanEval (94.34%), but shows inconsistent results across other benchmarks. In contrast, WILC demonstrates consistently robust and substantial performance improvements across all evaluated benchmarks and model scales. On average, WILC surpasses the strongest baseline by 3.45 percentage points at the 14B scale and by 2.83 percentage points at the 30B scale, highlighting its effectiveness across diverse tasks and models.

Table 2: Performance Comparison on Four Benchmarks: WILC vs. Baselines (14B Scale)

Method HumanEval MATH-500 MMLU VisEval AVG.

\cellcolor macaronPink!25Single Execution (Qwen2.5: 14B) \cellcolor macaronPink!2575.98 *

\cellcolor macaronPink!2572.68 *

\cellcolor macaronPink!2579.08 *

\cellcolor macaronPink!2559.46 *

\cellcolor macaronPink!2571.80 *

\cellcolor macaronPink!25Single Execution (Phi4: 14B) \cellcolor macaronPink!2581.70 *

\cellcolor macaronPink!2576.28 *

\cellcolor macaronPink!2578.72 *

\cellcolor macaronPink!2560.90 *

\cellcolor macaronPink!2574.41 *

\cellcolor macaronPink!25Single Execution (DeepSeek-R1: 14B) \cellcolor macaronPink!2585.58 *

\cellcolor macaronPink!2588.96 **

\cellcolor macaronPink!2581.32 *

\cellcolor macaronPink!2552.66 *

\cellcolor macaronPink!2577.13 *

\cellcolor macaronGreen!25ReAct (Qwen2.5: 14B) \cellcolor macaronGreen!2581.20 *

\cellcolor macaronGreen!2567.10 *

\cellcolor macaronGreen!2581.28 *

\cellcolor macaronGreen!2563.20 *

\cellcolor macaronGreen!2573.20 *

\cellcolor macaronGreen!25ReAct (Phi4: 14B) \cellcolor macaronGreen!2588.16 *

\cellcolor macaronGreen!2576.12 *

\cellcolor macaronGreen!2585.70 *

\cellcolor macaronGreen!2571.74 **

\cellcolor macaronGreen!2580.43 *

\cellcolor macaronGreen!25ReAct (DeepSeek-R1: 14B) \cellcolor macaronGreen!2593.22 **

\cellcolor macaronGreen!2589.32 \cellcolor macaronGreen!2585.00 *

\cellcolor macaronGreen!2552.08 *

\cellcolor macaronGreen!2579.91 *

\cellcolor macaronYellow!25Reflexion (Qwen2.5: 14B) \cellcolor macaronYellow!2574.28 *

\cellcolor macaronYellow!2574.24 *

\cellcolor macaronYellow!2578.34 *

\cellcolor macaronYellow!2558.06 *

\cellcolor macaronYellow!2571.23 *

\cellcolor macaronYellow!25Reflexion (Phi4: 14B) \cellcolor macaronYellow!2581.46 *

\cellcolor macaronYellow!2575.96 *

\cellcolor macaronYellow!2578.40 *

\cellcolor macaronYellow!2559.88 *

\cellcolor macaronYellow!2573.92 *

\cellcolor macaronYellow!25Reflexion (DeepSeek-R1: 14B) \cellcolor macaronYellow!2586.20 *

\cellcolor macaronYellow!2588.96 **

\cellcolor macaronYellow!2581.36 *

\cellcolor macaronYellow!2552.66 *

\cellcolor macaronYellow!2577.30 *

\cellcolor macaronBlue!25Self-ensemble (Qwen2.5: 14B) \cellcolor macaronBlue!2581.20 *

\cellcolor macaronBlue!2575.44 *

\cellcolor macaronBlue!2582.60 *

\cellcolor macaronBlue!2542.98 *

\cellcolor macaronBlue!2570.56 *

\cellcolor macaronBlue!25Self-ensemble (Phi4: 14B) \cellcolor macaronBlue!2584.76 *

\cellcolor macaronBlue!2579.04 *

\cellcolor macaronBlue!2582.34 *

\cellcolor macaronBlue!25 73.74 **

\cellcolor macaronBlue!2579.97 *

\cellcolor macaronBlue!25Self-ensemble (DeepSeek-R1: 14B) \cellcolor macaronBlue!2592.00 *

\cellcolor macaronBlue!25 89.88

\cellcolor macaronBlue!2585.30 *

\cellcolor macaronBlue!2559.32 *

\cellcolor macaronBlue!2581.62 *

\cellcolor macaronPurple!25Heterogeneous Ensemble \cellcolor macaronPurple!25 94.34 **

\cellcolor macaronPurple!2584.00 *

\cellcolor macaronPurple!25 87.72 *

\cellcolor macaronPurple!2571.26 *

\cellcolor macaronPurple!25 84.33 *

\cellcolor macaronOrange WILC

\cellcolor macaronOrange 95.48

\cellcolor macaronOrange 89.98

\cellcolor macaronOrange 88.92

\cellcolor macaronOrange 76.72

\cellcolor macaronOrange 87.78

Note. The coordinator is selected following the procedure described in Section 3.2 ; accordingly, DeepSeek-R1: 14B serves as the coordinator for HumanEval, MATH-500, and MMLU, while Qwen2.5: 14B coordinates VisEval. Significance levels p ∗ < 0.1 {}^{}p<0.1 , p ∗ ∗ < 0.05 {}^{*}p<0.05 , p ∗ ⁣ ∗ ∗ < 0.01 {}^{*}p<0.01 are from paired t t -tests comparing each baseline with WILC across five runs. The best result in each column is shown in bold and the second best is underlined . AVG. is the unweighted average computed from unrounded values.

Table 3: Performance Comparison on Four Benchmarks: WILC vs. Baselines (30B Scale)

Method HumanEval MATH-500 MMLU VisEval AVG.

\cellcolor macaronPink!25Single Execution (Qwen2.5: 32B) \cellcolor macaronPink!2586.70 *

\cellcolor macaronPink!2571.52 *

\cellcolor macaronPink!2584.08 *

\cellcolor macaronPink!2572.14 *

\cellcolor macaronPink!2578.60 *

\cellcolor macaronPink!25Single Execution (GLM4: 32B) \cellcolor macaronPink!2585.00 *

\cellcolor macaronPink!2576.72 *

\cellcolor macaronPink!2584.22 *

\cellcolor macaronPink!2570.58 *

\cellcolor macaronPink!2579.10 *

\cellcolor macaronPink!25Single Execution (Gemma2: 27B) \cellcolor macaronPink!2575.20 *

\cellcolor macaronPink!2550.36 *

\cellcolor macaronPink!2574.50 *

\cellcolor macaronPink!2564.44 *

\cellcolor macaronPink!2566.10 *

\cellcolor macaronGreen!25ReAct (Qwen2.5: 32B) \cellcolor macaronGreen!2588.20 **

\cellcolor macaronGreen!2573.20 *

\cellcolor macaronGreen!2583.90 *

\cellcolor macaronGreen!2573.70 *

\cellcolor macaronGreen!2579.80 *

\cellcolor macaronGreen!25ReAct (GLM4: 32B) \cellcolor macaronGreen!25 92.10

\cellcolor macaronGreen!2583.96 *

\cellcolor macaronGreen!2586.08 *

\cellcolor macaronGreen!2573.38 *

\cellcolor macaronGreen!2583.88 *

\cellcolor macaronGreen!25ReAct (Gemma2: 27B) \cellcolor macaronGreen!2574.04 *

\cellcolor macaronGreen!2552.36 *

\cellcolor macaronGreen!2578.20 *

\cellcolor macaronGreen!2557.04 *

\cellcolor macaronGreen!2565.41 *

\cellcolor macaronYellow!25Reflexion (Qwen2.5: 32B) \cellcolor macaronYellow!2587.20 *

\cellcolor macaronYellow!2572.64 *

\cellcolor macaronYellow!2584.70 *

\cellcolor macaronYellow!2573.60 *

\cellcolor macaronYellow!2579.50 *

\cellcolor macaronYellow!25Reflexion (GLM4: 32B) \cellcolor macaronYellow!2586.20 *

\cellcolor macaronYellow!2580.60 *

\cellcolor macaronYellow!2586.88 **

\cellcolor macaronYellow!2572.60 *

\cellcolor macaronYellow!2582.47 *

\cellcolor macaronYellow!25Reflexion (Gemma2: 27B) \cellcolor macaronYellow!2573.78 *

\cellcolor macaronYellow!2551.00 *

\cellcolor macaronYellow!2572.08 *

\cellcolor macaronYellow!2563.20 *

\cellcolor macaronYellow!2565.02 *

\cellcolor macaronBlue!25Self-ensemble (Qwen2.5: 32B) \cellcolor macaronBlue!2580.86 *

\cellcolor macaronBlue!2567.50 *

\cellcolor macaronBlue!2585.42 *

\cellcolor macaronBlue!2577.34 *

\cellcolor macaronBlue!2577.80 *

\cellcolor macaronBlue!25Self-ensemble (GLM4: 32B) \cellcolor macaronBlue!2584.66 *

\cellcolor macaronBlue!2568.92 *

\cellcolor macaronBlue!2586.60 **

\cellcolor macaronBlue!25 82.28 **

\cellcolor macaronBlue!2580.60 *

\cellcolor macaronBlue!25Self-ensemble (Gemma2: 27B) \cellcolor macaronBlue!2575.60 *

\cellcolor macaronBlue!2566.08 *

\cellcolor macaronBlue!2577.62 *

\cellcolor macaronBlue!2559.20 *

\cellcolor macaronBlue!2569.60 *

\cellcolor macaronPurple!25Heterogeneous Ensemble \cellcolor macaronPurple!2588.86 **

\cellcolor macaronPurple!25 87.20 *

\cellcolor macaronPurple!25 87.10 *

\cellcolor macaronPurple!2577.00 *

\cellcolor macaronPurple!25 85.04 *

\cellcolor macaronOrange WILC

\cellcolor macaronOrange 91.12

\cellcolor macaronOrange 88.56

\cellcolor macaronOrange 87.72

\cellcolor macaronOrange 84.06

\cellcolor macaronOrange 87.87

Note. Notation conventions follow Table 2 . The coordinator is selected following the procedure in Section 3.2 ; accordingly, Qwen2.5: 32B serves as the coordinator for HumanEval and VisEval, while GLM4: 32B coordinates MATH-500 and MMLU.

To more clearly illustrate the performance gains achieved by the wisdom of LLM crowds, we compare the performance of WILC against the median single-execution performance of worker models within an LLM crowd. As shown in Figure 3(b) , this comparison provides an intuitive view of the improvements attributable to the wisdom of LLM crowds. We choose the median 6 6 6 Using the mean as the reference point leads to the same conclusion. as the reference point rather than the mean or maximum because it provides a more robust baseline that is less sensitive to outliers. More importantly, in real-world applications where the best-performing model for a specific query is unknown a priori, the median better reflects the performance of a typical standalone model than the maximum does. The results reveal substantial and consistent performance improvements across all benchmarks and model scales. At the 14B scale, WILC achieves absolute performance gains ranging from +9.8 to +17.3 percentage points. Similarly, at the 30B scale, WILC achieves gains of +3.6 to +17.0 percentage points, with the most substantial improvement again observed on MATH-500.

Figure 3: Performance Gains from Wisdom of Crowds: WILC vs. Single Model Execution

(a) Model Scale: 14B

(b) Model Scale: 30B

Note. The increases depicted in the figure indicate the absolute difference between WILC and the median performance of individual model executions. This visualizes the incremental value added by the wisdom of crowds over individual model execution.

A closer examination of the figures reveals two key patterns. First , the magnitude of improvement correlates with task complexity: reasoning-intensive tasks like MATH-500 exhibit the largest gains, while knowledge-based tasks like MMLU show more modest improvements. This suggests that the wisdom of crowds mechanism is particularly valuable for problems requiring multi-step reasoning, where different models may excel at different reasoning stages. Second , the performance variance among individual models within each benchmark provides insight into the critical value of the LLM crowd in mitigating selection risks while improving the performance ceiling. In practice, organizations face substantial uncertainty when selecting a single model for deployment, as the optimal model for a given task is often unknown a priori. Making a suboptimal choice can lead to significant performance degradation and business losses. Consider the MATH-500 benchmark at the 30B scale as an illustrative example: Gemma2: 27B lags approximately 26 percentage points behind the best individual model GLM4: 32B in accuracy. An organization deploying such a model would suffer considerable opportunity costs due to this performance gap. However, through the wisdom of LLM crowds, WILC not only mitigates this risk by preventing any single weak model from dominating the outcome, but remarkably achieves 88.56% accuracy and surpasses the strongest individual model. This demonstrates that WILC can strategically leverage complementary strengths from all models in the crowd, including extracting valuable capabilities from otherwise weaker models, to deliver superior collective performance that transcends individual model limitations and maximizes organizational value.

We also compare WILC against dedicated query-routing approaches. 7 7 7 A fair comparison with routing methods is inherently difficult, as their performance critically depends on the quality and domain coverage of the labeled training dataset used to learn the routing function (Shnitzer et al. 2023 , Feng et al. 2024 ) . We provide a detailed discussion and comparison in Appendix D . As shown in Appendix D , routing methods perform competitively on benchmarks well-represented in their training data (e.g., HumanEval, MATH-500) but collapse on VisEval, a task type not directly represented in the cold-start data categories. Even restricting WILC to a single round ( r max = 1 r_{\max}{=}1 ) performs better than all routing baselines, because WILC’s burn-in evaluates all workers on the actual target query rather than predicting suitability from potentially mismatched training data. Moreover, the full WILC framework’s multi-round gains persist on VisEval because its successor selection is guided by model-bottleneck fitness—bottleneck patterns that generalize across task types regardless of training data coverage. This robustness property is practically significant, as real-world query distributions rarely conform perfectly to any pre-collected training dataset.

4.3 Ablation Study

To rigorously assess the contributions of key components within the WILC framework, we conduct a comprehensive and systematic ablation study involving five distinct ablation settings.

• Self-reflection: To assess whether multi-model collaboration provides advantages over single-model iterative refinement, this ablation examines the value of cross-model collaboration by comparing WILC with a setting where each worker merely reflects and refines its own solutions. 8 8 8 This ablation is conceptually aligned with Reflexion (Shinn et al. 2023 ) , as both rely on a single model iteratively refining its own output through self-generated feedback. For simplicity, we use worker 1–3 to represent the three workers in each crowd.

• Random Worker Selection: To assess whether our capability-aware worker selection is more effective than random assignment, this ablation replaces the PCF gate with a random draw from the worker pool while retaining all other components.

• Without Burn-In Strategy: To assess the necessity of the burn-in initialization strategy for the framework, this ablation evaluates the strategy’s importance by comparing WILC with the setting that removes the burn-in strategy, which instead uses LinUCB to select the worker at the first round.

• Without Multi-Round Collaboration: To assess the value of iterative multi-round refinement, this ablation restricts WILC to a single round (i.e., only the burn-in round), effectively reducing it to a one-shot model selection mechanism that selects the best worker (evaluated by proxy performance during the burn-in round) but forgoes any subsequent relay-style collaboration.

• Without OSFS: To assess the necessity of the one-step forward search (OSFS) for the proposed framework, this ablation study evaluates the setting in which the search is removed.

Figure 4: Ablation Study Results on Four Benchmarks

Note. Results averaged over five runs with standard errors reported. Significance levels p ∗ < 0.1 {}^{}p<0.1 , p ∗ ∗ < 0.05 {}^{*}p<0.05 , p ∗ ⁣ ∗ ∗ < 0.01 {}^{*}p<0.01 (ns: not significant) are from paired t t -tests against WILC. Pairing removes the run-to-run variation shared by both settings, so the markers reflect how consistently WILC leads, not by how much: on MATH-500 (14B scale), the two settings vary together across runs (run-to-run SD ≈ \approx 1.5 points), yet WILC leads in all five runs by a stable margin, giving p < 0.01 p<0.01 for an average gap of only 0.44 pp.

Figure 4 presents the results of the ablation studies across four benchmarks, revealing several key insights into the contributions of different components within WILC. First , WILC performs at least as well as every ablation setting in general, either significantly better or statistically comparable (the only settings without a significant advantage are those where WILC has already saturated, such as HumanEval at the 14B scale, where a single round already reaches 95.5%), demonstrating the synergistic value of integrating all proposed components. Second , the self-reflection ablation highlights that WILC, by coordinating complementary capabilities across the crowd, consistently outperforms any individual model relying solely on self-refinement. This advantage holds without exception: WILC leads in all 24 comparisons (3 workers × \times 4 benchmarks × \times 2 scales) by margins of + 0.84 +0.84 to + 37.56 +37.56 percentage points, indicating that its gains cannot be attributed to iterative refinement alone. Moreover, WILC demonstrates robustness even when a weak model exists in the LLM crowd by skillfully integrating the diverse strengths of different models while mitigating individual shortcomings. Third , comparing WILC against the other four ablation settings (i.e., random selection, w/o burn-in, w/o multi-round, and w/o OSFS) reveals clear differences in component contributions. Because random selection retains every other component and replaces only the PCF gate with a random draw, its comparison with WILC isolates the contribution of capability-aware successor selection. This comparison yields a significant improvement in all eight benchmark–scale settings ( + 3.28 +3.28 to + 19.80 +19.80 percentage points). Notably, access to a heterogeneous crowd alone is not sufficient: in six of the eight settings, random selection performs below the strongest single model under self-reflection, as an uninformed handover may pass the task to a worker ill-suited to the current bottleneck and degrade an already adequate solution. Complementarity is therefore latent in a heterogeneous crowd; realizing it depends on directing handovers toward workers whose estimated capabilities fit the diagnosed bottleneck. In our design, PCF-based selection supplies this direction, while PCG guards against proxy-assessed degradation after the handover. The w/o multi-round ablation shows that the value of additional rounds is task-dependent: it contributes 3.8 to 6.4 percentage points on the open-ended generation tasks (VisEval at both scales and HumanEval 30B), whereas on MMLU and MATH-500 the burn-in round already resolves most queries and further rounds add little. Meanwhile, the ablation settings w/o burn-in and w/o OSFS demonstrate moderate performance degradation compared to WILC, confirming that these robustness-oriented design elements contribute to enhancing the framework’s stability and reliability across diverse task scenarios.

4.4 How Capability Complementarity Emerges and Operates?

To better understand how capability complementarity contributes to WILC’s performance, we examine it from two perspectives. We first analyze the heterogeneity of worker capabilities as the basis for complementarity, and then investigate the dynamic collaboration patterns through which such complementarity is realized in the multi-round problem-solving process.

4.4.1 Capability Heterogeneity Across Workers

The effectiveness of WILC depends on heterogeneous capabilities across workers, such that one worker may be better able to resolve a problem-solving bottleneck on which another worker may underperform. To analyze this heterogeneity, we visualize rank-based capability distribution of each worker over the cold-start query set 𝒬 \mathcal{Q} . Specifically, we first encode each query in 𝒬 \mathcal{Q} into its embedding representation (as introduced in Section 3.3.2 ) and then apply K-Means clustering to partition the query set into 30 clusters. 9 9 9 The value of 30 30 is chosen for illustrative purposes; using other reasonable numbers of clusters should lead to similar conclusions. Queries within the same cluster tend to share similar semantic and bottleneck structures, whereas queries across clusters are more distinguishable. Next, for each worker listed in Table 1 , we compute its average accuracy within each cluster and convert these values into within-scale rankings. That is, for each cluster, workers are ranked relative to other workers in the same model scale, with Rank 1 indicating the strongest performer in that cluster. Figure 5 visualizes the resulting ranking matrix, with cluster index on the x-axis and worker on the y-axis. Compared with absolute accuracy values, this rank-based representation more directly highlights the comparative advantage structure that is most relevant to worker selection in WILC.

Figure 5: Worker Heterogeneity Across Query Clusters

Figure 5 reveals substantial cross-worker variation in relative rankings across clusters, indicating pronounced heterogeneity in capability profiles and suggesting meaningful opportunities for complementary collaboration. At the 14B scale, DeepSeek-R1:14B ranks first in most clusters, yet Qwen2.5:14B and Phi4:14B still attain the top rank in moderate number of clusters. At the 30B scale, the top two ranks are primarily occupied by GLM4:32B and Qwen2.5:32B across different clusters. Meanwhile, in our tested 30B pool, the weaker performance of Gemma2:27B suggests that the presence of a relatively weak worker does not prevent WILC from improving overall performance. Taken together, these patterns suggest that capability complementarity in LLM crowds is not characterized by uniform superiority of a single worker, but rather by shifting relative advantages across different regions of the query space. Such heterogeneous ranking patterns provide the foundation for WILC to translate capability heterogeneity into improved collective performance through adaptive worker selection.

4.4.2 Collaboration Dynamics in Multi-Round Problem Solving

We analyze collaboration dynamics by examining how workers alternate across rounds. For an instance that terminates after r r rounds, let the worker selection sequence be w 1 → w 2 → ⋯ → w r w{1}\rightarrow w{2}\rightarrow\cdots\rightarrow w{r} , where w i w{i} denotes the worker selected at round i i . We define a switch as a change in worker between two consecutive rounds, i.e., w i ≠ w i + 1 w{i}\neq w{i+1} . We then quantify such dynamics using the Switch Rate :

Switch Rate = # ​ switches r − 1 , \textit{Switch Rate}=\frac{#\ \textit{switches}}{r-1}, (9)

where r r is the total number of rounds at termination, and # ​ switches #\ \textit{switches} refers to the number of switches. For instance, if a dialogue terminates at round 4 4 with the sequence “Qwen2.5: 32B → \rightarrow GLM4: 32B → \rightarrow Qwen2.5: 32B → \rightarrow Qwen2.5: 32B”, then two switches occur across three possible transitions, yielding a switch rate of 2 / 3 2/3 (i.e., 0.67). A higher switch rate typically indicates more frequent activation of different workers to facilitate complementary collaboration, whereas a lower switch rate suggests preference and reliance on a particular worker.

Using the 30B-scale crowd as an example, Figure 6 shows the distribution of switch rates on four benchmarks. The plots show that WILC does not consistently adhere to a single worker throughout the entire multi-round reasoning process. Instead, across all benchmarks, we observe substantial switching, with many dialogues exhibiting switch rates concentrated between 0.2 and 0.8. This pattern suggests that worker alternation is common and that collaboration manifests in a dynamically diverse manner. Importantly, this phenomenon is consistent with WILC’s worker selection mechanism, which can perceive and capture the capability demands induced by different self-reflection rounds and proactively elicit contributions from workers whose strengths align with the evolving needs of the task. Such dynamic switching more fully exploits workers’ differentiated advantages across rounds, thereby improving the overall performance of the LLM crowds.

Figure 6: Switch Rate Distribution on Four Benchmarks

To understand how the framework orchestrates multi-model collaboration, we examine the collaboration topologies that emerge within the first three rounds of multi-round problem-solving processes. As shown in Fig. 7(b) , we identify five prototypical topologies grouped into two categories and show the distribution of these topologies across four benchmarks at two model scales. Several findings stand out. First, strategic handover accounts for a substantial share of multi-round processes across all settings, ranging from 30.0% on MATH-14B to 57.3% on VisEval-14B, confirming that WILC actively leverages the heterogeneity of the worker pool. Second, the topology profile varies meaningfully with task nature: VisEval and HumanEval, which involve open-ended generation, exhibit the highest handover rates (52.6–57.3%), whereas MATH, which demands sustained chain-of-thought reasoning, favors solo resolution (63.3–70.0%). This suggests that the topology profile is not arbitrary but rather shaped by the underlying task characteristics, indicating that WILC implicitly adapts its collaboration strategy to the nature of the problem at hand. Third, within strategic handover, quick switch is consistently the dominant subtype, indicating that the framework efficiently recognizes first-round failures and promptly redirects to a more suitable model.

Figure 7: Illustrative Multi-Round Collaboration Topologies within Three Rounds

(a) Topology Taxonomy

(b) Topology Breakdown on Four Benchmarks

4.5 Robustness Checks

4.5.1 Selection of Coordinator

As described in Section 3.2 , we designate the coordinator as the worker that most frequently achieves the highest proxy reward during the burn-in phase across an initial batch of target-task queries. A natural follow-up question is whether WILC’s performance is sensitive to this choice. To test this sensitivity, we compare WILC’s performance at the 14B scale when the coordinator is set to the strongest versus the weakest model in the worker pool. Table 4 reports the results averaged over five runs. Despite the substantial gap in the coordinators’ standalone capabilities, the performance differences between the two configurations are small and statistically insignificant across all four benchmarks ( p > 0.05 p>0.05 ). On MMLU, the weakest coordinator even yields a marginally higher average score than the strongest. Overall, the mean performance gap is only 0.66 percentage points (87.78% vs. 87.12%).

This robustness stems from two aspects. First , the coordinator’s role in WILC is diagnostic rather than generative —it identifies bottlenecks in the current solution (e.g., logical gaps, incorrect computations, or missing steps) rather than producing the solution itself. This task demands reflective capability, which is a qualitatively different skill from problem solving. Even a model that is relatively weaker at generating correct answers can still adequately recognize and articulate the issues present in another model’s output, because critiquing an existing solution is cognitively less demanding than producing a correct one from scratch. Second , the primary driver of WILC’s collaborative performance, namely the adaptive LinUCB-based worker selection mechanism, operates independently of the coordinator’s identity. The PCF gate selects workers based on learned capability–bottleneck fitness, and the PCG gate validates improvement via proxy rewards; both mechanisms depend on the content of the reflection (i.e., the number and nature of identified issues) rather than on which model produced it. Consequently, as long as the coordinator can surface salient issues with reasonable fidelity, the downstream selection and verification pipeline functions effectively regardless of the coordinator’s standalone problem-solving capability.

This insensitivity also implies that extending WILC to a dynamic coordinator selection, where the coordinator is re-designated per query based on real-time burn-in feedback, is unlikely to yield significant additional gains, since even a suboptimal coordinator produces sufficiently reliable reflections. Nonetheless, the burn-in mechanism already provides the infrastructure for such an extension if needed in future applications.

Table 4: Coordinator Robustness Analysis (14B Scale)

Coordinator HumanEval MATH-500 MMLU VisEval AVG.

Strongest 95.48 ± \pm 1.29 89.98 ± \pm 1.45 88.92 ± \pm 1.26 76.72 ± \pm 3.09 87.78

Weakest 92.92 ± \pm 2.22 89.24 ± \pm 0.86 89.70 ± \pm 1.65 76.62 ± \pm 2.88 87.12

Δ \Delta

• • 2.56
• • 0.74 − - 0.78
• • 0.10
• • 0.64

  Note. Performance is reported as mean ± \pm standard deviation over five random seeds. “Strongest” and “Weakest” refer to the model with the highest and lowest single-execution accuracy on each benchmark, respectively. Δ \Delta = Strongest − - Weakest.

  4.5.2 Exploration Parameter

  We further examine the role of the exploration parameter α \alpha in the LinUCB algorithm, which governs the exploration–exploitation trade-off. A larger α \alpha encourages the system to explore less-tried workers, while α = 0 \alpha=0 reduces LinUCB to pure exploitation with no exploration at all. Table 5 reports the performance at the 14B scale for α ∈ { 0 , 0.01 , 0.05 , 0.1 , 0.5 } \alpha\in{0,0.01,0.05,0.1,0.5} .

  Two observations emerge. First, the α = 0 \alpha=0 row serves as an ablation that removes exploration entirely. Compared with the non-zero α \alpha settings, pure exploitation achieves a comparable but slightly lower overall average: 86.86% versus 86.97%–87.78% when α > 0 \alpha>0 .This confirms that exploration, even in modest amounts, is beneficial, as it allows the system to discover complementary workers that pure exploitation would overlook. Second, across all non-zero α \alpha values spanning a 50-fold range (0.01–0.5), performance remains highly stable, with the average varying by only 0.81 percentage points (86.97%–87.78%). This demonstrates that WILC is robust to the specific choice of α \alpha and requires no elaborate tuning in practice.

  Table 5: Ablation and Sensitivity on the Exploration Parameter α \alpha (14B Scale, Single Run)

  α \alpha HumanEval MATH-500 MMLU VisEval AVG.

  0 (no exploration) 95.85 ± \pm 1.46 91.42 ± \pm 1.44 88.71 ± \pm 1.09 71.47 ± \pm 3.46 86.86

  0.01 95.85 ± \pm 1.46 90.52 ± \pm 0.27 88.29 ± \pm 0.29 73.20 ± \pm 3.23 86.97

  0.05 95.85 ± \pm 1.46 90.56 ± \pm 0.33 88.28 ± \pm 0.27 73.40 ± \pm 3.59 87.02

  0.1 (default) 95.48 ± \pm 1.29 89.98 ± \pm 1.45 88.92 ± \pm 1.26 76.72 ± \pm 3.09 87.78

  0.5 95.85 ± \pm 1.46 90.60 ± \pm 0.32 88.28 ± \pm 0.27 73.47 ± \pm 3.86 87.05

  4.6 Additional Analyses

  4.6.1 Value of Learning Model-Bottleneck Fitness

  As articulated in DP2 (see Section 3.1 ), the effectiveness of relay-style collaboration depends on selecting a successor worker whose capabilities specifically complement the bottleneck identified in the current problem-solving state. Accordingly, WILC relies on learning model-bottleneck fitness , rather than generic query routing based solely on the initial query or broad task category. Specifically, this section examines whether incorporating bottleneck-contextualized queries (i.e., queries containing the current bottleneck to be resolved) into the cold-start phase improves the learning of model-bottleneck fitness.

  As described in Section 3.3.2 , the cold-start dataset comprises two types of queries: (a) initial queries and (b) bottleneck-contextualized queries. We posit that the capability to resolve bottlenecks (e.g., rectifying logical inconsistencies or debugging code errors) is largely a scenario-agnostic trait inherent to a model, rather than being strictly tied to a particular problem domain. In other words, a specific type of bottleneck, such as a deficiency in long-chain reasoning, may exhibit shared structural patterns across diverse scenarios. Consequently, incorporating bottleneck-contextualized queries in the cold-start stage allows the LinUCB model to learn more generalizable model-bottleneck fitness patterns that go beyond surface-level domain features, thereby enabling the system to better identify the worker most capable of addressing a bottleneck even in new and previously unseen scenarios.

  Figure 8: Comparison of Performance with and without Bottleneck-Contextualized Queries

  (a) Model Scale: 14B

  (b) Model Scale: 30B

  To empirically evaluate this idea, we compare two cold-start configurations for training the LinUCB model to learn model-bottleneck fitness: one using only type (a) queries, and the other using both type (a) and type (b) queries. Figure 8(b) reports the performance of WILC under these two configurations. The results show that incorporating bottleneck-contextualized queries into the cold-start phase yields modest but generally consistent improvements in final performance across both model scales. Taken together, these findings provide empirical support for the value of learning model-bottleneck fitness. Training only on initial queries yields a basic understanding of each model’s general strengths, but this is insufficient for a system designed for iterative refinement. In relay-style collaboration, the system repeatedly encounters intermediate answers with specific bottlenecks that require targeted correction. By incorporating bottleneck-contextualized queries into the cold-start phase, the LinUCB-based selection mechanism develops a more nuanced and generalizable understanding of worker capabilities. It learns not merely which model performs better on a broad task type, but more importantly which model is best suited to resolve the bottleneck identified in a preceding worker’s failure. This shift from coarse query routing to selection guided by learned model-bottleneck fitness appears to be an important enabler of the deep capability complementarity achieved by WILC.

  4.6.2 Cost-Effectiveness Analysis

  A natural concern is whether the additional overhead from multi-round, multi-model collaboration is justified by the performance gains it delivers. To address this, we estimate WILC’s per-query cost based on its API call structure, empirical round counts, and per-token pricing, then benchmark the cost-performance tradeoff against frontier closed-source models.

  Taking the case of K = 3 K=3 workers as an example, Table 6 breaks down the API calls per round. Although the theoretical maximum over r max = 6 r_{\max}=6 rounds is 21 calls, the PCG gate adapts collaboration depth to problem difficulty: easy queries are resolved after the burn-in round alone, effectively reducing WILC to a one-shot model selection, while difficult queries proceed through additional refinement rounds. The empirical averages are 1.78 rounds (14B) and 1.46 rounds (30B) across our benchmarks, corresponding to roughly 8.3 and 7.4 API calls per query, respectively (see Appendix E for details). This adaptive depth control means that for simpler queries where the PCG gate terminates after the burn-in round, WILC effectively operates as a model selection mechanism. The additional API calls are invested only in genuinely difficult queries where iterative refinement yields meaningful gains, making the multi-round overhead both targeted and well-justified.

  Table 6: API Call Breakdown for WILC Framework ( K = 3 K{=}3 Workers, r max = 6 r_{\max}{=}6 Rounds)

  Component Total

  Round Burn-in OSFS † Relay Round Cumulative

  1 6 0 0 6 6

  2 0 1 2 3 9

  3 0 1 2 3 12

  4 0 1 2 3 15

  5 0 1 2 3 18

  6 0 1 2 3 21

  Note. Each component count includes both a worker call and a coordinator call. † OSFS is triggered only when the selected worker differs from the previous round. Assuming a 50% switching probability, the expected number of OSFS calls per round is 1 (0 when no switch occurs, or 2 when a switch occurs).

  To quantify cost effectiveness, we evaluate GPT-4o, GPT-5.2, and GPT-5.4 on the same four benchmarks used in our main experiments and compare their average performance and estimated per-query cost against WILC (see Table E3 in the Appendix for the full comparison). Using public API pricing as a common yardstick, a single GPT-5.2 query costs $0.0088, whereas WILC at the 30B scale costs only ∼ \sim$0.0013 per query, about 7 × \times cheaper, while achieving a comparable average accuracy (87.86% vs. 88.09%). At the 14B scale the gap is similarly striking: WILC-14B achieves 87.78%, nearly matching GPT-5.2, at roughly 15% of the per-query cost. In practice, the cost advantage can be even more pronounced: our experiments use locally served models via llama.cpp with 4-bit quantization, where inference cost is governed by fixed hardware expenditure rather than per-token charges. Under self-hosted deployment, the per-token API charge is eliminated entirely, and the amortized per-query cost decreases with throughput, though total cost of ownership depends on hardware utilization and infrastructure overhead. Meanwhile, self-hosted deployment ensures that no data leaves the organization, satisfying the privacy and compliance requirements that motivate private LLM deployments in the first place.

  5 Conclusion

  This research addresses a fundamental challenge in leveraging the wisdom of LLM crowds: how to enable heterogeneous LLMs to collaboratively solve complex problems through strategic model coordination that achieves deep capability complementarity. Following the computational design science paradigm (Hevner et al. 2004 , Abbasi et al. 2024 , Fang et al. 2025 ) , we designed and evaluated WILC, a novel IT artifact that models multi-LLM collaboration as a dynamic relay-style sequential process driven by complementarity among heterogeneous models. Our evaluation focuses on an enterprise setting in which organizations deploy multiple medium-sized open-source models from different developers under constraints related to computational budgets, data sovereignty, and customization flexibility, while the framework itself is not restricted to a particular parameter scale.

  This research contributes to the IS knowledge base in three ways. First , we extend the wisdom of crowds paradigm from static aggregation of parallel, independent judgments (Surowiecki 2005 ) to dynamic, relay-style sequential complementarity. Classic wisdom-of-crowds theory posits that aggregating diverse opinions yields superior collective judgment; our work reconceptualizes collective AI intelligence as an iterative coordination process where each participant builds upon predecessors’ identified bottleneck. This shift from “aggregate diverse opinions” to “orchestrate complementary capabilities through state-dependent coordination” provides a new theoretical lens for IS scholars studying collective intelligence in AI systems. Second , to operationalize this relay-style complementarity, we contribute prescriptive design knowledge (Abbasi et al. 2024 ) in the form of two transferable design principles for multi-AI coordination. DP1 (Iterative Reflection-and-Refinement) ensures the relay is feasible: effective multi-model collaboration requires a state-preserving workflow that enables successive models to reflect on and refine prior outputs. DP2 (Complementarity-Driven Model Selection) ensures the relay is productive: model transitions must be governed by verified complementarity rather than static assignment or random selection. These principles are not confined to the specific WILC instantiation but generalize to any multi-AI collaboration scenario requiring heterogeneous capabilities—such as collaborative content creation, multi-agent diagnosis, or iterative software development—offering actionable design guidance for practitioners building adaptive AI coordination systems. Third , we make a methodological contribution by introducing model-bottleneck fitness as the basis for an adaptive selection mechanism. Instantiated through online learning (LinUCB), this mechanism updates capability beliefs from real-time feedback, learning which model resolves which type of bottleneck through accumulated experience. This provides a methodological template for building AI coordination systems that can self-improve as task distributions evolve.

  Our findings yield several practical implications for enterprise deployment of LLM collaboration systems. First, the cost-performance tradeoff is highly favorable: WILC achieves performance comparable to GPT-5.2 at roughly 7 × \times lower per-query cost, and the multi-round overhead is targeted rather than uniform—simple queries terminate after a single burn-in round while additional rounds are invested only in genuinely difficult queries. Second, because all models are self-hosted, data remains within the organization, supporting the privacy and compliance requirements that motivate private LLM deployments. Third, unlike dedicated routing methods whose effectiveness is bounded by training data coverage (Appendix D ), WILC’s evaluate-then-select mechanism and bottleneck-driven successor selection remain effective on task types not covered by the cold-start data, an important property given that real-world query distributions are rarely fully predictable. More broadly, our results suggest that capability heterogeneity across an organization’s deployed models is not a liability but an asset: by strategically coordinating complementary strengths through the design principles articulated in this study, organizations can unlock collective intelligence that surpasses any individual model. The underlying principles—iterative complementarity verification and adaptive capability learning—may inform the design of multi-AI collaboration systems beyond the LLM domain, such as customer service pipelines, content moderation workflows, and collaborative analytics platforms.

  This study has several limitations that point to future research opportunities. First, our evaluation covers four benchmarks with crowds of three workers; validating the framework on larger worker pools and more diverse task domains would strengthen generalizability claims. Relatedly, our empirical evidence is drawn from open-source crowds at the 14B and 30B scales. Because relay-style complementarity rests on capability heterogeneity rather than on a particular parameter scale, the mechanism is in principle applicable to crowds composed of larger or frontier models; whether the gains persist in that regime remains to be tested. Second, the current proxy reward mechanism is general-purpose and effective across our benchmarks, but designing task-adaptive reward signals that better align with domain-specific evaluation criteria could further sharpen model selection and progress assessment. Third, extending the framework to handle highly complex, structured tasks that require multi-step decomposition or cyclic problem-solving processes presents an exciting opportunity. Incorporating automatic task decomposition and more advanced collaboration strategies along such problem-solving chains could broaden WILC’s applicability to a wider range of enterprise scenarios.