In recent years, LLMs have exhibited increasingly remarkable performance across a wide range of mathematical, scientific, and programming benchmarks wei2022chain ; zhou2022least ; yao2023tree ; besta2024graph . This progress is primarily attributed to the emergence of reasoning techniques, which have become pivotal methods for enhancing the inferential capabilities of LLMs. Chain-of-Thought (CoT) addresses complex problems by guiding the model to generate a sequence of intermediate reasoning steps wei2022chain . Least-to-Most Prompting (LtM) decomposes a task into a series of subproblems solved in order, where the solution to each subproblem supports subsequent ones zhou2022least . Tree-of-Thought (ToT) employs a tree structure that enables the model to explore multiple reasoning paths in parallel yao2023tree . Skeleton-of-Thought (SoT) improves generation efficiency by first producing an outline of the output, then filling in details in parallel ning2023skeleton . Graph-of-Thought (GoT) offers a more dynamic reasoning paradigm by modeling the reasoning process as a graph of interconnected thought nodes besta2024graph .

Recent studies have demonstrated that the collaborativeness of LLMs can effectively integrate their respective strengths, thereby enhancing the ability to solve complex problems li2025rethinking ; kim2024mdagents ; wang2025mixtureofagents . Existing frameworks can be broadly classified into two categories. The first framework is commonly referred to as role-playing. In this paradigm, multiple LLMs are assigned distinct roles or responsibilities, with each model focusing on tasks specific to its designated function kim2024mdagents ; wang2024survey . Through collaborative interactions, LLMs collaborate together to achieve complex overarching objectives. With the clear division of labor, this approach enables the effective decomposition of complex problems and leverages the specialized competencies of each LLM to generate integrated and comprehensive solutions. The second framework is referred to as multi-LLM debate du2023improving . In this paradigm, each LLM first attempts to solve the problem independently and then analyzes outputs of other LLMs to reach a consensus. Within this framework, existing works can be further delineated according to the composition and interaction strategies of the participating LLMs.

From the perspective of LLM composition, existing works can be classified into two main categories: debates involving multiple instances of a single LLM li2025rethinking and debates among heterogeneous LLMs wang2025mixtureofagents ; wang2024rethinking . In terms of deliberation mechanisms, representative strategies include majority voting schemes wang2022self , interdisciplinary collaborativeness paradigms tang2023medagents , structured group discussions chen2023reconcile , and negotiation-based protocols fu2023improving . Each of these approaches offers distinct advantages in facilitating consensus formation and improving the robustness of the solution.

As aforementioned, the collaborativeness of LLMs exhibits component-wise uncertainty where its cluster components lack explicit selection rules, making significant barriers to practical healthcare applications. Additionally, existing architectures wang2025mixtureofagents ; li2025rethinking heavily rely on a predefined LLM cluster with model sizes reaching 141B parameters, which imposes severe limitations on real-world healthcare deployment due to excessive computational resource requirements.

To this end, we propose an SD maximization mechanism that selects LLMs exhibiting a high diversity value within the scope of accessible resources to achieve the adaptive cluster construction. Such a mechanism is motivated by an empirical observation that LLMs generating more diverse outputs tend to achieve better performance, as illustrated in Figure?2. Accordingly, we select LLMs exhibiting a high diversity value from the candidate models as cluster components, where the detail is as follows:

We first employ a fast string matching algorithm max_bachmann_2024_10938887 to calculate the output diversity of LLMs since it is useful for detecting partial matches in string data. Specifically, we sample 10 outputs from a single LLM $\mathbf{L}_{\bf{I}}$ to the same question, denoted as $\{\mathbf{O}_{\bf{I}}^{j}\}_{j=1}^{10}$. For any given pair of outputs, take $\mathbf{O}_{\bf{I}}^{1}$ and $\mathbf{O}_{\bf{I}}^{2}$ ($|\mathbf{O}_{\bf{I}}^{1}|\leq|\mathbf{O}_{\bf{I}}^{2}|$) as an example, we compute their similarity by finding the best matching substring of $\mathbf{O}_{\bf{I}}^{2}$ that aligns with $\mathbf{O}_{\bf{I}}^{1}$. For each position $i\in\mathbf{O}_{\bf{I}}^{2}$, the substring is obtained as follows:

where $\mathbf{O}_{\bf{I}}^{1}$ slides over $\mathbf{O}_{\bf{I}}^{2}$ with a window of size $|\mathbf{O}_{\bf{I}}^{1}|$. The similarity value of each window can be computed using Levenshtein distance levenshtein1966binary with its mathematical definition being

where $D(\mathbf{O}_{\bf{I}}^{1},\mathbf{O}_{\bf{I}}^{sub})$ denotes the Levenshtein distance of $\mathbf{O}_{\bf{I}}^{1}$ and $\mathbf{O}_{\bf{I}}^{sub}$. Afterward, the output diversity of $\mathbf{O}_{\bf{I}}^{1}$ and $\mathbf{O}_{\bf{I}}^{2}$, termed as the SD value, can be computed as:

Similarly, we can obtain the SD values for all other pairwise outputs in $\{\mathbf{O}_{\bf{I}}^{j}\}_{j=1}^{10}$, resulting in a total of 45 SD values (i.e., $C_{10}^{2}$). Finally, we take the mean of the above SD values as the final SD value for the LLM $\mathbf{L}_{\bf{I}}$, where the SD value of the LLM is higher, its output is more diverse.

Previous models achieve the collaborativeness of LLMs through the iterative aggregation of entire outputs, where the current layer aggregates the outputs of all LLMs in the previous layer, inevitably leading to interference from low-quality redundant outputs and substantial time consumption.

To mitigate this issue, we use a CC maximization mechanism to iteratively mask the LLM with the lowest pairwise CC value layer by layer, allowing adjustable aggregation by setting the number of masked LLMs. The implementation involves three key steps: (1) measuring the pairwise CC value between the LLM with the highest SD value and other LLMs; (2) masking the LLM with the lowest pairwise cross-consistency value iteratively; (3) propagating the outputs of remaining LLMs, where each LLM within the current layer generates its output by integrating outputs of screened LLMs within the previous layer as auxiliary context.

The illustration is given in Figure 3 and its mathematical definition is as follows. Let $\mathbf{L}_{\bf{I}},\mathbf{L}_{\bf{II}},\mathbf{L}_{\bf{III}},\mathbf{L}_{\bf{IV}},\mathbf{L}_{\bf{V}},\mathbf{L}_{\bf{VI}}$ be the cluster of LLMs, $\mathbf{L}_{\bf{I}}$ is the LLM with a highest SD value. First, we obtain the inferred output $\mathbf{r}_{1}$ of the first layer by

where $\mathbf{L}_{clu}=\{\bf{I},\bf{II},\bf{III},\bf{VI},\bf{V},\bf{IV}\}$ denotes the cluster indexs of LLMs, $\mathbf{x}_{0}$ denotes the input information, $+$ and $\sum$ denote the concatenation of outputs, $\mathbf{L}_{j}(\mathbf{r}_{0})$ denotes the output of LLM $\mathbf{L}_{j}$ with $\mathbf{r}_{0}$ being the input, $\bigoplus\left(\cdot\right)$ denotes the application of the aggregation prompt. For the sake of readability, we simplify their outputs $\mathbf{O}_{\bf{I}}^{1},\mathbf{O}_{\bf{II}}^{1},\mathbf{O}_{\bf{III}}^{1},\mathbf{O}_{\bf{IV}}^{1},\mathbf{O}_{\bf{V}}^{1},\mathbf{O}_{\bf{VI}}^{1}$ as $\mathbf{O}_{\bf{I}},\mathbf{O}_{\bf{II}},\mathbf{O}_{\bf{III}},\mathbf{O}_{\bf{IV}},\mathbf{O}_{\bf{V}},\mathbf{O}_{\bf{VI}}$ subsequently. Afterward, we measure the pairwise CC values between $\mathbf{O}_{\bf{I}}$ and $\mathbf{O}_{\bf{II}},\mathbf{O}_{\bf{III}},\mathbf{O}_{\bf{IV}},\mathbf{O}_{\bf{V}},\mathbf{O}_{\bf{VI}}$ via Eq. (3) for obtaining the lowest pairwise CC index $\mathbf{c}$ by

Thus, we can mask the LLM with the index $\mathbf{c}$ in the $i$-th layer, which can be formalted as:

where $\sum_{j\in\mathbf{L}_{clu}\setminus\{\mathbf{c}\}}$ denotes the concatenation of outputs expect the LLM with the index $\mathbf{c}$. Finally, we can directly obtain the final result $\mathbf{r}_{l}$ with respect to the question $\mathbf{x}_{0}$, where $l$ is the number of layers. The inference process of our method is summarized in Alg. 1.

Datasets. To evaluate the medical decision support capacity of LLMs, we employ two publicly available medical datasets: NEJMQA katz2024gpt and MMLU-Pro-health wang2024mmlu . NEJMQA is derived from Israel’s 2022 medical specialist licensing examination, covering five core clinical disciplines: General Surgery, Internal Medicine, Obstetrics and Gynecology, Pediatrics, and Psychiatry. The dataset comprises 655 single-choice and multiple-choice questions across these disciplines. Notably, physicians must achieve a minimum passing score of 65% in each discipline to obtain board certification. We adopt this threshold as the benchmark for assessing whether LLMs demonstrate physician-level medical decision support capacity. MMLU-Pro-health, a health topic of the MMLU-Pro, contains 818 carefully curated questions spanning eight medical specialties: Virology, Professional Medicine, Nutrition, Medical Genetics, Human Aging, College Medicine, Clinical Knowledge, and Anatomy. Each question underwent rigorous processing, including initial filtering, integration, option augmentation, and expert review to enhance reasoning complexity and ensure precise healthcare evaluations. The detailed statistics are presented in Table 1, with the prompt templates provided in Appendix Tables A1 and A2.

Models. To gain a deeper understanding of the performance advantages of our method, we conduct comparisons with twelve open-access LLMs (phi4 14B abdin2024phi , qwen2.5 14B, qwen2.5 32B qwen2.5 , qwq 32B qwq32b , openthinker 32B openthoughts , deepseek-r1 32B guo2025deepseek , llama3 instruct 70B meta2024introducing , Qwen1.5 Chat 72B, Qwen1.5 Chat 110B bai2023qwen , dbrx-instruct 132B team2024introducing , Mixtral 8x22 141B jiang2024mixtral , and WizardLM 8x22 141B xu2023wizardlm ), two close-source LLMs (GPT-4 and GPT-4o-mini achiam2023gpt , and three SOTA models ( Debate du2023improving , MoA wang2025mixtureofagents , and SelfMoA li2025rethinking ).

Implementation Details. To achieve competitive performance while maintaining low inference costs, our model exclusively employs open-access LLMs ranging from 14B to 32B parameters since a single 32B parameter model requires 21,735 MB GPU memory, equivalent to one NVIDIA GeForce RTX 4090, making the configuration both practical and cost-effective. The specific cluster of LLMs is selected based on their SD values, which includes phi4 14B, qwen2.5 14B, qwen2.5 32B, qwq 32B, openthinker 32B, and deepseek-r1 32B in our model. For fair comparisons, we follow the same prompt template setting as wang2025mixtureofagents to conduct the aggregation of LLMs outputs, which is given in Appendix Table A3. We mask two LLMs in each layer until only one LLM is used to achieve the final inference. We test these open-access LLMs through the Ollama platform and the close-source LLMs via APIs through OpenAI. The model is implemented with PyTorch on NVIDIA GeForce RTX 4090. We ensure strict adherence to the licensing terms of all models utilized in this research.

Metrics. To comprehensively evaluate the performance of the compared models and our method, we exploit a series of evaluation metrics, including accuracy (ACC), weighted F1-score (F1), Precision (PRE) powers2020evaluation , Sensitivity (SEN) yerushalmy1947statistical , Specificity (SPE) saah1998sensitivity , Matthews Correlation Coefficient (MCC) matthews1975comparison , and Cohen’s Kappa (CK) mchugh2012interrater .

Comparisons on diverse disciplines. To assess whether LLMs demonstrate physician-level medical decision support capacity, we conduct the experimental comparison on NEJMQA across five clinical disciplines, and MMLU-Pro-health across eight medical specialties. Particularly, NEJMQA is derived from Israel’s 2022 medical specialist licensing examination, where physicians are required to achieve a minimum passing score of 65% in each discipline to obtain board certification. As shown in Figure?1, the performance of MoA is worse than that of a single LLM in terms of overall ACC, indicating that the collaborativeness of LLMs, which performs well in the general NLP domain, does not work in medical decision support scenarios. Even though GPT-4 achieves 66.41% in overall performance, it does not reach the official passing score of 65% in ‘Obstetrics and Gynecolog’ and ‘General Surgery’ disciplines, indicating that even the most advanced close-source models still have a gap compared with the professional physician in medical decision support scenarios. In contrast, our method obtains the best performance on both NEJMQA and MMLU-Pro-health across physician-oriented disciplines, which verifies the effectiveness.

Evaluation with multiple metrics. Moreover, we conduct the experimental results with seven evaluation metrics on both NEJMQA and MMLU-Pro-health. As shown in Tables 2 and 3, we have the following observations:

• ?

  Our model, composed of 14B to 32B open-access LLMs, can exceed that composed of 70B and 141B, indicating that the advantage of collaborative architecture optimization can improve the performance of LLMs. The reason for the significant improvement is two-fold. First, our model conducts SD-guide cluster construction to pursue the diversity of LLMs since we empirically observe that a single LLM with richer output tends to achieve better performance, also proven by tekin2024llm . Second, our model utilizes a CC-guide mask mechanism to ensure consistency between multiple LLMs layer by layer, achieving adaptive collaborativeness of LLMs.

• ?

  Our model outperforms GPT-4 and GPT-4o-mini close-source models among all the comparisons. For example, on NEJMQA, our approach improves 4.12% over the second-best comparison GPT-4 on ACC, 4.38% on F1, 3.57% on PRE, 4.12% on SEN, 1.47% on SPE, 5.12% on MCC, and 5.48% on CK. In addition, on MMLU-Pro-health, our approach improves 4.03% over the second-best comparison GPT-4 on ACC, 4.14% on F1, 4.20% on PRE, 4.03% on SEN, 0.45% on SPE, 4.51% on MCC, and 4.51% on CK.

Inference Cost Analysis. We analyze the inference costs of running time and memory occupation relative to ACC performance in Figure?4, where we observe that most single LLMs show insignificant performance improvements due to limitations in their model size and expressiveness. Although MoA surpasses most single LLMs through iterative aggregation of model outputs, it incurs substantial memory occupation and running time. In contrast, our model achieves over 6.11% higher ACC than MoA while reducing memory usage by 70,206 MB and running time by 41,855.52 seconds.

We conduct ablation studies to evaluate the effectiveness of the SD and CC strategies and further analyze the influence of different mask mechanisms.

Analysis of SD and CC strategies. We conduct comprehensive ablation studies with seven evaluation metrics to deeply understand the proposed SD and CC strategies. The experimental results are listed in Table 4, where the first row denotes the baseline wang2025mixtureofagents . The second and third row denotes the variant of baseline that exploits the SD and CC strategies, respectively. The fourth row is our whole model, i.e., Our. From Table 4, we have the following observations that the advantage of the SD and CC strategies could be validated by comparing the results of the second and third rows with Our of each metric. For example, on NEJMQA, it can be seen that the simultaneously considering the SD and CC strategies could produce a 6.11% to 9.47% performance improvement.

Analysis of the mask mechanism. To evaluate the advantage of our CC-driven adaptive mask mechanism, we investigate different mask mechanisms in Table 5, where the ‘random’ indicates the mask mechanism that randomly masks out the LLM layer by layer, ‘sequence’ indicates the mask mechanism that masks out the LLM with the smallest individual SD value of LLMs layer by layer in ascending order, ‘Our’ indicates the proposed mask mechanism using the CC maximization mechanism. From Table 5, we have the following observations that using CC maximization to adaptively mask low-consistency LLM in each layer is capable of improving the performance in medical decision support scenarios.