什么人不宜喝咖啡

Building upon the established ControlNet framework [36], we propose a distilled dual-branch diffusion architecture with shared latent projection, as illustrated in Fig.?1(a). The frozen VAE [31] encoder $\mathcal{E}$ establishes a deterministic mapping $\mathcal{E}:x_{0}\mapsto z_{0}$ through latent space embedding, where $x_{0}$ denotes the input image and $z_{0}$ its latent representation. The student branch (S) ingests conditional masks through a dedicated ControlNet (S) module, generating encoded mask features $c_{m}$ that integrate with the student diffusion U-Net Decoder (S) through feature injection for noise prediction $\epsilon_{\theta}^{S}$.

The teacher branch (T) processes the paired image through a parallel ControlNet (T) to extract encoded image features $c_{i}$. These image features are fused with the corresponding mask features $c_{m}$ through element-wise summation:

This fused representation $c_{\text{mix}}$ propagates through the teacher’s diffusion U-Net decoder (T) to predict the noise $\epsilon_{\theta^{\prime}}^{T}$. By sharing the forward process between the student and teacher branches, the architecture employs a unified latent space projection and diffusion U-Net encoder, significantly optimizing memory efficiency. The composite objective function integrates the following components:

with Denoising Objectives defined as:

where $\theta$ and $\theta^{\prime}$, as in ControlNet [36], are both initialized with the parameters of a pretrained diffusion model, denote mutually independent parameters for each branch, and are optimized separately during training. Meanwhile, $\epsilon\sim\mathcal{N}(0,I)$ ensures stochastic consistency.

During sampling, as shown in Fig.?1(b), medical images are generated using the student branch with arbitrary masks at the same speed as ControlNet [36].

The spatial alignment between synthesized lesion regions and their corresponding masks is critical for downstream segmentation tasks. However, the severe lesion-background imbalance in medical image synthesis often leads to the underrepresentation of lesion regions. To address this issue, we propose a spatially adaptive distillation mechanism that enables the teacher model to dynamically modulate the regularization intensity for the student model, thereby emphasizing the learning of lesion-specific morphological features in the student model.

Unlike previous approaches that apply reweighting techniques to denoising losses [5], our method introduces lesion-aware attention through dual-stream gradient modulation, effectively addressing the lesion-background imbalance. The adaptive weight $w_{\text{Ada}}$ is derived from the mask statistics, with distinct weights assigned to lesion and lesion-free regions:

where $N_{\text{lesion}}$ and $N_{\text{lesion-free}}$ denote pixel counts for respective regions, and $N_{\text{total}}=H\times W$ represents the total number of pixels in the image. These weights are normalized to form a spatially adaptive $W\times H$ weight matrix. The final adaptive distillation loss is formulated as:

where $\text{sg}(\cdot)$ indicates stop-gradient operation.

We evaluate our method on two publicly available medical datasets: Polyps [12, 1] (RGB) and KiTS19 [6] (CT, 2D slices), referred to as Real Datasets.

Generative Model Training: For Polyps, we use images from Kvasir [12] and CVC-ClinicDB [1]. For KiTS19 [6], 50 cases are randomly selected from 210 labeled cases, sliced into 2D, filtering out lesion-free slices.

Generative Model Sampling and Evaluation: Following [5], synthetic images are generated using masks from Real Datasets, referred to as Synthetic Datasets, and evaluated using FID [7] and CLIP-I [21].

Segmentation Model Training: Synthetic Datasets are combined with the Real Datasets as a new training set to train segmentation models.

Segmentation Model Testing and Evaluation: The Polyps test set includes images from five public datasets: EndoScene [32], CVC-ClinicDB [1], Kvasir [12], CVC-ColonDB [28], and ETIS [24]. For KiTS19 [6], 10 non-overlapping cases are selected from 210 labeled cases, sliced into 2D, filtering out slices without lesions. Evaluation metrics include mDice and mIoU for Polyps, and mDice, mIoU, Accuracy, and Recall for KiTS19.

We detail the configuration of the generative and segmentation models as follows:

Generative Model: We use the pre-trained Stable Diffusion v1.5 [20]. The training setup is the same for both datasets: the AdamW [15] optimizer with a learning rate of $10^{-5}$ and weight decay of $10^{-2}$ is used for 3,000 iterations on 8$\times$NVIDIA 4090 GPUs (global batch size of 32) with $384^{2}$ resolution inputs. A 5% probability for prompt dropout is applied. Sampling employs classifier-free guidance [9] (CFG=9) and deterministic DDIM [25] sampling ($\eta=0$, 50 steps), as described in [36]. T2I-Adapter [17] and ControlNet [36] share the same configuration as our method, while SinGAN [29] and ArSDM [5] use their default settings. Notably, for ControlNet [36], unlocking the weights of Stable Diffusion is more effective for medical image synthesis.

Segmentation Model: Both CNN-based and Transformer-based models are utilized with default configurations. Specifically, nnUNet [11] is trained for 200 epochs with five-fold cross-validation, and the final results are obtained by ensembling five models, followed by postprocessing.

Fig.?2 demonstrates that the teacher model’s adaptive regularization accelerates the student model’s data fitting within approximately 300 steps, mitigating the sudden convergence phenomenon in ControlNet [36].

Fig.?3 and Fig.?4 present kidney tumor and polyp images generated by various methods. SinGAN [29], although designed for the Polyps dataset, often introduces artifacts and lacks diversity. ArSDM [5] suffers from texture degradation in polyps and fails to generalize to KiTS19 due to its task-specific nature. T2I-Adapter [17] generates unrealistic textures in RGB data and underperforms on CT data. ControlNet [36] struggles with mask-lesion alignment. In contrast, our model excels in both mask-lesion alignment and morphological features, clearly outperforming the others.

Table?1 shows FID [7] and CLIP-I [21] results. Notably, more precise mask-lesion alignment does not significantly lower the FID score, with our method’s FID score slightly higher than ControlNet [36]. We attribute this to the inherent limitations of FID [10], which overfits with limited data. Nevertheless, CLIP-I [21] confirms our method achieves higher semantic similarity.

Table?2 and Table?3 highlight the enhancement of segmentation models using synthetic data from various generative models. We establish a new baseline by retraining models on a duplicated dataset (i.e., “Copy-Paste”). Our method significantly outperforms others. On KiTS19 [6], it improves mDice by 2.4%, mIoU by 4.2%, and Recall by 2.3% over TransUNet [2], and mDice by 1.4%, mIoU by 2.6%, and Recall by 1.4% over nnUNet [11]. On Polyps, our method outperforms nnUNet [11] by 1.2% in mDice and 1.8% in mIoU, SANet [33] by 2.6% in mDice and 3.5% in mIoU, and Polyp-PVT [4] by 1.1% in mDice and 1.3% in mIoU. Interestingly, in comparison to ArSDM [5] and ControlNet [36], we observe that there is no consistency between image quality and segmentation performance, indirectly highlighting that our method’s superior mask-lesion alignment is key to improvements across diverse segmentation models.