Pyunghwan Ahn
2026.06.26
[CVPR 2026] An upcycling method for creating true experts within a Mixture-of-Experts structure
The most intuitive way to improve the performance of an AI model is to make it larger. However, in a dense model where all parameters process every input, the costs of training and inference increase in proportion to the model’s size. Mixture-of-Experts, or MoE, is a structure developed to mitigate this problem. MoE consists of multiple subnetworks (Experts) within a single network, and by activating only a subset of these experts for each input token, it allows the overall model capacity to be increased while keeping the actual computational load relatively low.
One method for efficiently training such MoE structures is upcycling. This approach initializes the MoE model using the parameters of a pre-trained dense model and then performs additional MoE training for a relatively short period, thereby achieving high performance with relatively low training costs. The paper “Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling”[1], presented at CVPR 2026, analyzes how the initialization method used when upcycling a pre-trained dense model into an MoE model affects the final training results and proposes an initialization method for effective training.
Image 1. Sparse upcycling[2] concept
“We switched to MoE, but the expert doesn’t 'act like an expert'”
The appeal of the MoE structure lies in its ability to achieve high performance with minimal inference costs. Looking at real-world industrial applications, cost is just as important as performance when operating large-scale AI models. While performance may improve as models continue to grow, dense structures that use all parameters every time come with the burden of increasing both training and inference costs. In contrast, MoE structures retain the majority of the parameters but call upon only a subset of experts for each input, thereby ensuring inference efficiency.
However, training a MoE from scratch remains costly. To address this, a Google Research group proposed “sparse upcycling”[2] in 2022, a method for repurposing existing dense checkpoints as MoEs. The basic approach involves copying the FFN of a dense model to multiple experts and reinitializing the router. The problem is that all experts start with the same parameters. While it may appear that multiple experts have been created, they are actually closer to “duplicates” that perform similar functions, which can weaken early specialization. The paper defines this as the “expert symmetry” problem.
Previous studies have attempted to break this symmetry through methods such as noise injection, partial parameter reinitialization, and fine-tuning based on passive domain partitioning. However, noise may have limited effectiveness, methods that significantly alter parameters risk damaging the pretrained representation, and domain-specific fine-tuning becomes less scalable as the number of experts increases.
LG AI Research’s “Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling”
LG AI Research analyzed how the final training results and performance can vary depending on the initialization method used when upcycling pre-trained dense models into MoE models, and proposed an initialization technique for more effective training.
This paper is based on the core premise that “the activations of an already well-trained dense model contain the semantic structure of the data.” Therefore, rather than copying all experts with the same weights, it is possible to first examine the activation space of the dense model, identify regions where semantically similar inputs cluster, and initialize each expert to handle a different region. The paper refers to this strategy as “cluster-aware upcycling.”
To use an analogy, traditional sparse upcycling is like placing several employees who have received the same training in a room and telling them, “Now, each of you should find your own area of expertise.” In contrast, cluster-aware upcycling is more akin to first analyzing work data to group semantically related tasks such as “teams strong in image-text search” or “teams sensitive to specific visual patterns,” then assigning each employee to the appropriate task from the start.
Step 1. Clustering FFN activations with spherical k-means
In the first step, we extract the activations input to each FFN block of the pretrained dense model using a small calibration dataset. We then divide the extracted activations into a specified number of clusters using cosine-similarity-based spherical k-means clustering. This choice is important because the logit calculation in the MoE router is also based on directional alignment, that is, the similarity between the activations and the router weights. As a result of this process, a number of activation clusters and centroids equal to the number of experts are generated. The paper views these clusters and centroids as a semantic partition of the activations generated by the dense model and uses them as the basis for initializing the experts and the router.
Step 2. Initializing each expert according to the cluster subspace
The core objective of this paper was to preserve the capabilities of the pretrained dense model as much as possible while ensuring that the unique capabilities of each expert were differentiated from one another. In other words, each expert must be initialized to effectively preserve the input distribution of its corresponding cluster, and data-aware truncated SVD was used to achieve this. Rather than performing SVD based solely on the weight matrix, it preserves the principal singular directions by reflecting the input statistics of the respective cluster. This approach allows each expert to be responsible for a different principal subspace while retaining pretrained knowledge. As a result, it is possible to reduce expert redundancy while maintaining the benefits of upcycling.
Image 2. Cluster-aware initialization concept diagram. [1]
After dividing the FFN input activations of the dense model into clusters, the subspaces for each cluster are used for expert initialization, and the cluster centroids are used for router initialization.
EESD: Refer to the ensemble teacher when routing is uncertain
The paper proposes a new loss function to prevent potential losses arising from uncertain expert assignments during the training process. In MoE, when it is unclear which expert a token should be assigned to, resulting in nearly uniform router probabilities, that token may hinder expert specialization. To prevent this, EESD uses a dense ensemble teacher updated via the EMA method. While standard sparse MoE activates only the top-k experts, the ensemble teacher activates all experts simultaneously to provide more stable predictions; this teacher is designed to provide strong guidance for tokens with high routing uncertainty. Conversely, for tokens where the router is already confident, the loss is minimized so as not to interfere with existing specialization.
Verification through experiment: CLIP ViT-B/32 and ViT-B/16
The experimental setup for this paper used CLIP ViT-B/32 and ViT-B/16 as dense baselines. Dense pre-training followed the LAION-400M-based openCLIP setup, and for MoE upcycling, a dense checkpoint from the 15th epoch was used, alternating between the existing dense layer and the MoE layer.
Furthermore, the ablation study yielded significant results: even when only cluster-aware initialization was applied, retrieval and classification metrics improved compared to sparse upcycling; while the effect of EESD alone was limited, combining it with cluster-aware initialization resulted in additional performance gains. This demonstrates that the roles of creating initial semantic specialization and maintaining specialization during training are complementary.
Image 3. Cluster-aware upcycling has been shown to reduce similarity among experts and stabilize routing entropy.
This demonstrates that the internal structure of MoE has become more specialized, going beyond a simple performance improvement. [1]
Future plans
This research by LG AI Research suggests that cluster-aware upcycling can be applied not only to vision-language pre-training, but also to the broader problem of MoE upcycling. In particular, the approach of initializing both the expert and the router together by leveraging the representation geometry of dense models is scalable not only to CLIP but also to large-scale multimodal models, vision backbones, and LLM-based MoE. However, for actual scalability, further verification is needed to confirm whether the same effectiveness is maintained when model size, the number of experts, data domains, or routing methods vary.
During the actual model development phase, high-performance models can be easily utilized depending on the specific application. However, analyzing whether these models are functioning as intended and conducting research and development on the underlying technologies that support them are tasks that must be carried out in parallel on an ongoing basis.
As announced in this paper, LG AI Research plans to continue conducting research to verify whether each expert in a model incorporating the MoE structure is truly fulfilling its role as an “expert,” and to design models that operate as intended. Through these efforts, we aim to enhance the interpretability of the models we use.
- 참고
[1] Chu, Sanghyeok, et al. "Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.
[2] Komatsuzaki, Aran, et al. "Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints." The Eleventh International Conference on Learning Representations.