additional auxiliary loss for moe?

[maxin-cn]

Thanks for the previous responses! The LongLLaVA model utilizes the Mixture of Experts (MoE), but it seems that MoE has a "winner-takes-all" issue (i.e., Expert Balancing). Was any additional regularization applied to MoE during model training?

[wangxidong06]

Yes we use router_aux_loss_coef with 1e-3 and employ modeling.py in here

[maxin-cn]

Thanks for your reply!