Huawei's Ascend computing team has adapted Moonshot AI's Kimi K3.1 Pro for LoRA fine-tuning, validating the training on a 128-card Atlas 950 SuperPoD. According to Guandian and a Chinese technical write-up published on November 6 and 7, this work focuses on training, which is a separate step from the day-zero inference support Ascend announced following the release of the model's weights on November 4.
The Kimi K3.1 Pro is described as an open-weight, natively multimodal mixture-of-experts model featuring approximately 2.8 trillion total parameters. The reports state the model includes 896 routed experts with 16 activated per token, a hybrid attention design where roughly three quarters of attention blocks use KDA linear attention alongside gated sparse attention, and a context window reaching up to one million tokens.
Because of this scale, the reports note significant memory pressure, as the BF16 weights alone occupy roughly 5.6TB before accounting for gradients, optimizer states, and long-sequence activations. To address this, Ascend says it combined FSDP2 sharding of parameters, gradients, and optimizer states with expert parallelism to distribute experts across accelerators. Validation was conducted using an FSDP2 plus EP64 configuration on the 128-card system.
On the operator side, the team says it built large fused forward and backward kernels for KDA attention and used fused AscendC operators for sparse attention. The team claims this resulted in more than twice the performance of the earlier small-operator approach. Additionally, new residual structures were brought up through the Triton Ascend framework and the BiSheng compiler, while MoE routing steps like token permutation and grouped matrix multiplication were fused and overlapped with communication.
A selective caching mechanism called Swap Core was used to keep the outputs of chosen matrix-multiplication operators in a tiered store spanning HBM and DDR memory. Ascend says this raised average training performance by 15% in the current test configuration, though the company did not publish absolute throughput and the figure has not been independently verified. To contain memory for the large vocabulary, chunked loss computation along the sequence and vocabulary dimensions was used.
