Kimi K3 at Scale: Inside Our Serving Stack

Moonshot AI shipped Kimi K3's weights on July 27, and the numbers alone explain why serving it well is a genuinely different engineering problem than serving most models that came before it: 2.8 trillion total parameters, with only 104 billion active on any given token. That sparsity is exactly what makes K3 viable to run at all — and exactly what makes serving it efficiently a real infrastructure challenge rather than a routine deployment.

What Kimi K3 Actually Is

K3 is a mixture-of-experts model with 896 routed experts, 16 of which are selected per token, plus 2 shared experts that are always active. Across its 93 layers — 69 built on Kimi Delta Attention and 24 using Gated MLA — the model presents a 1,048,576-token (1M) input context window, a default max output of 131,072 tokens, and a maximum output that can extend to the full 1M-token window. It ships with native vision through a 401M-parameter MoonViT-V2 encoder, always-on reasoning with adjustable effort levels (low, high, max), and was released as a native MXFP4 checkpoint under the Kimi K3 License — free for internal use, with a separate commercial agreement required only past a $20M Model-as-a-Service revenue threshold.

On independent benchmarks, Moonshot reports 81.2 on FrontierSWE and 88.3 on Terminal-Bench 2.1; Artificial Analysis's own testing puts K3 at 44 on their Intelligence Index v4.3, competitive with GLM-5.3 at 45 among open-weight models. Whatever angle you look at it from, K3 is a serious, frontier-class open-weight model — and serving it well requires taking its specific architecture seriously rather than treating it like a bigger version of a dense model you've already deployed.

The Serving Challenge: Sparse Compute, Dense Memory

The defining tension in serving any large MoE model is that its compute cost and its memory footprint scale completely differently. Only 104B parameters activate per token, which keeps the actual FLOPs per forward pass in a manageable range relative to the model's total size. But all 2.8T parameters — the full set of 896 routed experts plus shared experts across 93 layers — still have to live somewhere accessible enough that any of them can be reached with low latency the moment the router selects them.

This means the memory planning problem for K3 isn't "how much memory does inference need," it's "how do we distribute 2.8T parameters across a GPU fleet such that expert routing doesn't become the bottleneck." Our serving stack shards K3's experts across multiple nodes with routing-aware placement — experts that co-occur frequently in the router's selections are kept topologically close, so a token's forward pass doesn't spend more time on cross-node communication than on actual computation. Getting this placement wrong is the single most common reason MoE models underperform their theoretical throughput in naive deployments.

Why Kimi Delta Attention Changes the Long-Context Economics

A 1M-token context window is only useful in production if serving it doesn't come with prohibitive memory and latency costs, and this is where K3's Kimi Delta Attention layers do real architectural work: Moonshot reports up to 75% KV-cache reduction versus standard attention mechanisms at comparable context lengths. For a serving stack, this is the difference between "1M context is a marketing number nobody actually uses at full length" and "1M context is something we can actually serve at a sane cost per request."

Our inference stack takes advantage of this directly — the reduced KV-cache footprint means we can hold more concurrent long-context sessions per GPU than a standard-attention model of comparable size would allow, which is the practical lever that turns K3's context window from a spec-sheet number into usable capacity for customers actually working with long documents, codebases, or extended conversation history.

Native MXFP4: A Deployment Advantage We Didn't Have to Build Ourselves

K3 shipping as a native MXFP4 checkpoint — rather than a full-precision release that downstream providers have to quantize themselves — simplifies a meaningful part of the serving pipeline. Quantization introduces its own accuracy and stability risk when it's applied post-hoc by a serving provider trying to compress a model that wasn't designed with that precision in mind; a native low-precision checkpoint means the model's own creators validated behavior at this precision level before shipping it. Our stack serves the native MXFP4 weights directly, which keeps both our memory footprint and our confidence in output quality closer to Moonshot's own reported benchmarks than a third-party quantization pass would.

Serving the Vision Encoder Alongside the Core Model

K3's native vision capability, via the 401M-parameter MoonViT-V2 encoder, is small relative to the 2.8T-parameter language backbone but architecturally distinct — it has its own compute profile and its own latency characteristics, and treating it as an afterthought bolted onto the text pipeline is how multimodal serving degrades quietly under load. Our stack routes vision-bearing requests through a dedicated encoder path sized independently from the language model's serving capacity, so a burst of image-heavy requests doesn't starve text-only throughput, and vice versa.

Adjustable Reasoning Effort as a Cost and Latency Lever

K3's low/high/max reasoning effort levels give us — and our customers — a real lever that most model deployments don't expose: the ability to trade latency and cost against reasoning depth per request, rather than committing to one fixed configuration for every call regardless of whether the task actually needs it. Our routing layer lets this be set per request, so a simple classification call and a genuinely hard multi-step reasoning task aren't paying the same computational price by default.

What This Looks Like for You

Whether you're running K3 through our API for a production application or proxying it into a roleplay or chat front end, the serving-stack decisions above are what stand between "the model works" and "the model works at the latency and cost you actually need." That's the part of running a 2.8T-parameter open-weight model that most teams don't want to build themselves — and it's exactly what our infrastructure is built to handle underneath a simple API call.

Want to try Kimi K3 on infrastructure built for its specific architecture? Sign up and explore now.

What’s Next?

Sign up and explore now.

🔍 Learn more: Visit our blog and documents for more insights or schedule a demo to optimize your enterprise AI context management.

📬 Get in touch: Join our Discord community for help or Contact Us.

Stay Connected

💻 Website: meganova.ai

🎮 Discord: Join our Discord

👽 Reddit: r/MegaNovaAI

🐦 Twitter: @meganovaai