← Insights
中文 English
Essay DARE to B2B 2026-07-21 ~11 min read

When AI Engineering Moves Inside the Model

Over the past three years, AI engineering has shifted from applications to infrastructure—and now into model architecture, training, and inference.

When AI engineering moves inside the model

Every major wave of AI has had its engineering bottleneck.

In 2024, the challenge was building compelling AI products. In 2025, it became scaling models and infrastructure efficiently. In 2026, something more fundamental is happening: engineering constraints are becoming model-design variables.

This does not mean applications or infrastructure no longer matter. They continue to evolve in parallel. What has shifted is where engineering effort creates the largest leverage.

I still believe in a full stack—“chimney”—view of AI:

Chip → AI datacenter → AI Infra → Model → Agent → App

That chimney remains valid. What changed is which segment absorbs the pressure first—and whether the pressure stays outside the model, or gets written into the model itself.

By AI Infra, I mean both training infrastructure (clusters, storage, interconnect, scheduling) and serving infrastructure (inference frameworks, model hosting, deployment systems). The interesting story of the last three years is not that one layer replaced another. It is that the definition of “AI engineering” kept moving.

I. 2024 — Application Engineering

In 2024, the loudest competition was not “who has the strongest foundation model.” It was “who can turn a usable model into a product people open every day.”

That year, product engineering meant:

You could see this across very different bets.

ChatGPT set the consumer template: conversational interface as the default shell for general intelligence. Claude Artifacts showed that the product surface could move from chat transcript to working artifact. AI coding assistants turned latency, tool use, and editor integration into the real product—not the underlying model card. In China, Doubao optimized for mass-market distribution; Kimi productized long-context reading; Kling moved generative video from demo to creator workflow; ByteDance and others embedded models into content, office, and consumer surfaces users already understood.

The pattern was consistent:

Better models enabled better products. Engineering differentiation mostly happened outside the model.

That approach worked—until it didn’t.

This wasn’t surprising. Most applications simply followed where model capability moved. Many apps acquired users quickly and then discovered that their “moat” was a thin wrapper. When context windows grew, reading apps lost their specialness. When tool use improved, workflow glue got thinner. When the next model upgrade landed, yesterday’s product definition needed rewriting.

There was also a subtler failure mode: users adopted features, not products. They wanted a summary, a draft, an image, a coding patch. Loyalty to any one AI app stayed fragile because switching costs were low and model providers kept raising the floor underneath everyone.

None of this means 2024 was wasted. Application engineering proved that distribution, UX, and workflow design can create real demand even before foundation-model leadership is settled. It also proved the limit of that strategy. If the model moves, the app has to redefine itself. By late 2024, the center of gravity was already sinking one layer deeper.

II. 2025 — Models × Infrastructure

In 2025, the industry’s consensus shifted. Large Language Models (LLMs) were absorbing capabilities that used to live in applications. Search, writing, coding, research, and even parts of agent execution became less like separate product categories and more like surfaces on top of model progress.

A useful way to say it: model upgrades forced applications to redefine themselves. Longer context made “long-document reader” less differentiated. Stronger reasoning collapsed workflows that used to require hand-built planners. More reliable tool use pushed agent execution toward shared infrastructure rather than one-off app logic.

At the same time, user tolerance collapsed. Slow responses, unstable reasoning, forgotten context, and expensive calls no longer felt like infrastructure issues. They felt like product failure.

So attention returned to foundation models—and immediately ran into infrastructure.

This is where the discussion often gets misread as “more GPUs.” Hardware matters, but the binding constraints were more specific:

In other words: once model capabilities started converging quickly, infrastructure efficiency became product capability.

DeepSeek made that hard to ignore: open weights plus systems co-design showed frontier-ish reasoning could diffuse faster than the closed-lab template assumed. Qwen mattered for a different reason—becoming the model developers actually run. Moonshot’s Mooncake treated serving as a competitive surface, not a post-training afterthought.

None of these were only “model releases.” They were arguments about training–serving co-design under hard constraints. Especially where “just buy a larger cluster” is not free, systems efficiency becomes part of the model’s identity.

By the end of 2025, a useful simplification held:

2024 asked: how do we wrap a model into a product? 2025 asked: how do we train and run a model continuously, stably, and cheaply enough for the product to survive?

Applications were still busy. The industry’s center of gravity moved down the stack.

III. 2026 — Engineering Moves Inside the Model

2026 is when the next shift becomes visible.

The important change is not that Chinese labs suddenly “won” against overseas SOTA. The important change is that more of the decisive engineering work is no longer sitting around the model. It is landing in architecture, training objectives, and inference mechanisms.

Put differently: problems that used to be deferred to frameworks and clusters are increasingly treated as design inputs at model-creation time.

I find it useful to split this into three layers.

Architecture

DeepSeek’s MLA and Kimi’s KDA (Kimi Delta Attention) are not interesting merely as new attention variants. They are interesting because they treat inference pressure as an architectural objective.

MLA compresses key–value representations to reduce KV-cache pressure. KDA explores hybrid / linear-leaning attention to lower compute and cache cost on long sequences. In both cases, the claim is structural:

Memory and long-sequence cost should not only be scheduled at serving time. They can be reduced by changing what the model is.

That is what makes this phase fundamentally different. The bottleneck enters the math. Once an attention design absorbs a serving constraint, every downstream framework inherits the gain—often more cleanly than a pile of runtime heuristics stacked on top of an unchanged architecture.

Training

On the training side, engineering is also migrating inward.

Aux-loss-free load balancing for MoE is not a cosmetic training trick. It is an attempt to put expert load and communication efficiency into the training mechanism itself, rather than endlessly patching imbalance from outside. Moonshot’s MuonClip pushes a related idea further into optimizer design: stability at scale becomes part of the learning algorithm, not only a bag of learning-rate heuristics and emergency gradient clips.

MTP (multi-token prediction) sits at an especially revealing junction. As a training objective, it densifies the learning signal by predicting multiple future tokens. The same family of ideas can also support speculative decoding on the serving side. Training and inference stop looking like sequential departments and start looking like one efficiency program with two interfaces.

Inference

DeepSeek’s DSpark is a clean example of the third layer: speculative decoding designed for high-concurrency serving, combining semi-autoregressive drafting with confidence-based dynamic verification.

This is not “train a model, then bolt on an accelerator.” Draft generation, verification policy, and online serving have to be co-designed around the same latency / throughput target. Training–inference co-design stops being a slogan and becomes the product. The model is no longer only the thing being served; it becomes part of how serving works.

Zoom out, and the 2026 pattern is consistent:

Memory, communication, throughput, stability, and cost are no longer merely system concerns. They increasingly shape the model itself.

This is what I mean by “engineering moving inside the model.” Not that infrastructure disappeared. Not that apps stopped mattering. Rather: the boundary between model engineering and systems engineering is blurring, because the model is becoming one of the places where systems problems get solved.

Open ecosystems reinforce the same shift. Qwen’s push is not only about releasing weights; it is about becoming a default-compatible model across Hugging Face, vLLM, SGLang, llama.cpp, Ollama, and the rest of the serving graph. In an open world, “who has the best checkpoint” and “who is the easiest model to run everywhere” are converging competitions.

So 2026 is not mainly a story of one lab leaping past another on a single leaderboard. It is a story of independent technical rhythm: papers, architectural bets, training recipes, serving systems, and open ecosystems that no longer wait for an overseas release to define the next move. The gap to SOTA has been narrowing for years. What is new is how it narrows—by pushing engineering constraints into the model rather than only catching up feature-by-feature after the fact.

IV. AI Runtime

If models keep closing the capability gap—and especially if open models keep compressing the scarcity premium of closed frontiers—then a different question rises:

If models become commodities, what becomes the operating system?

I do not mean another chat UI. I mean the layer between models and work: the place that decides what runs, with what memory, under what permissions, through which skills, inside which workflow.

Call it an AI Runtime.

A useful runtime is not a single router that picks “the smartest model.” That framing is too model-centric. Real workloads are messy: some need deep reasoning; some need cheap classification; some need long memory; some need strict permissions; some need a specialized harness for coding, browsing, or enterprise retrieval. The runtime’s job is to make those tradeoffs continuously, not once at procurement time.

In practice, that looks closer to an intent-and-scenario switchboard:

This is why harness, memory, skills, permissions, and workflow matter more than any one protocol buzzword. Protocols can standardize connections between models and tools. They do not automatically standardize identity, state, authorization, execution semantics, or cross-harness portability. A skill that cannot move across models without rewriting its permissions and memory model is not yet an open ecosystem asset; it is a platform feature.

There is also a product truth hiding here. Many teams still equate “best experience” with “strongest model.” That was a reasonable heuristic when capability gaps were enormous. It becomes less reasonable as models converge and as cost, latency, and reliability dominate lived experience.

The best user experience is not always powered by the most capable model. It is powered by the right model at the right time—inside a runtime that can compose models, harnesses, skills, and memory.

Enterprises will not deploy “an LLM.” They will deploy a production system: models + harness + enterprise memory + permissions + tools + workflows + interface. As model interchangeability rises, the sticky layer becomes the runtime that can mix and replace models without rewriting the business. That is also where safety, observability, and evaluation become permanent products rather than release-week checklists.

V. Beyond the LLM Tax

There is a commercial consequence to all of this.

When frontier intelligence is scarce, buyers pay a premium simply to access it. By LLM Tax, I mean that scarcity premium—the rent paid because a few closed models sit at the capability frontier—not the full cost of compute, electricity, or engineering labor. Confusing those two leads to bad strategy. Cutting compute cost helps. Compressing scarcity rent is a different lever.

Open-source progress, stronger reasoning under open weights, cheaper inference paths, and better runtimes all attack that tax from different angles.

If the tax falls, value migrates. Some of it moves into open infrastructure: orchestration, memory, safety, observability, evaluation, and the operational layer that lets companies treat models as replaceable components. Some of it stays in differentiated model design—especially where architecture and training encode systems advantages competitors cannot casually copy. Some of it returns to applications, but only where the product owns a workflow, distribution channel, or proprietary data loop the model cannot trivially absorb.

The strategic mistake is to treat these as mutually exclusive bets. The chimney still needs every segment. The question is where incremental engineering effort buys the most leverage this year.

In 2024, that leverage was often productization. In 2025, it was training and serving under constraint. In 2026, more of it is inside the model—and, immediately above the model, in the runtime that decides how intelligence is applied.

Closing

Applications, infrastructure, and models will continue to evolve together.

The shift is not where innovation happens.

The shift is where engineering creates the greatest leverage.

And increasingly, that leverage is moving inside the model.

2024–2026 shift in AI engineering leverage

Chinese long-form companion: AI 工程化，正在进入模型内部 →