The AI Distillation Fight Now Reaches Into the Answers Customers Receive

The most consequential line in Washington's new AI distillation warning is not the list of Chinese companies. It is the recommendation that a model provider quietly change the answers it gives a suspected adversary. That moves the dispute beyond who trained on whose output. It reaches into the reliability of the service everyone else is buying, and it makes accurate attribution a product requirement as much as a security requirement.

On September 8, the National Security Agency, FBI, and CISA issued a joint advisory alleging industrial-scale extraction of capabilities from American frontier models. The agencies named DeepSeek, Moonshot AI, Alibaba, MiniMax, StepFun, and Z.AI. They said campaigns dating to at least late 2024 involved billions of tokens across millions of exchanges or requests. Those are the agencies' claims, not independently reconstructed measurements from this publication. Their assessment of likely Chinese government awareness is also not proof of government direction.

China disputed the allegations. The Associated Press reported on September 9 that the Commerce Ministry rejected the accusations as groundless and warned of countermeasures if the United States used distillation as a pretext to suppress Chinese companies. The ministry also argued that distillation is common throughout the industry. That response establishes a contested account, not a resolution. Neither a government advisory nor a government denial substitutes for examining the underlying evidence.

Start with the distinction both the advisory and the industry recognize. Distillation is a training technique, not inherently an attack. A model can learn from another model's outputs, including when a developer uses its own stronger system to improve a smaller one. The dispute concerns authorization, the means of obtaining access, and the intended use of collected responses. Calling every use of synthetic training data theft would erase that distinction. Calling every coordinated extraction campaign ordinary research would erase it in the other direction.

The advisory describes an operation spread across the service supply chain. Requests allegedly traveled through native model APIs, cloud providers, aggregators, relays, and pools of subscription accounts. Its account includes centralized routing, removal of identifying metadata, and quality checks designed to distinguish ordinary service trouble from defensive changes. The significance is organizational. The agencies are describing managed acquisition of training material, with infrastructure and cost controls, rather than a collection of people asking clever questions in a browser.

That architecture creates a visibility problem for defenders. A model company may see one portion of a campaign while a cloud intermediary sees another. An aggregator can sit between the end customer and the underlying provider. If each organization investigates only the activity in its own logs, coordinated behavior can look like unrelated accounts. The advisory therefore calls for information sharing across providers, clouds, and aggregators. Its proposed unit of analysis is the campaign, not merely the individual request.

The requested material is also more specific than a pile of generic prose. According to the agencies, campaigns targeted capabilities including coding, tool use, agentic workflows, reasoning, and the evaluation of answer quality. A useful way to understand the commercial concern is to distinguish an answer from a repeatable behavior. An answer can solve one problem. A carefully selected training collection may help another system reproduce a category of behavior. The advisory alleges the latter was the objective; it does not establish that every queried capability transferred successfully.

Anthropic had already published a company account in February. It said three laboratories generated more than 16 million exchanges through approximately 24,000 fraudulent accounts. It described attribution using request metadata, infrastructure signals, and, in some cases, information from industry partners. That earlier disclosure supplies context for the new warning, but it is an interested party's assessment. Its numbers should not be added to the government's figures as though they describe separate, nonoverlapping incidents. The public material does not establish that.

The new advisory's operational recommendations deserve scrutiny on their own merits. It asks providers to correlate unusual throughput, shared account activity, rapid use of new subscriptions, synchronized behavior, and routing patterns. Those can be useful investigation signals. They are not individually sufficient proof that a customer is stealing a model. A legitimate automated service can run continuously. A real company can onboard quickly. A benchmark can ask many similar questions. The distinction requires corroboration, not a dashboard that treats productive customers as suspicious because they are productive.

Consider a hypothetical research team evaluating coding tools. Its test harness could issue repetitive requests, compare multiple providers, and exhaust a quota during a scheduled experiment. Now consider an abusive operation using misrepresented identities and coordinated intermediary accounts to collect restricted training material. Some traffic features could overlap. The practical task is to connect behavior with verified account context and permitted use. A classifier that recognizes repetition but cannot tell those situations apart has identified a pattern, not completed an investigation.

The advisory's most aggressive proposal is targeted alteration of responses. For high-confidence malicious distillation, it discusses using a less capable model or modifying outputs to reduce their usefulness for training. It recommends not telling confirmed malicious distillers about a model downgrade, while saying AI safety researchers and third-party evaluators should be informed of model changes. That exception matters. A researcher cannot meaningfully evaluate a particular system if the provider silently substitutes a different one and lets the result carry the original model's name.

This is a recommendation in a published advisory. It is not evidence that every provider has implemented covert downgrades, or that a particular customer's disappointing answer was deliberately weakened. Those would require separate evidence. But the recommendation does create a legitimate procurement question: what service identity is a customer actually promised? If a provider reserves the ability to change model behavior for security reasons, an enterprise should understand the boundary between ordinary model updates, abuse interventions, and the version its evaluation process approved.

There is an obvious defense of targeted degradation. An attacker should not receive a detailed tutorial on which detection rule fired, and a provider has an interest in reducing the value of unauthorized collection. The counterargument is equally concrete. A mistaken classification could make a legitimate workflow fail in ways that look like random model weakness. Quiet changes can be harder to diagnose than an explicit access refusal. The appropriate conclusion is not that all intervention is wrong. It is that the confidence threshold and recovery process deserve real engineering.

For an operator, I would want the intervention recorded internally, reviewed when a legitimate customer challenges it, and reversible without destroying the evidence that led to the decision. That is an operational recommendation, not a feature the advisory proves any vendor already offers. The record should distinguish why access was considered abusive from what response action followed. Otherwise the team that owns detection can improve its apparent success rate while the team that owns customer reliability absorbs an invisible cost.

The agencies also discuss differential privacy and other defensive techniques. Their document acknowledges a tradeoff between protective noise and useful output, along with the need for tuning and empirical auditing. That is more important than treating the technique's name as a guarantee. A provider needs to specify what information a protection is designed to conceal and measure what happens to legitimate work. Protecting sensitive training information and preventing all useful learning from a service are not interchangeable promises. The report does not supply a universal setting that achieves both.

Information sharing introduces a second set of design choices. The advisory describes sharing infrastructure and behavioral indicators to connect activity across organizations. A responsible implementation should be precise about the information needed for that purpose. The existence of a coordinated-defense goal does not make every customer's prompt relevant to every participating company. For a business buying access through an intermediary, questions about retention, onward sharing, and investigation responsibilities belong in the service review. These are consequences of the proposed defense model, not allegations that a particular intermediary has mishandled data.

An aggregator therefore has two jobs that can pull in different directions. It can make access simpler by normalizing several providers behind one interface. It also needs enough account and routing accountability to investigate abuse when an upstream supplier raises a concern. The advisory's account shows why stripping all context may be convenient for routing but problematic for attribution. Conversely, retaining everything indefinitely is not a thoughtful privacy policy. The useful design is deliberate traceability with defined access and retention, not maximum surveillance marketed as security.

Enterprises developing their own models should separate approved distillation from prohibited collection before a project starts. Identify the source of the training examples, the permission covering their use, and the route through which they were obtained. Have the relevant commercial and legal teams resolve ambiguous rights rather than asking engineers to guess from the fact that an endpoint answered. This is not a legal conclusion about a particular contract. It is a way to avoid building a valuable model around data whose permitted use nobody can explain later.

The same discipline should govern public claims about national advantage. The agencies argue that the alleged campaigns reduce development costs and threaten American technological leadership. That is a consequential assessment. The advisory does not provide a controlled accounting of what each named company's model would have cost without the disputed material, nor a clean experiment separating distillation from every other contribution to its performance. Reporting the allegation is warranted. Declaring that an entire country's AI progress has been explained by one mechanism goes beyond what the public evidence can support.

The next useful proof is not another dramatic adjective attached to the word distillation. It is better evidence of attribution, independently assessable detection performance, and clear handling of legitimate research and customer appeals. Providers need to protect the capabilities they built. Customers need to know they are receiving the service they evaluated. A defense that treats those interests as incompatible will create its own reliability problem. The stronger system detects coordinated abuse while preserving an accountable, testable service for everyone entitled to use it.

LaunchPad positionProtect model access with corroborated attribution and an accountable intervention process, not throughput alone.