Mistral Large 4 Makes Refusal Part of the Cybersecurity Contest

Mistral's new cybersecurity pitch turns a model's willingness to answer into part of its competitive argument. Large 4 entered public preview on October 6, with downloadable weights still pending. The launch puts a practical question in front of security teams: how much does a benchmark score reveal about finding and repairing software flaws, and how much does it reveal about which tasks a provider permits?

TechCrunch's reporting makes the release boundary clear. For now, access is through a guardrailed public endpoint, not a set of weights a customer can already run locally. Mistral science executive Pierre Stock told the publication that trusted partners and governments would participate in safety testing before the planned release. That separates today's service from the future deployment proposition. The testing period is relevant to the announcement, but its existence does not establish that the eventual downloadable model is safe under every operator's policies.

Mistral reports an 82% score on the reproduce-and-patch test within the Artificial Analysis Cyber Index and 93% on Cybench. Its launch chart identifies the former as CyberGym-E2E. These are attributed results, not measurements reproduced for this article. The company's own announcement targets the end of October for the weights and says vetted partners and state authorities will receive access with reduced moderation and expanded cyber capabilities. A reader should not assume that every endpoint, partner configuration and future release will behave identically.

The Cyber Index is narrower than an all-purpose ranking of security expertise. Artificial Analysis combines three evaluations with equal weight: CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA. The first and third use single-attempt pass rates; the middle uses the median F2 score across three runs. It measures defensive work with source-code access, excluding the conversion of a vulnerability into a working exploit. It is also separate from the company's general Intelligence Index. Calling a model strong on this composite does not establish its ability to handle a live intrusion or investigate software without source access.

Refusals have an unusually visible role in that composite. Artificial Analysis assigns a declined task zero and separately reports safety blocks. That is useful for a buyer because a service that declines an authorized task cannot finish it. It also creates an interpretive trap: a zero caused by refusal is not evidence that the underlying model lacked the technical ability. Conversely, agreement to attempt a task is not evidence that the attempt succeeded. Reading the completion score alongside the refusal measure is more informative than treating either as a complete description of capability.

There is a legitimate argument for making willingness part of an operational score. A security engineer buys access to a usable service, not hypothetical intelligence concealed behind a restriction. There is an equally legitimate reason to resist turning that argument into a universal preference for fewer refusals. Authorization is part of the job. A useful comparison must distinguish declining legitimate defensive work from declining abuse, rather than rewarding every answered request. That is an evaluation-design problem, not something a single ascending leaderboard can settle.

The Artificial Analysis implementation of CyberGym uses 131 tasks, one from each selected C/C++ project, rather than the original dataset in its entirety. Its methodology gives agents 90 minutes per task on the Stirrup harness. The agent must produce an input that crashes an unpatched build, supply a patch that stops that crash, and preserve the project's functionality tests. Those three cumulative checks determine success. This is a more demanding exercise than describing a suspected bug in fluent prose: the submission must survive execution.

A fourth check asks whether the patch also fixes the benchmark's designated ground-truth vulnerability. Artificial Analysis records that result as a diagnostic, not as part of the headline score. Verification runs separately from the agent's environment, rebuilding from original source. These choices matter when interpreting Mistral's reported result. A passing submission is evidence of a validated discover-and-patch sequence under that test. It is not automatically evidence that the originally catalogued defect was eliminated. The methodology also warns against directly comparing its subset and harness with the original researchers' leaderboard.

The original CyberGym-E2E project explains why this distinction is defensible. Its dataset contains 920 vulnerabilities across 139 open-source projects, and an agent can find a different real flaw from the one used to construct a task. The researchers regard fixing that alternative flaw as useful work. Their end-to-end setting withholds the reference vulnerability information; their patch-only setting supplies the reference crash input and log. Those are different assignments. A model that excels once a defect has been localized has not necessarily demonstrated the same ability to discover one unaided.

CyberGym's authors also describe shallow patches that suppress a crash at the reported location while leaving the underlying defect unresolved. Execution-based grading can count a patch that passes the checks, and the researchers recommend treating generated patches as candidates for further review. This is not a reason to dismiss automated repair. It is a reason to retain a precise vocabulary for its output. For a maintainer trying to close a particular security issue, a different valid fix and a complete repair of the target issue belong in different buckets.

The other components of the Cyber Index ask different questions again. Artificial Analysis's launch account describes CWE-Bench-AA as 120 private, held-out tasks covering ten OWASP categories. A deterministic verifier requires both that the attack no longer works and that legitimate behavior remains functional. There is no partial credit for a persuasive explanation. That paired requirement is important: disabling the affected feature may stop a demonstration of the flaw without constituting an acceptable repair. The benchmark attempts to make preservation of useful behavior part of success.

DeepsecBench-AA instead reviews scanner-flagged application code against expert-verified findings. Artificial Analysis uses an LLM judge to assess reports and match them to that reference set. Its F2 metric favors recall over precision, while duplicate reports count against precision. The launch explanation frames that weighting as a tradeoff between missed flaws and the time spent triaging false alarms. Consequently, the composite blends execution-verified repairs with judged discovery quality. Equal weighting is a declared design choice, not proof that every security team experiences those workloads in equal proportions.

Cybench supplies another kind of evidence. Its developers assembled 40 professional-level capture-the-flag tasks from four competitions, with intermediate subtasks available for more granular evaluation. The framework distinguishes unguided completion from subtask-guided completion and from progress through those smaller steps. Tasks include starter material and evaluators, with agents interacting inside a container and, where required, with task servers. The leaderboard's footnotes also identify results obtained on different subsets and under different reporting conventions. Those details should accompany any comparison built around a single percentage.

A competition task tests whether an agent can reach a defined answer in a constructed challenge. It can expose meaningful differences in technical problem-solving, but it is not interchangeable with maintaining an application after a repair. Mistral's reported Cybench result therefore complements the source-auditing evidence rather than converting it into one universal security score. The right follow-up is to identify the evaluation setting and inspect what the successful runs actually accomplished, not to silently equate differently assembled test suites.

Attack resistance is a separate claim. Mistral reports that Large 4 resisted 93.3% of attacks on Lakera's public B3 benchmark. The dataset page linked from that claim explicitly describes a weaker, lower-quality public version and says the strongest collected attacks were excluded. Its 210 attacks appear across three defense levels, producing 630 entries. The examples are contextual prompt-injection tests tied to snapshots of agent activity. The dataset describes ten applications and configurations ranging from an undefended system prompt to stronger prompts and a self-judging arrangement.

This qualification should travel with the score. The linked dataset does not, by itself, tell a reader which configuration Mistral used or demonstrate resistance to the strongest private attacks. Nor does evaluating a snapshot establish the security of every step in a deployed agent. A strong result on the public material can still be informative. What it cannot justify is dropping the dataset's selection rules and presenting the percentage as the chance that a real application will withstand an arbitrary attacker.

Mistral also cites AgentHarm among the sources of malicious cyber prompts used in its refusal evaluation. AgentHarm's documentation provides both harmful and benign evaluation settings, and describes semantic judging for refusals and parts of the grading. Its authors explicitly warn that the resulting grades are not an overall measure of agent safety. That warning is especially useful here. Refusing a harmful direct request and ignoring an adversarial instruction embedded in material an agent encounters are different behaviors. Neither should be inferred solely from success at finding a software defect. The benign comparison matters too: blocking useful work indiscriminately can make an apparently cautious system unsuitable for its intended job.

For a team considering the preview, a productive trial would preserve those distinctions in its records. Count an authorized task that was refused separately from an attempted task that failed verification. Record a valid new finding separately from closure of the issue the team asked to resolve. Ask maintainers to judge whether accepted patches repair the cause and preserve behavior, and include their review effort in the assessment. These are proposed acceptance criteria, not claims about measured Large 4 performance. They prevent an evaluation from declaring victory while leaving the original engineering obligation open.

The prospect of downloadable weights makes this more consequential, not less. If that release happens as planned, organizations could gain more control over the rules under which the model operates. Control would also leave them responsible for deciding which requests and tool actions are authorized. Mistral is offering a serious candidate for defensive experimentation. The strongest case for trying it is the possibility of completing useful, verifiable work that a current service will not undertake. The decision to trust it should rest on the resulting findings and repairs, with refusals and attack resistance examined separately.

LaunchPad positionEvaluate authorized task completion, target-issue repair and attack resistance separately before turning a benchmark result into operational trust.