KaliBench: Open Models Top Out at 42% on Kali CLI Commands

A new benchmark accepted to the NeurIPS 2026 Evaluations and Datasets Track suggests that open-weight language models still struggle with a deceptively mundane skill: turning a security analyst's plain-English request into a command line that actually works. KaliBench, posted to arXiv on October 1, found that no open-weight model it tested exceeded 42% exact-command accuracy when given only the query. The result matters because agent products increasingly hand shell access to models, and a single misplaced flag can turn a useful command into a failed or harmful one.

What the researchers built

According to the arXiv abstract, the paper comes from Pengfei Li, Naufal Suryanto, Sicheng Zhang and Muzammal Naseer. KaliBench contains 8,504 pairs of natural-language queries and matching commands, spread across 1,642 tools that ship with Kali Linux, the Debian-based distribution widely used for penetration testing. The pairs are organized into 23 capability dimensions and five security phases: reconnaissance, vulnerability analysis, exploitation, post-exploitation and defensive analysis. After deduplication, 5,000 samples are reserved for evaluation and 3,504 for training.

The authors motivate the work by pointing out that cybersecurity operations depend on strict command-line interfaces, where small errors in syntax, flag-value pairing or argument order can invalidate a command. Most existing evaluations of tool use test whether a model can pick the right function from a schema. KaliBench instead targets the "schema-free" case, where the model must produce a raw command with exactly the right options.

How the dataset was filtered

The full paper describes a four-stage pipeline. Candidate query-command pairs were first generated with Qwen3-Max, grounded in the official Kali documentation. An LLM-based validation step then removed hallucinated or misspelled options, rejecting 53.8% of the initial candidates. Sandboxed terminal execution filtered a further 11.95% of commands that would not run, and human review refined another 4.9% where the command did not match the intent of the query.

That attrition rate is itself informative. More than half of the machine-generated candidates failed the first automated check, which is a reminder that even a strong model drafting commands from documentation produces a large share of invalid output. It also means the benchmark's quality depends heavily on its filtering stages rather than on generation alone.

Three ways of testing, one consistent weakness

The paper evaluates models in three modes that give progressively more help. In the unrestricted mode the model receives only the query. In the restricted mode it also sees 20 candidate tool names that include the correct one. In the hinted mode it receives the full documentation for the relevant tool. The authors tested 24 configurations of general-purpose and security-focused open-weight models.

The headline finding is the 42% ceiling on exact-command accuracy in the unrestricted setting. The authors conclude that building the arguments is the main bottleneck, rather than choosing the tool. Their reported numbers support that reading: choosing the right tool is comparatively easy, while getting optional flags and positional arguments exactly right is where accuracy falls away, and supplying documentation in the hinted mode lifts exact-match results sharply. In practical terms, a model that knows which tool to reach for may still not know its precise invocation, and access to current documentation appears to help far more than extra hints about which tool to use.

The authors also tested how sensitive models are to the wording of a query. Paraphrased queries changed unrestricted accuracy by only +0.61 percentage points, while messier, less polished inputs lowered it by 3.20 points. The drop is modest, but real analysts rarely write clean prompts, so the messy-query condition arguably reflects deployment better than the tidy one.

Training without running the commands

The second contribution is a training approach the authors call runtime-free verifiable rewards. Because each query is paired with a structured reference command, a model can be scored on the commands it proposes by comparing them to the reference, without executing anything. That matters for security work, where running arbitrary generated commands during reinforcement learning would require elaborate sandboxing and carries obvious risk.

Using supervised fine-tuning followed by reinforcement learning with verifiable rewards, the authors improved an 8-billion-parameter model to performance they describe as comparable to a 685-billion-parameter mixture-of-experts model. The project repository lists three released variants built on RedSage-Qwen3-8B-Ins (a supervised version, a reward-trained version and a combined one), along with the training and evaluation code, the 3,504-example training split, the 5,000-example test split and a set of 2,809 tool-documentation entries. The reported 79.2% average score for the best variant comes from the authors' own evaluation and has not, as far as we could determine, been independently reproduced.

Why it matters

For teams building agents that operate terminals, the benchmark isolates a failure mode that end-to-end agent tests can blur together. When an agent fails a multi-step penetration-testing task, it is hard to say whether it chose the wrong tool, built the wrong command or misread the output. By separating command generation from the rest, KaliBench lets developers attribute the failure. The finding that documentation access matters so much also supports a design pattern already common in agent systems: retrieve the tool's manual at call time instead of relying on what the model memorized in training.

The result also illustrates a wider point about model size. If a small model trained on verifiable, narrowly scoped data can approach a much larger general model on one task, specialized fine-tuning may be a cheaper route than scaling up for organizations that need local, auditable models. Security teams in particular often cannot send sensitive queries to external APIs, which makes capable small open-weight models more attractive.

Limits and what to watch

The authors acknowledge limitations in an appendix, including a focus on single-turn generation and reliance on documentation-grounded data rather than real-world security workflows. Since the commands were generated from documentation by a language model, the benchmark may reward familiarity with documented usage patterns more than the improvisation seen in live engagements. Exact-match scoring is also strict: a command that is functionally equivalent but written differently may be marked wrong, so the 42% figure should not be read as the share of commands that would fail in practice.

Several questions remain open. Results are reported for open-weight models only, so how proprietary frontier models compare is not addressed in the sources we reviewed. It is also unclear whether gains from runtime-free rewards will carry over to multi-turn settings, where a command's output shapes the next step. Follow-up work that adds multi-step tasks, or that compares exact-match scoring against execution-based success, would show how much of the benchmark's signal transfers to real tool-using agents. Until then, KaliBench is best read as a careful measurement of one narrow, important skill, and a useful template for building similar benchmarks for other command-line tools.