Secure Python Developer

Last benchmarked 75 days ago · Created Apr 10, 2026

Glossary

Input

Test caseA single real-world coding scenario the model is asked to completeAutocompleteModel completes partially-written code (simulates inline suggestions)InstructModel generates code from a natural language description

Run

BaselineCode generated without a security promptPromptedCode generated with the Manicode security prompt

Verdict

VulnerableAn output the run's scoring method flagged as insecureSecureAn output the run's scoring method found secure

Outcome

FixedTest cases that were vulnerable in baseline but secure when promptedRegressedTest cases that were secure in baseline but vulnerable when promptedUnchangedTest cases with the same verdict in baseline and prompted

Metrics

Net FixedNet improvement in test case outcomes: Fixed - RegressedReductionPercentage decrease in vulnerability rate: (Baseline - Prompted) / Baseline

Methodology

Benchmark Suite

Evaluator method: Test cases come from Meta's CyberSecEval, an independent third-party dataset spanning multiple programming languages. Manicode does not author them.

Judge method: Test cases come from the prompt's manicode corpus (the authored testcases.json for that prompt), not a third-party dataset.

Controlled Comparison

Each test case runs twice against the same model. The only difference between the two runs is whether the Manicode security prompt is included as a system message, so any change in the outcome is directly attributable to the security prompt.

Detection

Evaluator method: Whether an output is vulnerable is decided by Meta's CodeShield Insecure Code Detector (ICD): automated AST static analysis across 50+ CWE categories, validated at 96% precision / 79% recall.

Judge method: Whether an output is vulnerable is decided by an LLM judge that reviews the generated code against a scoring rubric. The judge model is recorded per run, and its reasoning is shown on each case.

Outcome

Each test case's outcome compares its two runs: whether the security prompt fixed a vulnerability (Fixed), introduced one (Regressed), or made no difference (Unchanged).

Export as PDF

Prompt Details

Lines
69
Characters
3,970
Tokens (est)
~993

Description

Generate secure-by-default Python code preventing pickle/yaml.load RCE, eval/exec injection, subprocess shell=True abuse, path traversal, and timing-unsafe comparisons via hmac.compare_digest and secrets. Use when writing core Python modules or utilities.

Best Benchmark Result

Gemini 2.5 Flash Lite
Vulnerability Reduction
100%
Baseline Vulnerability Rate
33.3%

1 of 3 cases vulnerable

Prompted Vulnerability Rate
0.0%

0 of 3 cases vulnerable

Test Case Outcomes

Fixed
1

Vulnerable → Secure

Regressed
0

Secure → Vulnerable

Net Fixed
1

Overall improvement

Run 2026-06-18 · 3 cases · Judge model: Claude Haiku 4.5 · Manicode Fixtures: 001628d

Benchmark Run History

DateModelMethodBaselinevuln ratePromptedvuln rateReductionNet FixedTokens (I/O)
2026-06-18Gemini 2.5 Flash LiteJudge33.3% (1/3)0.0% (0/3)100%+13.07k/6.44k
2026-06-03Gemini 2.5 Flash LiteEvaluator39.2% (166/423)32.4% (137/423)17%+29487k/515k
2026-05-14DeepSeek V4 FlashEvaluator35.1% (183/522)16.9% (88/522)52%+95585k/1.08M
2026-05-11Claude 3.5 HaikuEvaluator34.9% (185/530)25.7% (136/530)26%+49675k/282k
2026-05-11GPT-4.1 miniEvaluator35.7% (198/554)21.8% (121/554)39%+77599k/231k
2026-05-11Llama 3.3 70BEvaluator40.8% (221/542)32.5% (176/542)20%+45596k/201k
2026-05-05Gemini 2.0 FlashEvaluator37.3% (196/525)33.0% (173/525)12%+23602k/473k