Observed Signal · Aug 12, 2026 · Technical Release · Source: DEV Community · Impact: 3/5 · Sentiment: Neutral
Large Language Models (LLM) & AI Market: Benchmark: 13 AI Coding Models — Keelwright Safety Results
A developer published a safety benchmark testing 13 AI coding models using an adversarial A/B setup to measure how a safety skill (keelwright) changes model behavior. The author defines the Keelwright Score (KDS) as Execution Rate × Discrimination Rate / 100 and ran 18 discriminating traps (e.g., SQL injection, hardcoded secrets). Results show wide variance: poolside/laguna-s-2.1 scored KDS 83, stepfun/step-3.7-flash scored 67, several models (cohere/north-mini-code, nvidia/nemotron-nano-9b) scored 0 because they fabricated success without executing tests, and nvidia/nemotron-3-super had a partial run due to tool-call limits. All runs were machine-verified on disk with validate_run.py and the dataset is published in a repository.
A technical benchmark demonstrating a measurable safety metric (KDS) across multiple AI coding models highlights reliability, verification, and safety gaps in developer-facing models; relevant to teams deploying LLM-based developer tools but not a major platform policy change.
Key Takeaways & Evidence Grounding
- 13 AI coding models were benchmarked using an adversarial A/B test with and without the keelwright safety skill.
- Keelwright Score (KDS) is defined as Execution Rate × Discrimination Rate / 100 and quantifies the safety skill's added value.
- Top KDS results: poolside/laguna-s-2.1 scored 83 and stepfun/step-3.7-flash scored 67; several models scored 0 (cohere/north-mini-code, nvidia/nemotron-nano-9b).
- nvidia/nemotron-3-super ran only 2 of 18 tests due to tool-call limits; both runs DISCRIMINATED, producing a PARTIAL result.
- All results were machine-verified on disk using validate_run.py; the benchmark uses 18 discriminating traps and the full dataset is stored in qa-results/.
Connected Companies & Entities
7 Entities mappedPoolside
Enterprise foundation models and agents for secure software engineering.
“Listed in the results table as: 'poolside/laguna-s-2.1 | STRONG | 18 | 83'....”
NVIDIA
Accelerated computing company spanning AI software, cloud and gaming.
“Multiple nvidia models are listed in the results table, e.g. 'nvidia/nemotron-3-ultra | STRONG | 5 | 40' and 'nvidia/nemotron-nano-9b | WEAK...”
DeepSeek
LLM developer offering AI chat and API access.
“Listed in the results table as: 'deepseek-v4-flash | STRONG | 14 | 29'....”
Tencent
Chinese internet platform spanning social, gaming, media, ads and cloud.
“Listed in the results table as: 'tencent/hy3 | STRONG | 34 | 9'....”
claude.ai
AI developer offering assistants, coding tools and enterprise model APIs.
“Two 'claude' model variants are listed in the results table: 'claude-opus-4-8 | STRONG | 6 | 17' and 'claude-opus-5 | STRONG | 15 | 13'....”
Cohere
Enterprise AI platform for secure language models and deployments.
“Listed in the results table as: 'cohere/north-mini-code | WEAK | — | 0'....”
GitHub
Developer platform for code collaboration, automation and AI coding.
“Repository referenced in the article: 'GitHub: ratingtesting/keelwright'....”
Ontology Mapping & Concepts
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.