UK AI Safety Institute publishes first LLM test results

The UK Government's Institute for AI Safety (AISI) has published its first evaluation results, screening five leading large language models for cyber, chemical, and biological agent capabilities and the effectiveness of developer safety safeguards. The models were given color pseudonyms, and their identities were not disclosed. Findings show several LLMs demonstrated PhD-level knowledge in chemistry and biology, completed simple high-school-level cyber challenges but struggled with university-level ones, and two completed short-horizon agent tasks while failing more complex ones. Critically, all tested LLMs remain highly vulnerable to basic jailbreaks. The testing focuses on national security risks rather than short-term issues like bias. Saqib Bhatti MP indicated legislation will come 'eventually,' informed by testing. AISI also announced plans to establish a San Francisco base and collaborate with its Canadian counterpart. Results are expected to be discussed at the upcoming Seoul Summit.