Cisco vs Ubuntu

Small Language Models: When Smaller Is Better

This MLPills newsletter issue explains small language models (SLMs) — compact AI models designed to run under resource constraints such as limited memory, power, and latency budgets. It clarifies that 'small' is a comparative concept rather than a specific parameter count, and distinguishes SLMs from quantized frontier models and distillation. The article covers four main routes to building SLMs: curated data training, distillation, pruning, and quantization, and highlights examples including Microsoft's Phi-4 family, Google's Gemma 3n and Gemma 4 edge models, Hugging Face's SmolLM3, and Cisco's Antares vulnerability-localization models. It describes ideal use cases like classification, entity extraction, and tool selection, and recommends a layered architecture using deterministic code, small models, large models, and human oversight. The piece also cautions about evaluation, over-pruning, and privacy limitations of local inference.

• A 4B parameter model at 4-bit precision needs roughly 2 GB of memory; the same model at 16-bit needs roughly 8 GB.
• Microsoft's Phi-4 is a 14B parameter model, with a 3.8B Phi-4-Mini sibling and a Phi-4-Multimodal variant.
• Google's Gemma 3n models used per-layer embeddings, KV cache sharing, and activation quantization to reduce memory footprints.