AI Infrastructure Market: SemiAnalysis Tests Engram Offloading to DRAM and SSD

SemiAnalysis analyzes the Engram architecture, a model design that extends standard token embeddings with learned multi-token lookups, allowing for efficient parameter offloading to DRAM or SSD. This reduces HBM capacity requirements for models like DeepSeek-V4.1-Flash. Their experiments show that offloading Engram tables to DRAM can improve performance per dollar, while SSD offloading is currently not economically viable due to overhead. The article also benchmarks inference performance across NVIDIA and AMD GPUs, highlighting NVIDIA's CUDA moat and AMD's slower software support. The analysis includes findings on model behavior, such as gate scan results and ablation studies, and discusses the implications for HBM demand and model architecture innovation.