Observed Signal · Sep 18, 2026 · Technical Release · Source: SemiAnalysis · Impact: 4/5 · Sentiment: Positive
SemiAnalysis Tests Engram Offloading to DRAM and SSD
SemiAnalysis analyzes the Engram architecture, a model design that extends standard token embeddings with learned multi-token lookups, allowing for efficient parameter offloading to DRAM or SSD. This reduces HBM capacity requirements for models like DeepSeek-V4.1-Flash. Their experiments show that offloading Engram tables to DRAM can improve performance per dollar, while SSD offloading is currently not economically viable due to overhead. The article also benchmarks inference performance across NVIDIA and AMD GPUs, highlighting NVIDIA's CUDA moat and AMD's slower software support. The analysis includes findings on model behavior, such as gate scan results and ablation studies, and discusses the implications for HBM demand and model architecture innovation.
Technical release of a novel model architecture optimization that could reduce HBM demand and impact inference costs, relevant to AI/AdTech infrastructure.
Track SemiAnalysis Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your Polaris7 workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- SemiAnalysis tested offloading Engram tables to DRAM and SSD for DeepSeek-V4.1-Flash.
- Offloading to DRAM improved performance per dollar, reducing needed HBM capacity.
- SSD offloading was not economically viable, with DRAM delivering 121 million tokens per dollar versus 52 million for SSD.
- NVIDIA vLLM worked out of the box for DeepSeekV4.1 Flash, while AMD's vLLM support was delayed.
- AMD MI355X performance per dollar remains 2-4x worse compared to NVIDIA B200.
Connected Companies & Entities
9 Entities mapped“SemiAnalysis analyzes the Engram architecture and provides benchmarks.”
“NVIDIA vLLM worked out of the box for DeepSeekV4.1 Flash across all 6 SKUs.”
“AMD's vLLM support for DeepSeekV4.1 Flash was delayed, and MI355X performance lags.”
“DeepSeek did not release the original Engram models; SemiAnalysis replicated the setup.”
“SemiAnalysis benchmarks are validated by Google Cloud among others.”
“SemiAnalysis benchmarks are validated by Microsoft Azure among others.”
“SemiAnalysis benchmarks are supported by Meta.”
“SemiAnalysis benchmarks are supported by major labs like OpenAI.”
“SemiAnalysis benchmarks are supported by the ML community including Hugging Face.”
Ontology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Processing‑in‑Memory Could Shift LLM Decode from GPUs
The article argues that the main bottleneck for LLM inference—especially autoregressive token generation (Decode)—is memory bandwidth and interconnect, not raw GPU compute. Processing‑in‑Memory (PIM) moves computation into the memory stack to eliminate data movement; commercial and announced products include SK Hynix's AiM (shipping), Samsung's LPDDR5X‑PIM (announced Feb 2026), and HBM4 with integrated logic dies (mass production slated Feb 2026). Recent arXiv papers (HPIM, PAM) propose PIM architectures that place different compute capabilities across the memory hierarchy to accelerate Decode (GEMV). PIM offers significant advantages for Decode but is not competitive with GPUs for training or batched Prefill (GEMM). Barriers include immature programming models, higher memory cost, and limited consumer availability (estimated 3–5 years). Datacenter adoption could lower cloud inference costs and reallocate parts of the inference stack toward memory vendors.
Memory Costs Surge as AI Infrastructure Complexity Grows
TechCrunch reports that memory (DRAM and cache management) is becoming a central cost and operational factor for running AI models. DRAM prices have risen roughly sevenfold in the past year, and companies are increasingly focused on orchestrating memory so the right data is available to agents at the right time. Anthropic’s prompt-caching pricing (with 5-minute and 1-hour cache windows) illustrates commercial trade-offs: cached reads are much cheaper, but adding data can evict other cached items. Semiconductor analyst Doug O’Laughlin and Val Bercovici (Weka) discuss hardware choices (DRAM vs HBM) and higher-level orchestration. Startups such as Tensormesh are addressing cache optimization. Better memory orchestration and more efficient models can materially reduce token use and inference costs, improving the economics of AI applications.
High‑Bandwidth Flash Emerges as AI Memory Option
The report analyzes High Bandwidth Flash (HBF), a stacked-NAND packaging approach that mimics HBM stacking (TSVs + bonded controller/CBA) to deliver very high read bandwidth ( ~1.6 TB/s) while offering substantially more capacity (Sandisk states ~512 GB per stack). HBF trades higher latency and lower write endurance for much greater capacity-per-stack versus HBM, making it a candidate for storing model weights for inference decode workloads. SanDisk expects memory samples in H2 2026 and AI inference devices using HBF in early 2027. Sandisk and SK Hynix are collaborating on stacking and began an OCP standardization effort in February 2026. The piece outlines supply-chain implications, comparative power/cost metrics versus HBM4, and competitive players including Sandisk, SK Hynix, Samsung and YMTC.
Receive all future signals for SemiAnalysis & competitors in your briefing
Never miss material M&A deals, leadership changes, account wins, or product rollouts. Polaris7 monitors 20,000+ primary sources 24/7, delivering verified telemetry directly into your executive briefing.