Observed Signal · Jul 17, 2026 · Technical Release · Source: DEV Community · Impact: 2/5 · Sentiment: Positive
Porting Gemma-4 Family to AWS Inferentia2
The author ported the Google Gemma‑4 family (E2B, E4B, 12B, 31B and 26B‑A4B MoE) to run on AWS Inferentia2, documenting per‑model constraints, compile recipes, and a recurring correctness bug. Each model was made to decode token‑for‑token with a CPU fp32 reference and artifacts (Docker Hub images and Hugging Face repos under xbill9) were published. Porting evolved from single‑core tracing (torch_neuronx.trace) to hand‑driven tensor‑parallel aliases and finally an NxD ModelBuilder single‑rank compile required for large dense (31B) and sparse MoE (26B‑A4B) configs. The google/gemma‑4‑12B‑it unified multimodal checkpoint required three device fixes—correct global attention sharding for nkv=1, drop on‑device logits softcap, and force eager attention to avoid SBUF overflow—yielding token‑for‑token equality with ~101 ms prefill and ~12 GB bf16 per rank on a single inf2.8xlarge (TP=2).
Practical, reproducible guidance for running large Gemma-4 family models on AWS Inferentia2 and a matured compile recipe (single-rank ModelBuilder) that enables dense 31B and sparse MoE deployments; useful to engineers deploying LLM inference but not industry-shifting policy or platform news.
Track Hugging Face Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your free Explorer workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- Ported the Gemma‑4 family (E2B, E4B, 12B, 31B, 26B‑A4B) to AWS Inferentia2; published Docker Hub images and Hugging Face repos under xbill9.
- All five models decode token‑for‑token identically to a CPU fp32 reference in the author's runs.
- Compilation recipes progressed: torch_neuronx.trace → tp_alias → NxD ModelBuilder; ModelBuilder was required to compile 31B and the 26B‑A4B MoE.
- google/gemma‑4‑12B‑it required three device fixes (correct nkv=1 global attention sharding, drop on‑device logits softcap, force eager attention to avoid SBUF overflow) and runs on a single inf2.8xlarge (TP=2) with ~101 ms prefill and ~12 GB bf16 per rank.
- 26B‑A4B is a MoE with 128 experts and top‑8 routing; it activates ~4B parameters but all 128 experts (~49 GB) remain resident, requiring full‑residency memory budgeting.
Connected Companies & Entities
5 Entities mapped“Hugging Face:`xbill9/gemma-4-{E2B,E4B,31B,26B-A4B}-it-inferentia2` (+ `-int8`)...”
“I ported the whole Gemma-4 family — E2B, E4B, 12B, 31B, and the 26B-A4B MoE — to run on AWS Inferentia2....”
“Docker Hub:`xbill9/gemma4-optb{,-e4b,-31b,-26b,-26b-int8}`...”
“Written with AI assistance (the ports, the debugging, and this write-up were done across Claude Code sessions)...”
Ontology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Gemma 4 in Pure JAX: TPU-to-GPU Port Lessons
A developer ported a Gemma 4 E2B checkpoint to a single pure-JAX codebase and ran it on Cloud TPU v5e/v6e and an NVIDIA T4G (on an AWS Graviton2 host). Most model code, compilation cache behavior, and static-shape discipline transferred unchanged, but two hardware-dependent issues surfaced: a fused W4A16 kernel written in Pallas (tiled for TPU VMEM) cannot run on GPUs due to much smaller shared-memory limits, and compute-dtype selection must be detected at runtime (float16 vs bfloat16) to avoid hidden conversion costs. The article documents Gemma 4’s four model irregularities, a KV-ring padding bug that produced silent token loops, measured decode throughput (13.10 tok/s on T4G), and profiling that shows unexpected conversion overhead on a Turing GPU.
Gemma 4 Enables Local Multimodal, Long-Context Workflows
A developer reports replacing fragmented OCR + RAG stacks with local Gemma 4 models, claiming the model family makes coherent, private, on-device multimodal intelligence practical on consumer hardware. Using the Ollama Python SDK and local inference, the author says Gemma 4’s 26B MoE and 31B Dense variants reason over pixel layouts directly (no separate OCR), achieving ~94% extraction accuracy on complex receipts with simple image preprocessing on an M1 MacBook Pro (16GB). Gemma 4’s native 128K context window allowed the author to ingest a continuous 115K-token log stream and trace a multi-month causal chain in ~70 seconds, highlighting temporal coherence benefits over chunked RAG. The post lists recommended model/context budgets, notes limits (very degraded inputs, real-time latency, knowledge cutoffs), and cites Gemma developer docs and Ollama resources. Publication date: 2026-05-21.
Run Gemma 4 26B on GTX 1080 with llama.cpp
A developer how-to demonstrating how to run Google’s Gemma 4 26B‑A4B Mixture‑of‑Experts model locally on an 8 GiB NVIDIA GeForce GTX 1080 using an enhanced llama.cpp fork (AtomicBot-ai/atomic-llama-cpp-turboquant). The guide details system setup (driver pinning, CUDA nvcc, gcc-14 workaround, glibc patch), building the fork with CUDA, downloading the main GGUF and MTP assistant head, and tuning offload parameters. Key optimisations include keeping most MoE expert weights in host RAM (streamed over PCIe), using RotorQuant/TurboQuant KV cache to enable 128k context, and forcing the assistant embedding table onto the GPU with --override-tensor-draft to enable effective MTP speculative decoding. The author reports ~24.5 tokens/sec at 128k context and describes the memory/PCIe tradeoffs and the final recommended command-line configuration.
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.