Observed Signal · Jul 4, 2026 · Technical Release · Source: DEV Community · Impact: 3/5 · Sentiment: Positive
Local LLMs Reach Practical Usability
A developer revisits running large language models locally and reports that the landscape has shifted: newer Qwen models (dense and MoE variants) now run acceptably on consumer-class hardware with two RX6800 GPUs and 64 GB RAM. The author highlights Qwen3.6-27B (dense) for accuracy, Qwen3.6-35B-A3B (MoE) for speed, and Qwen-Coder-Next-80B (MoE) for coding tasks. Infrastructure improvements include llama.cpp's experimental router mode, ongoing work to persist attention checkpoints and context slots, and a personal fork that adds slot save/restore. The piece also compares harnesses (Hermes, Pi) and argues that capable local inference enables offline, libre-software experimentation without relying on commercial inference providers.
Locally runnable, capable LLMs reduce reliance on external inference providers and enable private, low-cost experimentation and on-premise usage—relevant for teams evaluating inference deployment, privacy, and cost trade-offs, but not a platform-level industry shift.
MARKET SIGNALS
Track GitHub Signals & Strategic Shifts
Polaris7 connects regulatory filings, primary sources, leadership changes, and M&A activity into structured intelligence. Set up your Polaris7 workspace to monitor relevant companies in your executive briefing.
- Verified primary sources
- Automated briefings
- Instant watchlist setup
Key Takeaways & Evidence Grounding
- Author tested local LLMs on a machine with two RX6800 GPUs (16 GB each) and 64 GB RAM.
- Qwen3.6-27B (dense) is reported as the most accurate local model in the author's tests and runs reasonably well on the described hardware.
- Qwen3.6-35B-A3B is a Mixture-of-Experts (MoE) variant that is much faster and suitable for agentic tasks that don't require deep reasoning.
- Qwen-Coder-Next-80B is an MoE model fine-tuned for coding; a REAM (weight-merge) variant reportedly matches full-model accuracy within benchmark error margins.
- llama.cpp now has an experimental "router mode" that loads/unloads models and saves slots to disk; the author created a GitHub fork adding slot save/restore and other unmerged PRs to handle newer models' checkpoint requirements.
Connected Companies & Entities
“I've created my own fork of llama.cpp with this and some other yet-unmerged PRs included, which you are welcome to try!”
“I can literally hear my LLMs working (apparently AMD GPUs are famous for their coil whine, which I consider a great feedback feature).”
“On one hand, this is more VRAM than any normal person can have with one GPU - unless you've got something specifically for AI, like an unifi...”
“On one hand, this is more VRAM than any normal person can have with one GPU - unless you've got something specifically for AI, like an unifi...”
Ontology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Qwen 3.5 Wins Local Benchmark Using llama.cpp
An independent Round 3 benchmark replaced Ollama with a direct llama.cpp server to measure local LLM performance more precisely. The author built a 12-task automated suite across five categories (coding, multi-file agentic coding, reasoning, tool use, and speed) and tested five models: Qwen 3.5, Gemma 4, Devstral, Codestral, and DeepSeek R1. Running on an NVIDIA RTX 5090 system, Qwen 3.5 swept the leaderboard—best coding, best agentic performance, and best single-model weighted score—reaching ~206.7 tokens/sec and a weighted overall score of 85.3. The migration reclaimed ~44 GB of disk from Ollama, enabled fine-grained inference flags (e.g., --reasoning-budget, --chat-template chatml), and highlighted Mixture-of-Experts (MoE) models’ throughput advantage for local deployment.
How to Run LLMs Locally
A hands-on tutorial published by Nilesh Raut on 2026-05-21 that explains how developers can run large language models locally to reduce API costs, improve privacy, enable offline use, and speed experimentation. The guide walks through installing Ollama, pulling models (examples: llama3, qwen2.5-coder:7b), running a local REPL, integrating local models into VS Code via Continue.dev and Cline, and hosting a ChatGPT‑like UI locally using Open WebUI in Docker (exposed on http://localhost:3000). The article lists recommended models (Qwen2.5 Coder, DeepSeek Coder, Llama 3, Phi, Mistral), minimum hardware (16GB RAM, SSD, NVIDIA GPU recommended) and common local use cases such as coding help, refactoring, documentation and small agents.
Running Qwen 3.6-35B-A3B on a 30GB Mixed GPU Rig
A developer describes upgrading an autonomous Minecraft AI (“Kiwi‑chan”) running on a heterogeneous 30GB consumer-GPU rig (RTX 3060, RTX 3050, GTX 1660 Ti, GTX 1660 Super) to use the MoE model Qwen 3.6-35B-A3B-Instruct. The author found Qwen’s Active 3 Billion design (≈3.5B parameters active per token) outperforms dense 30B+ models on mixed PCIe rigs by reducing compute and bandwidth bottlenecks. Combined with ik_llama.cpp’s Split Mode Graph tensor parallelism and 8-bit quantization of the KV cache (-ctk q8_0 -ctv q8_0), the setup yields higher throughput (~70–90 tokens/s vs ~20–35 tokens/s for a dense model) and a larger usable context window (32K–64K) while running on Ubuntu Server 24.04 LTS with CUDA 13.0. The post includes deployment flags and manual tensor-split recommendations for maximizing performance on mismatched GPUs.
WATCHLIST & BRIEFINGS
Receive future signals for GitHub & competitors in your briefing
Stay ahead of material M&A deals, leadership changes, account wins, and product rollouts. Polaris7 connects primary filings, company newsrooms, and trade reporting directly into your executive briefing.
- Regular primary-source checks
- Structured Watchlist briefings
- Preserves current context
Market Signal Monitoring
Follow affected companies in Polaris7 with regular source checks and structured Watchlist briefings.
Monitor GitHub
Follow affected companies in Polaris7 with regular source checks and structured Watchlist briefings.
