Observed Signal · Aug 6, 2026 · Technical Article · Source: DEV Community · Impact: 2/5 · Sentiment: Neutral
Why Most Teams Don't Need Real-Time Streaming
Lucas Ehara argues that many organizations overvalue millisecond-level real-time data pipelines and should instead consider simpler, cheaper batch or micro-batch approaches. The article recommends asking whether the business can act in milliseconds before adopting streaming, highlights streaming's operational complexity and higher cloud costs, and proposes hourly or 15-minute micro-batches as a pragmatic middle ground. The author advises starting with day‑lag (D-1) pipelines and only moving to streaming when measurable business impact justifies the added cost and engineering effort.
Practical guidance for data architecture affects engineering cost, operations, and analytics cadence across MarTech/AdTech stacks, but it's an opinion/operational piece rather than a major platform policy change or product launch.
Track DEV Community Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your free Explorer workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- Lucas Ehara published an article on 2026-08-06 arguing that many companies do not need real-time streaming.
- The author states the key question before implementing streaming is whether the business can act in milliseconds.
- The article claims streaming increases operational complexity (state management, out-of-order events, exactly-once semantics) and raises continuous cloud compute costs compared with batch.
- Micro-batch processing (e.g., hourly or every 15 minutes) is presented as a middle ground that delivers near-real-time experience with lower cost and complexity.
- Recommendation: start with D-1 (previous day's data) and only adopt streaming when business metrics demonstrate latency is causing real financial loss.
Connected Companies & Entities
6 Entities mapped“DEV Community — A space to discuss and keep up software development and manage your software career...”
“Powered by Algolia...”
“Neon is the official database partner of DEV...”
“Built on Forem — the open source software that powers DEV and other inclusive communities....”
“Watch AWS and AWS Partners live from the floor at Black Hat in Las Vegas as they discuss today's biggest cloud security challenges....”
Ontology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Combine Real-Time and Batch Indexing
The article argues that indexing pipelines should not force a binary choice between real-time and batch approaches. Real-time indexing is necessary when staleness causes measurable user harm (e.g., live dashboards, inventory), while batch indexing is better for high-throughput backfills, model refreshes, and rebuilds. The recommended pattern is a hybrid pipeline: stream processors feed a short-term real-time index while events are also persisted to object storage for scheduled batch processing; queries fan out to both layers with the real-time layer taking precedence. The author also emphasizes that the size of the "freshness window" is a product decision and that purpose-built streaming infrastructure can route events reliably to both layers without duplicating ingestion logic.
Stock-Market Lessons for Trustworthy Real-Time Pipelines
The author draws lessons from stock market data infrastructure to highlight design principles for correct real-time pipelines. Unlike many systems where latency is a comfort metric, market data treats latency as correctness: every subscriber must see every tick, in order, exactly once. Key architectural patterns include fan-out with per-consumer sequencing, partitioning by logical identity to preserve causal order, and making backpressure explicit so slow consumers don't accumulate invisible lag. The article includes a simple sequencing-gap-detection example and argues engineers should explicitly define behaviors for dropped messages, slow consumers, and out-of-order events before shipping. It notes that tools built for this space (e.g., Turboline) bake these tradeoffs into their architectures rather than leaving them to application developers.
Don't Default to WebSockets for Real-Time
This technical blog argues teams often choose WebSockets by default for "real-time" features when other protocols are more appropriate. It explains differences between WebSockets (persistent, bidirectional), Server-Sent Events (SSE; unidirectional HTTP text/event-stream with built-in reconnection via EventSource), and gRPC streaming (typed, binary service-to-service streams). The author shows how choosing the wrong protocol increases operational complexity and cost at scale—connection state, proxy/load-balancer timeouts, and resource limits—and recommends selecting WebSockets for conversations, SSE for broadcasts to browsers, and gRPC streaming for backend high-throughput typed pipelines.
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.