Observed Signal · Jun 23, 2026 · Technical Guide · Source: DEV Community · Impact: 2/5 · Sentiment: Neutral
How I Built a Reliable Webhook Delivery System
A developer describes building a production-grade webhook delivery system using FastAPI, PostgreSQL and Redis to solve common reliability issues. The post explains design changes: make delivery asynchronous (return 202 Accepted), add a watchdog to requeue stale IN_FLIGHT jobs, implement exponential backoff retries via a Redis sorted-set delay queue, apply a per-subscription circuit breaker (5 failures, 60s cooldown) and sign payloads with per-subscription HMAC‑SHA256 verified with hmac.compare_digest. Observability is provided with Prometheus and Grafana. The author reports achieving 99.9% delivery reliability across 10,000+ daily webhooks and promises a deeper technical deep-dive later.
Practical engineering guidance on reliable webhook delivery and observability patterns (backoff, circuit breaker, watchdog, HMAC) that benefit engineering teams building integrations, but it is not platform-level or industry-shifting news.
Track PostgreSQL Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your free Explorer workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- Author used FastAPI, PostgreSQL and Redis to implement the webhook system.
- Delivery was made asynchronous by returning 202 Accepted and persisting events for later delivery.
- A watchdog runs every 30 seconds to requeue stale IN_FLIGHT jobs.
- Retries use exponential backoff (2s → 32s) with a Redis sorted-set delay queue; max 5 attempts.
- A per-subscription circuit breaker trips after 5 consecutive failures with a 60s cooldown; payloads are signed with per-subscription HMAC‑SHA256 and verified using hmac.compare_digest.
- Observed result: 99.9% delivery reliability across 10,000+ daily webhooks with Prometheus + Grafana visibility.
Connected Companies & Entities
5 Entities mappedOntology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Developer Builds Reliable Webhook Relay Layer
The article explains a common reliability problem with webhooks: many providers use a ‘fire-and-forget’ model where a single POST (or limited retries) can silently drop events when receivers are down or overloaded. The author proposes a reliability layer — a relay that immediately acknowledges sender requests, stores raw payloads, and asynchronously delivers to downstream endpoints with logged attempts and exponential-backoff retries. They built an open, self-hostable project called Webhook Relay Layer using FastAPI for ingestion, Celery + Redis for queuing and retries, PostgreSQL for durable storage, and a dashboard for monitoring and manual retries. The post outlines the design principles, lists implementation details, and previews future posts on retry engine internals, webhook security, high-throughput ingestion, and production deployment.
Making Webhook Delivery Resilient with a Delivery Layer
The article explains how direct webhook integrations fail when receivers are offline and describes a resilient architecture that inserts a persistent delivery layer (Adal) between webhook providers and receivers. Adal accepts and stores incoming webhooks at a permanent HTTPS endpoint, provides delivery history and retries, and can forward events via Direct HTTP to public endpoints or via an outbound WebSocket tunnel (Adal CLI) to private/local services. If automatic retries are exhausted, stored events remain available for manual redelivery during their retention period. The article also covers idempotent receiver design, signature validation, encrypted storage, and the importance of complying with network and security policies when using an intermediary.
Bulletproof Webhook Ingestion for Ruby on Rails
This technical tutorial describes a resilient webhook ingestion pattern for Ruby on Rails applications (Rails 7/8). It recommends immediate acknowledgement of incoming webhooks and asynchronous processing: verify signatures, persist the raw payload to an inbound webhooks table, enqueue a background job, and return 200 OK. The article provides concrete Rails examples including an InboundWebhook model with enum statuses (pending, processing, completed, failed), a lean controller that verifies Stripe signatures and enqueues ProcessWebhookJob, and a background job that retries on deadlocks, updates lifecycle status, logs failures, and re-raises errors for monitoring (Sentry/Honeybadger). It also discusses idempotency strategies (database uniqueness constraints or Redis locks using provider event IDs) and suggests Solid Queue or Sidekiq for background execution. The guide emphasizes decoupling storage from execution to improve throughput, reliability, and recoverability.
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.