SigNoz

Deploying SigNoz with ClickHouse v25 and OTel Gotchas

A developer describes troubleshooting and deployment patterns for self-hosting SigNoz in 2026, focusing on ClickHouse v25 configuration changes, OpenTelemetry Collector incompatibilities, and Docker network isolation. Key recommendations include using config.d/users.d for ClickHouse overrides (instead of replacing the main config), enforcing a deterministic docker-compose boot sequence with one-shot migrator containers (signoz_init_clickhouse and signoz_telemetrystore_migrator), updating exporter names and image tags for the OTel collector (use signozclickhousemetrics and include the 'v' prefix), and relying on internal Docker network isolation rather than per-component passwords to simplify connectivity. The post emphasizes that maintaining self-hosted observability infrastructure requires ongoing engineering effort.

Observability Reveals Why Sites 'Go Down'

The author built a fake exam-results site as a hackathon project to intentionally inject outages and observe them with the open-source observability tool SigNoz. By triggering four failure modes—database latency, cache (Redis) failure, a bad deploy that throws 500s, and traffic spikes—the author demonstrates that “server down” is not a single failure but distinct fingerprints visible in traces and metrics (fat slow spans, flatlined cache-hit rates, error-rate cliffs, rising RPS with latency). The project includes a Mission Control panel with toggles to inject incidents and a load generator, plus instructions to install SigNoz via a Foundry installer. The piece argues that pre-wired telemetry and alerts produce fast, specific diagnoses during incidents rather than vague “try later” responses.

SigNoz Hackathon: AI Agent with Failover and Diagnostics

A developer published a DEV.to post describing a hackathon project that implements an AI agent system which automatically fails over to a backup model when the primary model fails. The implementation sends observability data to SigNoz to track requests, monitor performance, and detect failures. The author also created a diagnostic agent that reads the monitoring data and explains failure reasons in plain language. During testing the system detected a real AI provider outage and automatically identified the root cause. The post was published on DEV on 2026-07-26.