Grafana Labs

Push API alternative to Prometheus for small SaaS

The article advises beginners building a small SaaS custom metrics dashboard to prefer a push-style metrics API for app-level business measurements, while retaining Prometheus plus Grafana for Kubernetes, host monitoring, or where a richer infrastructure ecosystem is required. It explains the architectural differences between Prometheus's pull/scrape model and push APIs, highlights operational failure modes (duplication, staleness, delivery errors), and recommends starting by validating a single metric and the failure model before committing to a full stack. The author suggests running short bake-offs including Infrai (a REST-based option), Datadog, New Relic, Honeycomb, and emphasizes adding heartbeats and notification/deduplication components when needed.

Terraform Auto-Removes Manual AWS EC2 Tag

A developer tested an automatic remediation control loop that detects Terraform drift in AWS, classifies the severity, and runs a separate remediation pipeline to restore declared infrastructure. The pipeline uses a drift detector CodeBuild job that runs terraform plan, publishes structured changes via SNS, a Lambda that classifies changes (LOW/MEDIUM/HIGH) and, for eligible LOW non-deletion updates, starts a remediation CodeBuild job that runs terraform apply. The author adjusted the classifier to treat in-place updates on MEDIUM resources as LOW, separated detector and remediation IAM roles, ensured the correct Git commit is cloned for remediation, and observed that a manually added EC2 tag was removed when Terraform applied the Git-declared state.

Structured Handoffs Reduce Repeat On-Call Incidents

An engineer managing 40+ HIPAA-scoped production databases describes a repeat-incident problem caused by loss of operational context at rotation handoffs. The team introduced a 30-minute structured "readiness review" between outgoing and incoming on-call engineers with a fixed agenda: (1) what paged and why, (2) what changed in the platform, and (3) which runbooks are stale. Runbooks were moved into version control as "runbooks as code" with fields like last_validated and known_repeat to make staleness and repeat incidents visible. A simple SQL grouping to find frequently repeating alerts is provided. The changes correlated with roughly 35% fewer repeat incidents and ~30% lower MTTR; cross-team dependency handoffs remain an open challenge.