"Server is up." That is the message your monitoring tool sends you every five minutes. Green checkmarks across the board. Everything looks healthy. Meanwhile, your users are experiencing 4-second page loads, intermittent 502 errors on checkout, and a memory leak that will bring down your primary application server in roughly 36 hours.
Most companies only discover problems when users complain. A support ticket comes in, someone checks the dashboard, and sure enough — everything is green. Because the monitoring is checking the wrong things. An HTTP 200 response on your homepage tells you almost nothing about the health of your application. Real monitoring means knowing about issues 10 minutes before they affect anyone.
This is the difference between monitoring and observability. And most setups are stuck firmly in the first category.
Why Monitoring Fails
The typical monitoring setup looks like this: a third-party service pings your website every 5 minutes and checks for an HTTP 200 response. If it gets one, everything is "up." If it does not, you get an email. This approach fails in almost every meaningful scenario.
Only checking HTTP status codes. An HTTP 200 tells you your web server responded. It does not tell you that response took 8 seconds, that 15% of requests are returning 500 errors, or that your database connection pool is at 95% capacity. Status codes are binary. Infrastructure health is a spectrum.
No application-level health checks. Your web server can return 200 while your application is in a degraded state. Can it connect to the database? Is Redis responding? Is the message queue processing? A proper health check endpoint tests actual functionality, not just "is the process running."
Alert fatigue from noisy thresholds. When you set a static threshold like "CPU > 80% = alert," you will get woken up at 3 AM for a scheduled backup job that runs every night. After two weeks, you start ignoring alerts. After a month, you mute the channel. After two months, a real incident goes unnoticed for 45 minutes because nobody reads the alerts anymore.
No baseline metrics. Is 70% CPU usage normal for your application? You cannot answer that question without historical baselines. Without baselines, every threshold is arbitrary, and every alert is either too sensitive or not sensitive enough.
Five-minute check intervals. A lot can happen in 5 minutes. Your application can crash, restart, and crash again — and your monitoring will show 100% uptime because both checks happened during the brief windows when the process was running. Thirty-second intervals are the minimum for any production system.
No correlation between metrics. CPU spike + memory increase + disk I/O surge happening simultaneously tells a story. Individually, each might not trigger an alert. Together, they indicate a runaway process. Without correlation, you are looking at isolated data points instead of understanding system behavior.
Common Mistakes
Relying solely on uptime monitoring. Pingdom-style checks answer one question: "Is the server responding?" That is roughly 5% of what you need to know. Uptime monitoring is table stakes, not a strategy. As we covered in our guide on preventing website downtime, the real work happens in the layers below the surface — and that requires deeper instrumentation.
Not monitoring disk fill rate. This is the one that catches everyone. Disk usage at 60% seems fine. But if it was at 40% yesterday and 50% this morning, you have a problem that will become critical in about 20 hours. Exponential log growth, uncleaned temp files, or database transaction logs can fill a disk faster than you expect. Monitor the rate of change, not the absolute value.
Ignoring memory leak patterns. A process that consumes 500MB of RAM is not a problem. A process that consumed 200MB yesterday, 350MB this morning, and 500MB now is a memory leak that will trigger the OOM killer in roughly 18 hours. Without trend analysis, these patterns are invisible.
No database query time monitoring. Your application might respond in 200ms today. But if a specific query is gradually slowing from 50ms to 500ms due to table growth or index degradation, you will not notice until it crosses the threshold that makes your entire application feel sluggish. Query time percentiles (p50, p95, p99) are essential metrics.
No log aggregation. Logs scattered across 6 servers, each with different rotation policies, are effectively invisible. When an incident happens, you do not have time to SSH into each server and grep through files. Centralized, structured logging is not optional for any production system running more than one instance.
Same alerting thresholds day and night. Your traffic patterns are not constant. 80% CPU at 2 PM during a sale is expected. 80% CPU at 3 AM when traffic is near zero is a serious problem. Time-aware thresholds reduce noise and catch anomalies that static thresholds miss.
What Actually Works
Effective monitoring operates on multiple layers, each providing different levels of insight.
Multi-layer monitoring
You need three distinct layers working together:
- Infrastructure metrics: CPU, memory, disk, network, process counts. These are your foundation. Tools like Prometheus node_exporter or Datadog agent collect these at 10-15 second intervals.
- Application metrics: Request latency (p50/p95/p99), error rates, throughput, queue depths, connection pool usage, cache hit rates. These come from your application code or APM tools.
- Business metrics: Orders per minute, signups per hour, revenue per hour. A sudden drop in orders with no infrastructure alerts means something is broken that your technical monitoring is not catching — a broken payment flow, a JavaScript error on checkout, a third-party API timeout.
Health check endpoints that test actual functionality
A proper /health endpoint does not just return {"status": "ok"}. It actively tests every dependency:
GET /health
{
"status": "degraded",
"checks": {
"database": {"status": "healthy", "latency_ms": 3},
"redis": {"status": "healthy", "latency_ms": 1},
"queue": {"status": "unhealthy", "error": "connection refused"},
"disk": {"status": "warning", "free_gb": 4.2, "fill_rate_gb_per_hour": 0.8}
},
"version": "2.4.1",
"uptime_seconds": 482910
}This endpoint runs actual queries against each dependency. Load balancers use it to route traffic away from unhealthy nodes. Monitoring systems use it to understand the state of each component independently.
Trend-based alerting
This is the single biggest improvement most teams can make. Instead of alerting on absolute values, alert on rates of change:
- Disk filling at 2% per hour → alert (you have ~12 hours to respond, but you know now)
- Memory growing 100MB per hour with no corresponding traffic increase → alert
- Database connection pool usage trending upward over 6 hours → alert
- Error rate increased 3x compared to same time yesterday → alert
This approach catches slow-burn problems days before they become incidents. It is also how you improve server performance proactively — by detecting degradation trends rather than reacting to failures. For more on this, see our detailed guide on improving server performance.
Distributed tracing
When a request touches 4 services, a load balancer, a cache layer, and a database, you need to trace its entire path. Tools like Jaeger or Zipkin attach a trace ID to each request and record timing at every hop. When a user reports a slow page, you can see exactly where the 3 seconds were spent: 50ms in the API gateway, 200ms in the auth service, 2,700ms waiting for a database query in the order service.
Centralized logging with structured data
Every log entry should be structured JSON, not free-text strings. Structured logs are searchable, filterable, and aggregatable:
{
"timestamp": "2026-03-28T14:23:01Z",
"level": "error",
"service": "payment-api",
"trace_id": "abc-123-def",
"user_id": 48291,
"error": "stripe_timeout",
"latency_ms": 30000,
"retry_count": 3
}Ship these to a centralized system (ELK stack, Loki, Datadog Logs). Set up saved searches for known error patterns. Create alerts on log volume anomalies — a sudden spike in error-level logs is often the first signal of an incident.
Dashboards that show rate of change
Your primary dashboard should not show "CPU is at 45%." It should show "CPU usage increased 12% in the last hour" or "Request latency p95 is trending 200ms higher than the 7-day average." Rate-of-change visualizations surface problems that absolute values hide. A Grafana dashboard with deriv() or rate() functions on your key metrics is significantly more useful than raw gauges.
Real-World Scenario
A SaaS company running a multi-tenant application came to us after a 2 AM outage that lasted 4 hours. Their monitoring dashboard showed 99.9% uptime for the month — because their check only verified that the homepage returned HTTP 200.
The actual problem: their PostgreSQL connection pool was slowly exhausting over 3 days. The application used a pool of 100 connections. A code change had introduced a connection leak — under certain error conditions, connections were not returned to the pool. Each leaked connection reduced available capacity by 1%.
Day 1: pool at 85% utilization (normal was 40-60%). No alert. Day 2: pool at 92%. Still no alert — their threshold was set at 95%. Day 3, 2 AM: pool at 100%. New requests queued, then timed out. Application returned 503 errors. The uptime check finally failed.
Four hours of downtime, significant revenue loss, and a very long incident review.
After implementing proper monitoring:
- Connection pool utilization tracked with 30-second granularity
- Trend-based alert: pool usage increasing more than 5% per day with no corresponding traffic increase
- Health check endpoint that reported pool utilization as a metric
- Auto-remediation runbook: when pool exceeds 80%, automatically restart the leaking service and page the on-call engineer
Result: the next time a similar leak occurred (different code path, same symptom), the trend alert fired 48 hours before exhaustion. The auto-remediation recycled the service during a low-traffic window. The engineering team fixed the leak the next business day. Zero user impact. Zero downtime.
The monitoring investment paid for itself in the first incident it prevented.
Building Your Monitoring Stack
If you are starting from scratch, here is a practical stack that covers 90% of use cases:
- Metrics collection: Prometheus + Grafana (open source) or Datadog (managed)
- Log aggregation: Loki + Grafana or ELK stack
- Tracing: Jaeger or Zipkin for distributed systems
- Uptime/synthetic monitoring: Uptime Robot or Checkly for external checks
- Alerting: PagerDuty or Opsgenie with escalation policies
- Incident management: Defined runbooks for every alert that pages someone
Start with infrastructure metrics and health check endpoints. Add application metrics next. Add business metrics last. Each layer builds on the one below it.
Stop Guessing, Start Observing
The difference between monitoring and observability is the difference between asking "Is it up?" and understanding "How is it behaving?" One gives you a binary answer. The other gives you the context to prevent incidents before they happen.
If you only find out about problems when users complain, your monitoring needs work. A proper observability setup is not a luxury — it is the foundation that makes everything else possible: reliable deployments, confident scaling, and engineers who sleep through the night.
Ready to build monitoring that actually works? Talk to our engineering team about implementing observability that catches problems before your users do.
