The Art of Observability: Stop Flying Blind

Junior Engineers add logging to debug their code. Senior Engineers add observability to debug the system.

There is a massive difference between "Monitoring" and "Observability."

• Monitoring: The dashboard turns red. "The API is returning 500s."
• Observability: The ability to ask new questions. "Is it returning 500s only for users in Europe on iOS devices?"

In this guide, we break down the three pillars you need to stop guessing and start knowing.

Here is what we'll cover:

• The 3 Pillars: Metrics, Logs, and Traces.
• The Golden Signals: What Google SREs measure.
• The cardinal sin of "Average Latency."
• How to build a dashboard that is actually useful.

1. The Three Pillars

To understand a distributed system, you need three different types of data.

Pillar 1: Metrics (The "What")

Metrics are cheap numbers over time. They tell you traffic patterns and health.

• Example: http_requests_total, cpu_usage_percent.
• Use Case: Triggers the alert. "CPU is at 99%!" (But it doesn't tell you what process is doing it).

Pillar 2: Logs (The "Why")

Logs are expensive text records. They tell you the story of a specific request.

• Example: [Error] User 123 failed payment: Insufficient Funds.
• Use Case: Debugging the root cause. You search logs after the metric alert fires.

Pillar 3: Traces (The "Where")

Traces follow a request across multiple microservices.

• Example: Frontend (20ms) -> Auth Service (50ms) -> Database (3000ms).
• Use Case: Finding latency bottlenecks in complex architectures.

2. The Golden Signals (What to Measure)

If you strictly follow the Google SRE handbook, you only need to measure four things to know if a user is happy.

1. Latency: How long does it take?
2. Traffic: How much demand is there? (Requests per second).
3. Errors: How often does it fail? (HTTP 5xx).
4. Saturation: How "full" is the service? (Memory/CPU capacity).

Mentor Tip: Never measure "Average Latency." It is a lie. If 99 users get a 10ms response and 1 user gets a 10-minute response, the "Average" looks fine, but that 1 user is furious. Always measure the 99th Percentile (p99).

3. How to Write Good Logs

Most logs are garbage. "Error: Something went wrong." This helps nobody. Structured Logging is the standard. Write logs as JSON objects, not strings.

Bad Log:

text
[Info] User uploaded file.

Good Log:

json
{
  "level": "info",
  "event": "file_uploaded",
  "user_id": 42,
  "file_size_mb": 150,
  "duration_ms": 450,
  "region": "us-east-1"
}

Now, you can query: "Show me all uploads > 100MB that took > 500ms in us-east-1." You have turned text into a database.

Summary

Observability is not something you "add later." It is a feature.

1. Metrics tell you the system is broken.
2. Traces tell you where it is broken.
3. Logs tell you why it is broken.

If you build a feature without these three, you haven't finished the feature. You've just built a ticking time bomb.