Debugging in Production

No matter how good your test suite is, production will surprise you. The real world has data you didn’t expect, traffic patterns you didn’t model, and timing issues you can’t reproduce locally.

The tools that actually help

Forget stepping through code with a debugger — you can’t attach a debugger to a Kubernetes pod serving 10k requests per second. What works:

• Structured logging. JSON logs with correlation IDs. If your logs aren’t structured, you’re grepping in the dark.
• Distributed tracing. Follow a request across service boundaries. OpenTelemetry makes this approachable.
• Metrics with percentiles. Averages lie. P99 latency tells you what your worst-off users experience.

The mindset

Production debugging is detective work. You start with symptoms, form hypotheses, and narrow down. The key skill isn’t knowing the tools — it’s staying calm when the dashboard is red and Slack is on fire.

# The most useful production debugging command
kubectl logs -f deployment/my-service --since=5m | jq '.level == "error"'

Build systems that are observable from day one. You’ll thank yourself later.