How I Troubleshoot Production Systems: A Solution Architect's Guide

Production Troubleshooting Is an Architecture Skill

A production incident rarely arrives with a message saying “the root cause is line 142.” Users report that checkout is slow, dashboards show partial failures, Kafka lag grows, pods restart, or a database pool reaches its limit. The architect's job is to turn symptoms into a structured investigation while protecting the business.

My first objective is not intellectual perfection. It is stabilization. Once the system is safe, the team can investigate deeply and prevent recurrence.

1. Start With Business Impact

Before reading thousands of logs I want to know what users cannot do. Is login failing? Are payments delayed? Is only one region affected? Is the issue on read traffic, write traffic or an asynchronous workflow?

Incident
   ↓
Business impact
   ↓
Scope
   ↓
Recent changes
   ↓
Rate / Errors / Duration
   ↓
Saturation / Dependencies
   ↓
Stabilize
   ↓
Root cause

2. Establish the Timeline

Find the first clear signal: latency change, error spike, traffic increase, deployment, configuration change, certificate issue, database failover or third-party degradation. A timeline immediately reduces the search space.

3. Check Recent Changes

Deployments are not always the cause, but they are high-value evidence. I check application releases, database migrations, infrastructure changes, feature flags, secrets, DNS, certificates and scaling events around the incident start time.

4. Use RED Metrics for Request Systems

Rate, Errors and Duration provide a fast picture. If request rate doubled and latency rose with it, capacity may be the issue. If rate stayed normal but errors suddenly increased after a release, the investigation goes elsewhere.

5. Use Saturation Metrics

CPU alone is not enough. I inspect memory, event-loop delay or thread pools, database connections, queue depth, Kafka lag, disk I/O, network, pod throttling and downstream rate limits.

6. High CPU: What Do You Check?

I correlate CPU with traffic and deployments, then profile the work. Causes can include serialization, compression, expensive loops, regex, encryption, retry storms, garbage collection, or application logic accidentally processing the same data repeatedly.

7. High Memory: Leak or Workload?

Memory growth can come from a leak, larger traffic, bigger payloads, caches, buffers, batch size or an upstream change. For JVM services I look at post-GC heap and allocation. For Node.js I look at heap, external memory, event-loop behavior and retained objects.

8. Pods Are Restarting

Check whether restarts are caused by OOMKill, failed liveness probes, application exits, node pressure or manual rollout. A probe misconfiguration can create a restart loop even though the actual dependency outage is elsewhere.

9. Latency Is High but CPU Is Low

This often indicates waiting: database locks, connection-pool exhaustion, slow external APIs, DNS, storage I/O, network, queue waits or thread-pool starvation. Low CPU does not mean a service is healthy.

10. Database Pool Is Exhausted

Find whether queries became slower, transactions became longer, pool size changed, replicas increased or connections are leaking. Increasing the pool without understanding database capacity can move the bottleneck from the application into the database.

11. The Database CPU Is High

Inspect top queries, execution plans, lock waits, missing/unused indexes, changed cardinality, large scans and new reporting traffic. A recently added endpoint can turn one inefficient query into a platform incident at scale.

12. Deadlocks Increased

Look for changed transaction order, longer transactions and new concurrent workflows. The immediate application response may be safe retry, but the structural fix is usually consistent lock order or transaction redesign.

13. Replication Lag Is Growing

Check write rate, replica I/O/CPU, long-running queries, network and replication errors. Understand business impact: stale dashboards may be acceptable, stale authorization or order status may not be.

14. Kafka Lag Is Growing

Consumer lag means processing is not keeping up. I compare arrival rate, consumer throughput, partition count, consumer count, per-message latency and downstream saturation.

Kafka lag ↑
   ↓
Arrival rate increased?
   ├── yes → capacity / partitions / batch tuning
   └── no
        ↓
Processing got slower?
   ├── DB latency
   ├── external API
   ├── retries
   ├── poison message
   └── CPU / memory

15. Should You Add More Kafka Consumers?

Only if there are available partitions and downstream systems can handle the extra concurrency. Scaling consumers against an already saturated database is a common mistake.

16. One Poison Message Keeps Failing

Do not let one bad event block the entire partition forever. Define retry limits, classify permanent versus transient failures, and move unrecoverable events to a DLQ with enough context for investigation and replay.

17. External API Became Slow

Check your timeout budget and circuit-breaker state. If there is no timeout, requests can accumulate until your service fails too. If every failure is retried aggressively, you may amplify the partner's outage.

18. DNS Problems Look Like Application Problems

Intermittent name resolution or stale DNS can manifest as connection timeouts. Include DNS and network layers in the investigation, especially after infrastructure changes.

19. TLS Certificate Expired

Certificate monitoring should catch this before users do. During an incident verify expiry, chain, hostname, intermediate certificates and whether clients trust the issuing authority.

20. Only One Availability Zone Is Failing

Compare error and latency metrics by zone/node. The root cause may be networking, node pressure, storage or a zonal dependency. Routing traffic away can be the fastest stabilization measure.

21. Only One Tenant Is Affected

Look for tenant-specific data volume, hot keys, permissions, configuration, feature flags or oversized workloads. Multi-tenant systems often hide these patterns in global averages.

22. Error Rate Is Low but Users Are Complaining

Technical success codes can hide business failure. An API can return 200 with an empty or incorrect result. Business metrics and synthetic journeys help detect this class of incident.

23. Logs Show Nothing Useful

That is an observability design problem. Logs should be structured, correlated and tied to operations. I include trace IDs, safe business identifiers, dependency timings and error classification.

24. Why Distributed Tracing Matters

A user request may cross gateway, service A, Kafka, service B and a database. Tracing reveals where time was spent and which hop failed, reducing the need to manually correlate timestamps across systems.

25. When Should You Roll Back?

If the incident clearly aligns with a recent release and rollback is safe, stabilizing quickly is often better than debugging live under customer impact. Preserve evidence before rollback where practical.

26. When Should You Disable a Feature?

Feature flags are valuable when one optional capability threatens the core transaction. Degrading recommendation, analytics or non-critical enrichment may preserve checkout or login.

27. Load Shedding Can Be Better Than Total Failure

If capacity is exhausted, reject lower-priority work early rather than let all requests time out. Rate limits, queues, priority classes and admission controls can preserve critical paths.

28. Why Timeouts Must Be End-to-End

A downstream 10-second timeout is useless if the user-facing request budget is three seconds. Allocate the total request budget across hops so services fail early enough to return a controlled response.

29. Retry Only What Is Safe

Transient network failures may be retryable. Validation errors and authorization failures are not. Write retries require idempotency. Retry classification should be explicit.

30. Check the Queue Before the CPU

Many incidents are waiting problems. Thread pools, connection pools, Kafka partitions, job queues and API concurrency limits can saturate long before CPU reaches 100%.

31. How I Use a Troubleshooting Matrix

32. Create an Incident Channel and One Decision Owner

During serious incidents, uncontrolled parallel action can make things worse. One incident lead coordinates stabilization while focused engineers investigate separate hypotheses.

33. Keep a Decision Log

Record timestamps, observations, actions and outcomes. It helps handovers, post-incident analysis and prevents teams from repeating failed experiments.

34. Separate Mitigation from Root Cause

“We restarted pods and it recovered” is a mitigation, not a root cause. Ask why the pods entered the bad state and why monitoring did not prevent customer impact.

35. Post-Incident Review Should Change the System

A useful review produces engineering actions: alerts, capacity changes, safer deployment, better timeouts, tests, runbooks, ownership or architecture changes. The goal is not blame.

36. Build Runbooks for Known Failure Modes

Document how to confirm the condition, safe mitigation, escalation path and rollback. Runbooks reduce decision time when pressure is high.

37. Test Failure Before Production Finds It

Load tests, dependency fault injection, backup restores and game days reveal assumptions. You do not need a full chaos-engineering program to learn from controlled failure tests.

38. Capacity Planning Is Part of Reliability

Track growth in traffic, data, partitions, database size and dependency quotas. Many “sudden” incidents are actually predictable capacity thresholds.

39. Watch Cost During Incidents

Autoscaling, retry storms and runaway logging can create large cloud costs even after user impact falls. Operational dashboards should include relevant cost or resource-efficiency signals.

40. The Architect's Incident Questions

• What user journey is broken?
• When did it start?
• What changed?
• Is demand higher or capacity lower?
• Where is time being spent?
• Which resource is saturated?
• Which dependency is unhealthy?
• Can we degrade or shed load?
• Can we roll back safely?
• What evidence must we preserve?

FAQ

What is the first thing to do during a production incident?

Understand business impact and stabilize critical functionality. Deep root-cause analysis is important, but customer impact often requires mitigation first.

Should I restart services when troubleshooting?

Only when it is an appropriate mitigation and you understand the operational risk. Capture useful diagnostics first where possible; repeated blind restarts can hide the root cause.

What observability tools are essential?

Metrics, structured logs and distributed tracing form the core. The specific products matter less than consistent instrumentation, correlation and useful dashboards.

Related Guides

• Circuit Breaker, Retry and Timeout Patterns
• Kafka Backpressure in Production NestJS Systems
• Developer to Solution Architect Roadmap