Two months ago, one of our core APIs was averaging 2 seconds per request. At peak traffic of 12,000 requests per second, users were experiencing frequent timeouts. Our 99.9% SLA was at serious risk.
Here is the exact step-by-step process I followed to bring it down to 200ms.
Step 1: Profiling with New Relic
The first thing I did was set up distributed tracing. I assumed the database was the problem, but the data told a different story. The bottleneck was actually a synchronous external API call that we were making for every request.
The lesson: never guess. Measure first.
Step 2: Introducing Redis Caching
The external API response rarely changed. I implemented a Redis cache with a 5-minute TTL. This single change cut the response time from 2 seconds to 800ms.
Cache key: api:user:{id}:profile
Hit rate: 94%
Step 3: Database Indexing
After caching, the next bottleneck was a slow JOIN query on a 10-million-row table. I added a composite index on (user_id, created_at). The query went from 450ms to 12ms.
Step 4: Async Processing for Non-Critical Paths
Not everything needs to happen synchronously. I moved log writing and analytics tracking to a background job queue using RabbitMQ. The API now returns a response immediately and processes these tasks in the background.
Step 5: Connection Pool Tuning
Finally, I tuned the database connection pool. The default was 10 connections — way too low for 12k req/s. I increased it to 50 and added a timeout mechanism.
The Final Result
Before: 2000ms avg response, 4.2% timeout rate, 78% CPU
After: 200ms avg response, 0.01% timeout rate, 34% CPU
The SLA is now safe. The team is happy. I learned that performance optimization is not about one big fix — it is about many small, deliberate improvements.