This is about an interesting but strenuous exercise I had gone through in life as a software engineer. This happened many years back and some parts are blurry. Disclaimer: The methods followed in this might not the best ways to debug the issue. Scene So the story goes like this. We were developing a gateway product and were in the last stages of doing the performance tests on the gateway.The performance expectations of the gateway was so well documented that we have a completely detailed input and output model which exactly predicts what is the kind of load this gateway can handle (probably one of the best performance models I ever worked with). During the performance tests we observed that the system CPU usage of the Linux kernel is going really beyond what is expected as we were loading the system towards the peak load it can handle. The details of this will be explained once I finish the architecture overview in the next section. Architecture From an architecture point of view, at ...