High CPU utilization - how to analyze high CPU utilization for java application

Problem:-
During load testing, many time we observed high CPU utilization and we need to analyze it usages. Here some steps to identify high CPU utilization bottleneck
1. For java application, we can take thread dump(jstack) to check how many thread are blocked/waiting
3. Check GC frequency, If there is less memory allocated then there could be chance of high GC run, for which it requires more CPU cycles.

Analysis using monitoring tools:-

1. using vmstat,check waiting for running(r), uninterruptible sleep (b) thread count.
    1. vmstat :-
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 16836884 202724 2405112    0    0     0     1    0    0  0  0 100  0  0
   - check CPU section for user(us) utilization.
   - check for context switches(cs)
 
  2. top : -

Possible solutions(as per my experience):-
 Application level tuning: -
   1. First, identify high cpu process id and convert it into hex. Now take thread dump and search for nid=<process id hex>, here you can identify which code/class causing issue
   2. Increase total no. of threads in configuration file
   3. Increase connection pool in configuration file

 
 Reference:-
 http://careers.directi.com/display/tu/Understanding+CPU+Utilization+and+Optimization