Linux/AIX Resource monitoring tools used during performance/load testing

 1. NMON  :-
   - we can use NMON in 2 ways 1. interactive 2. recording . Following is example of recording various parameters
     nmon  -ftm <nmon_stat> -s 300 -c 7200000
      - above command will start nmon for 7200000*300 times in nmon_start folder. It will create <timestamp>.nmon file to save the stats
 - to analyze results, we can use nmon_analyser_v42(provide .nmon file as input to MS excel sheet to generate various graphs)
 - Download analyser from below link.


  2. sar :-
    - sar collects the stats for CPU,memory,IO,Network related activity
- We can use ksar to generate graph from collected data.
   LC_ALL=C sar -A -f /var/log/sa/sa > /tmp/sar_report.txt

  3. vmstat :-
  procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
   0  0      0 16836884 202724 2405112    0    0     0     1    0    0  0  0 100  0  0
    r - no. of running processes
    b - no. of blocked processes,waiting for resource
    buff- buffer for I/O operation
    cache- cached by OS
    bi- blocks(1k) read from hard disk
    bo- blocks write into hard disk
    in - interrupts per sec
    cs - no. of context switched per second. - switching of CPU from one process to another

>> How to free memory in AIX
set as 
$vmo -o minperm%=1 -o maxperm%=1 -o strict_maxperm=1 -o minclient%=1 -o maxclient%=1
Restore origninal value:- 
$vmo -o minperm%=3 -o maxperm%=90 -o strict_maxperm=90 -o minclient%=67 -o maxclient%=90

  4. TOP

  5. PS
   - $ ps -eLF | grep java |wc -l :-  it will give total count of java processes(threads)

Memory leak analysis for C++, Java application

Once we observe linear memory growth during endurance load testing, we could suspect about memory leak in application

 1. C++ -
       we can use following tool to analyse memory leak in c++ application
1. valgrind : -
          -   Valgrind is tool with which we can analyses memory leak for C++ application.
On linux machine it should available.  Start the application with below parameters. Note- Your application will become slow once you start with valgrind.
valgrind --tool=memcheck --leak-check=full --show-reachable=yes --num-callers=20 --track-fds=yes --leak-resolution=high --trace-children=yes -v <binary file> <arg1> <arg2>
    - valigrind report analysis : -  Check following categories from text report.
Still reachable blocks
Definitely Lost blocks
Indirectly Lost:
Possibly Lost


 Reference :-
  http://valgrind.org/docs/manual/manual.html


 2. Java -
        1. Eclipse Memory Analysis Tool - We used E-MAT to analyse heapdump(generate by application because of HeapDumpOnOutOfMemoryError). We loaded 2G heapdump in E-MAT ( Note:- E-MAT machine having 32G memory and 12 core cpu)
- once heapdump gets load in tool, it will provide following high level report
 - Leak Suspect Report :- shows classes/object which consumed more heap.
 - Check Histogram of dominator tree :- Check Retained Heap size ( retained heap si addition of the shallow heap size of all objects that would be removed when this object is garbage collected)
             here max retained heap size instances could be suspected objects. If we drill down dominator tree more, we can identify object which holds more memory

Reference link:-
 http://eclipsesource.com/blogs/2013/01/21/10-tips-for-using-the-eclipse-memory-analyzer/
 
        2. JVisualVM :-
          we can analyaze java application using jvisualvm tool(D:\test\jdk1.7.0_45\bin\jvisualvm.exe) by locally or remotely
         - we can monitor  :-  CPU usage, heap, no. of threads
- Thread analysis
- Profiler - we can monitor CPU uses by each method
           - In memory profiling visualvm, gives max usage of heap by classes, instances.
- VisualGC - We need install VisualGC plugin from Tool > Plugins
           - This plugin monitor GC activity - the Perm gen, Eden space, GC time
 For more detail please go through below link :-
  http://myloadtesting.blogspot.in/2015/02/performance-analysis-using-jvisualvm.html
   

How to debug jmeter script

There are different way to debug jmeter script. You can use following Postprocessor
 1. Debug PostProcesssor
    - You need to add as child in every sampler (e.g HTTP Request)
- Check value of variable in View Result Tree > Sampler's output

 2. BeanShell Postprocessor
   - You can log the variable value in jmeter.log file
     e.g ${__log("-Counter value-------------${counter} --------------")};

 3. You can user PostProcesssor > Regular Expression Extractor to check regex and its output

 4. Response Assertion :- We can use it to check both things. 1. Verify response and any specific Pattern by giving different rules

High CPU utilization - how to analyze high CPU utilization for java application

Problem:-
During load testing, many time we observed high CPU utilization and we need to analyze it usages. Here some steps to identify high CPU utilization bottleneck
1. For java application, we can take thread dump(jstack) to check how many thread are blocked/waiting
3. Check GC frequency, If there is less memory allocated then there could be chance of high GC run, for which it requires more CPU cycles.

Analysis using monitoring tools:-

1. using vmstat,check waiting for running(r), uninterruptible sleep (b) thread count.
    1. vmstat :-
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 16836884 202724 2405112    0    0     0     1    0    0  0  0 100  0  0
   - check CPU section for user(us) utilization.
   - check for context switches(cs)
 
  2. top : -

Possible solutions(as per my experience):-
 Application level tuning: -
   1. First, identify high cpu process id and convert it into hex. Now take thread dump and search for nid=<process id hex>, here you can identify which code/class causing issue
   2. Increase total no. of threads in configuration file
   3. Increase connection pool in configuration file

 
 Reference:-
 http://careers.directi.com/display/tu/Understanding+CPU+Utilization+and+Optimization

Jmeter Functions

While working on jmeter functions, tried following functions in scripting
- For logging in log file, we can use following function in <BeanShell Postprocessor> OR <BeanShell Sampler>


1. ${__log("-Thread Number---------------${__threadNum()} --------------")};
sample output :-
INFO  - jmeter.functions.LogFunction: Thread Group 1-1 : "-Thread Number---------------1 --------------"

2. ${__log("-Counter---------------${__counter(true,c)} --------------")};
sample output :-
INFO  - jmeter.functions.LogFunction: Thread Group 1-1 : "-Counter---------------38 --------------"

3. ${__log("-Request/Sampler---------------${__samplerName(s)}--------------")};
sample outout :-
INFO  - jmeter.functions.LogFunction: Thread Group 1-1 : "-Request/Sampler---------------/static/94262ec5/images/48x48/notepad.png--------------"

4. ${__log("----------------${__machineIP(ip)}--------------")};

${__log("----------------${__machineName(mname)}--------------")};

${__log("-Time---------------${__time(YMDHMS)} --------------")};
sample output in log file:-
INFO  - jmeter.functions.LogFunction: Thread Group 1-1 : "-Time---------------20150518-112329 --------------"

${__log("----------------Success--------------")};

${__log("-split---------------${__split(testamark,var,a)} --------------")};
${__log("-Var1---------------${var_1} --------------")};
${__log("-Var2---------------${var_2} --------------")};
${__log("-Var3---------------${var_3} --------------")};
sample output:-
jmeter.functions.LogFunction: Thread Group 1-1 : "-split---------------testamark --------------"
jmeter.functions.LogFunction: Thread Group 1-1 : "-Var1---------------test --------------"
jmeter.functions.LogFunction: Thread Group 1-1 : "-Var2---------------m --------------"
jmeter.functions.LogFunction: Thread Group 1-1 : "-Var3---------------rk --------------"

${__log("-Property Value---------------${__property(user.dir)} --------------")};
sample output:-
INFO  - jmeter.functions.LogFunction: Thread Group 1-1 : "-Property Value---------------D:\data_bkup\PEG\apache-jmeter-2.7\apache-jmeter-2.7\bin --------------"

Automate Jmeter load tests with Jenkin and Maven

As you know is Jenkin is widely used  tool for Continuous Integration in performance testing, So given tried for installation/configuration of Jenkin. Following are short step to automate load testing process(jmeter scripts) in Jenkin.

1. Configure apache-maven for jmeter script repository
  #cd /usr/local/apache-maven-3.1.1
  #export PATH=$PATH:/usr/local/apache-maven-3.1.1/bin
2. Create pom.xml in above repository(/var/tmp/jmeter-demo)
  #/var/tmp/jmeter-demo
3. Verify repository by - mvn verify
   -  Maven will download and install jmeter related jar in repository
   - e.g Repository path :
[root@Cassandra jmeter-demo]# ls
pom.xml  src  target
jmeter jar: target/jmeter/lib/ext/
Content of pom.xml
---------------------
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
        <modelVersion>4.0.0</modelVersion>
        <groupId>com.example</groupId>
        <artifactId>jmeter-demo</artifactId>
        <packaging>jar</packaging>
        <version>1.0-SNAPSHOT</version>
        <name>jmeter-demo</name>
        <url>http://maven.apache.org</url>

        <build>
                <plugins>
                        <plugin>
                                <groupId>com.lazerycode.jmeter</groupId>
                                <artifactId>jmeter-maven-plugin</artifactId>
                                <version>1.4.1</version>
                                <executions>
                                        <execution>
                                                <id>jmeter-tests</id>
                                                <phase>verify</phase>
                                                <goals>
                                                        <goal>jmeter</goal>
                                                </goals>
                        <configuration>
                                <proxyConfig>
                                    <host>proxyhost</host>
                                    <port>8181</port>
                                    <username>username</username>
                                    <password>password</password>
                                    <hostExclusions>localhost</hostExclusions>
                                </proxyConfig>
                            </configuration>
                                        </execution>
                                </executions>
                        </plugin>
                </plugins>
        </build>
</project>
---------------------------

4. Install jenkin rpms and start jenkin ( #/etc/init.d/jenkins start)
5. Load performance plugin through jenkin UI
6. Create Project from jenkin UI
7. onfigure project for jmeter
    - In Build section, add pom.xml path
    - In Post-build Action > Add New report > Jmeter > give path for .jtl file
- Save the configuration
7. Start build. Once build complete check performance Trend

Reference:-

https://ribblescode.wordpress.com/2012/04/16/how-to-automate-jmeter-tests-with-maven-and-jenkins-hudson-8/

https://wiki.jenkins-ci.org/display/JENKINS/Performance+Plugin