JVM Heap Tuning Flags

Some of the most commonly used JVM Heap tuning flags are mentioned below.

Heap size

Ø  Initial Heap size: -Xms   e.g. -Xms2048m

Ø  Maximum Heap size: -Xmx e.g. -Xmx2048m

Ø  To set ratio of young generation to old generation: -XX:NewRatio

Ø  To set initial size of the young generation: -XX:NewSize

Ø  To set Maximum size of the young generation: -XX:MaxNewSize

Ø  To set permgen space initial size:

-XX:PermSize=N (JDK 7 only) OR -XX:MetaspaceSize=N (JDK 8 only)

Ø  To set maximum size of Permgen space:

-XX:MaxPermSize=N (JDK 7 only) OR -XX:MaxMetaspaceSize=N (JDK 8 only)

Ø  Instruct to take heap dump on out of memory error: -XX:+HeapDumpOnOutOfMemoryError

Ø  To specify path for Heap dump file: -XX:HeapDumpPath=<path>

Garbage Collection

Ø  To instruct use of Serial Garbage collector:     -XX:+UseSerialGC

Ø  To instruct use of parallel Garbage collector:  -XX:+UseParallelGC

Ø  To instruct use of Concurrent Mark & Sweep collector:   

-XX:+UseConcMarkSweepGC -XX:+UseParNewGC

Ø  To instruct use of G1 GC collector: -XX:+UseG1GC

Ø  To set no of threads (in case of parallel GC): -XX:ParallelGCThreads=N

Ø  To enable GC logging: -verbose:gc

Ø  To specify separate log file name for GC log instead of standard output:

-Xloggc:<path>

Ø  To enabled detailed GC logging: -XX:+PrintGCDetails

Ø  To print relative timestamp for each GC event: -XX:+PrintGCTimeStamps

Ø  To set the amount of the young generation set aside for survivor spaces: -XX:InitialSurvivorRatio=N

o    survivor_space_size = new_size / (initial_survivor_ratio + 2)

o    For the default initial survivor ratio of 8, each survivor space will occupy 10% of the young generation.

Ø  To specify initial number of GC cycles the JVM attempts to keep an object in the survivor spaces: -XX:InitialTenuringThreshold=N

o    The default is 7 for the throughput and G1 collectors, and 6 for CMS.JVM may change this value on his own.

Ø  To specify maximum  number of GC cycles the JVM attempts to keep an object in the survivor spaces:

o    -XX:MaxTenuringThreshold=N

o    For the throughput and G1 collectors, the default maximum threshold is 15, and for CMS it is 6.

 CMS & G1 specific

Ø  To specify when CMS/G1 should begin background scanning of the old generation: -XX:CMSInitiatingOccupancyFraction=N

               Can try to reduce this value if there are concurrent mode failures.

Ø  To set the number of threads to use for CMS/G1 background scanning: -XX:ConcGCThreads=N

Resource Utilization Monitoring [Linux]

Let’s check what all metrics we can collect & how these values help us understand the system load.

We will take examples from Linux OS with the help of utilities like NMON (it’s free tool).

Using ‘r’ option in NMON, we see the Linux and CPU details for the machine.

e.g.

Linux: Linux version 2.6.32-431.el6.x86_64 (mockbuild@ca-build44.us.oracle.com)

Build: (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) )

Release  : 2.6.32-431.el6.x86_64

Version  : #1 SMP Wed Nov 20 23:56:07 PST 2013

cpuinfo: model name : Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz

cpuinfo: vendor_id : GenuineIntel

cpuinfo: cpu MHz  : 2599.973

cpuinfo: wp  : yes

# of CPUs: 4

Machine  : x86_64

/etc/*ease[1]: LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-n

/etc/*ease[2]: arch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:print

/etc/*ease[3]: ng-4.0-noarch                                                      

/etc/*ease[4]: Oracle Linux Server release 6.5                                    

lsb_release: Distributor ID:        OracleServer                                   

lsb_release: Description:   Oracle Linux Server release 6.5                       

lsb_release: Release:       6.5                                                   

lsb_release: Codename:      n/a     

    A.      CPU Utilization

First of all, we can check CPU architecture details using command ‘lscpu’.

It gives you details of CPU family, no of CPUs , no of Cores, therads, socktes, speed etc.

e.g. ]$ lscpu

Architecture:          x86_64

CPU op-mode(s):        32-bit, 64-bit

Byte Order:            Little Endian

CPU(s):                4

On-line CPU(s) list:   0-3

Thread(s) per core:    1

Core(s) per socket:    4

Socket(s):             1

NUMA node(s):          1

Vendor ID:             GenuineIntel

CPU family:            6

Model:                 63

Stepping:              2

CPU MHz:               2599.973

BogoMIPS:              5199.94

Hypervisor vendor:     Microsoft

Virtualization type:   full

L1d cache:             32K

L1i cache:             32K

L2 cache:              256K

L3 cache:              25600K

NUMA node0 CPU(s):     0-3

Or you can similar info in ‘proc/cpuinfo’ file. It shows info processor wise. I have copied info for only first processor out of total 4.

e.g. $ cat /proc/cpuinfo

processor       : 0

vendor_id       : GenuineIntel

cpu family      : 6

model           : 63

model name      : Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz

stepping        : 2

cpu MHz         : 2599.973

cache size      : 25600 KB

physical id     : 0

siblings        : 4

core id         : 0

cpu cores       : 4

apicid          : 0

initial apicid  : 0

fpu             : yes

fpu_exception   : yes

cpuid level     : 13

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good unfair_spinlock pni cx16 hypervisor lahf_lm

bogomips        : 5199.94

clflush size    : 64

cache_alignment : 64

address sizes   : 42 bits physical, 48 bits virtual

power management:

Now, once we understand CPU architecture, lets have a look at CPU utilization using NMON utility. For this, on command open it by typing ‘nmon’ and hit ‘c’.

e.g.  here it showing separate stats for each CPU and also showing average stats.

CPU  User%  Sys% Wait% Idle

  1  33.0  11.2   0.0  55.8

  2   7.7   0.0   0.0  92.3

  3  19.9   0.5   0.0  79.6

  4  29.2   3.6   0.0  67.2

---------------------------

Avg  22.4   3.8   0.0  73.7

User% - used by user processes.(usage by your application processes)

Sys% - used by Kernel processes.

Wait% - waiting for I/O to finish. (high percentage can indicate I/O issues)

Idle% - waiting for idle for next tasks to be executed.

   B.      Memory:

If we check memory utilization using NMON or top on Linux, it will always show free  memory percentage as very low, giving you impression that all the memory is being used & you need more memory. But Linux is borrowing unused memory for disk caching. If your applications want more memory, they just take back a chunk that the disk cache borrowed. Disk cache can always be given back to applications immediately.

We can check this using free –m command.

e.g. NMON is showing

                RAM  

x Total MB        80575.8

x Free  MB        1355.7

x Free Percent    1.7%

when I hit free –m

$ free -m

             total       used       free     shared    buffers     cached

Mem:         80575      79218       1357          0       2168      23596

-/+ buffers/cache:      53453      27122

Swap:         4095        122       3973

So here its showing 1357 MB of free memory but out of memory used for caching 26.4 GB memory is free which can be allocated to processes as required.

     C.      I/O

As mentioned in CPU stats ‘wait%’ for CPU indicates waiting on IO resources.

Then we can check I/O stats.

e.g. using ‘d’ option for NMON, we can see busy% , reads and Writes (KB per seconds) per disk and on average.

DiskName Busy  Read WriteKB

sda        0%    0.0    0.0

sda1       0%    0.0    0.0

sdb        0%    0.0    0.0

dm-0       0%    0.0    0.0

dm-1       0%    0.0    0.0

dm-2       0%    0.0    0.0

dm-3       0%    0.0    0.0

dm-4       0%    0.0    0.0

Totals Read-MB/s=0.0      Writes-MB/s=0.0      Transfers/sec=0.0

Using iostat:

e.g. $ iostat –xtc

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           5.93    0.00    0.69    0.08    0.00   93.29

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util

sda               0.02     2.84    0.05    2.36     3.38    41.58    18.60     0.00    0.48   0.15   0.04

sdb               1.76    80.30    1.69   53.15   112.90   831.04    17.21     0.05    0.94   0.13   0.71

dm-0              0.00     0.00    0.01    0.10     0.45     0.80    11.41     0.00    1.99   0.09   0.00

dm-1              0.00     0.00    0.03    0.09     0.21     0.75     8.00     0.00    2.55   0.18   0.00

dm-2              0.00     0.00    0.06    0.08     1.19     0.65    12.71     0.00    1.32   0.54   0.01

dm-3              0.00     0.00    0.49    1.30     5.24    10.39     8.74     0.01    4.85   0.13   0.02

dm-4              0.00     0.00    2.87  131.72   105.80   818.45     6.87     0.57    4.22   0.05   0.68

columns explained:

rrqm/s

The number of read requests merged per second that were queued to the device.

wrqm/s

The number of write requests merged per second that were queued to the device.

r/s

The number of read requests that were issued to the device per second.

w/s

The number of write requests that were issued to the device per second.

rsec/s

The number of sectors read from the device per second.

wsec/s

The number of sectors written to the device per second.

avgrq-sz

The average size (in sectors) of the requests that were issued to the device.

avgqu-sz

The average queue length of the requests that were issued to the device.

await

The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.

svctm

The average service time (in milliseconds) for I/O requests that were issued to the device. Warning! Do not trust this field any more. This field will be removed in a future sysstat version.

%util

Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.

System Performance monitoring - using vmstat command

One major aspect of monitoring a performance test is to capture system performance metrics and analyze it.

When monitoring machine level or OS level resource utilization, at high level can be categorized in

      ·         CPU Utilization

      ·         Memory Utilization

      ·         I/O stats

There are multiple commands/utilities available in different flavors or Unix/Linux. Let’s check vmstat today!

vmstat – this command displays virtual memory statistics.

e.g.

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----

 r  b   swpd       free          buff         cache       si   so    bi    bo    in   cs us sy id wa st

 0  0 102716 4587628 2528320 18105548    0    0    18   114    4    5  6  1 93  0  0

Processes

r - The number of processes waiting for run time.

     When this number exceeds the number of CPUs on the server, a CPU bottleneck exists, and some tasks are waiting for execution.

b - The average number of kernel threads on the wait queue at one-second intervals. (awaiting resource, awaiting input/output). Kernel threads are placed on the wait queue when scheduled for execution and are waiting for one of their process pages to be paged in.

memory

swpd - the amount of virtual memory used

buff: the amount of memory used as buffers.

cache: the amount of memory used as cache.

inact: the amount of inactive memory. (-a option)

active: the amount of active memory. (-a option)

swap

si: Amount of memory swapped in from disk (/s).

so: Amount of memory swapped to disk (/s).

 Swapping the memory pages to the swap file will be seen in the so (swap out - memory swapped to disk).

  

IO

bi: Blocks received from a block device (blocks/s).

bo: Blocks sent to a block device (blocks/s).

High values for these shows the I/O operations happening.  Those could be related to swapping as well.

System

       in: The number of interrupts per second, including the clock.

       cs: The number of context switches per second.

CPU

       These are percentages of total CPU time.

       us: Time spent running non-kernel code. (user time)

       sy: Time spent running kernel code. (system time)

       id: Time spent idle.

       wa: Time spent waiting for IO.

       st: Time stolen from a virtual machine.

High percentage for ‘wa’ indicates, there is issue with I/O. sometimes it’s classified as "waiting on I/O". A wa value over 40 percent could indicate that the disk subsystem may not be balanced properly, or it may be the result of a disk-intensive workload.

Using with modes:

There are additional modes for vmstat which can be used to get detailed information about specific area:

$ vmstat --help

usage: vmstat [-V] [-n] [delay [count]]

              -V prints version.

              -n causes the headers not to be reprinted regularly.

              -a print inactive/active page stats.

              -d prints disk statistics

              -D prints disk table

              -p prints disk partition statistics

              -s prints vm table

              -m prints slabinfo

              -t add timestamp to output

              -S unit size

Providing sampling time

for capturing the statistics during performance test, we can specify the sampling interval & no of invocations.

e.g.  displays statistics at interval of 1 second for 5 times .

vmstat -t 1 5

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ ---timestamp---

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

 0  0 102648 4220480 2537344 18213972    0    0    18   114    7    0  6  1 93  0  0    2016-04-14 06:17:19 EDT

 0  0 102648 4220472 2537344 18213972    0    0     0   741 4468 10644  1  0 99  0  0   2016-04-14 06:17:20 EDT

 0  0 102648 4220472 2537348 18213968    0    0     0    61 4386 11545  1  0 99  0  0   2016-04-14 06:17:21 EDT

 0  0 102648 4220456 2537348 18213972    0    0     0     6 4588 12701  3  0 97  0  0   2016-04-14 06:17:22 EDT

 0  0 102648 4220448 2537348 18213988    0    0     8   248 4522 12081  1  0 98  1  0   2016-04-14 06:17:23 EDT

Can further redirect the statistics to a file and later on open file and analyze it.

e.g. $ vmstat -t 1 5 >> vmstat_output.txt

Performance Workload Modelling

Often, Performance testing is said to be Art + Science.  And the major part of this ‘Art’ lies in designing workload model for the application or product under test.

What is Workload ?

To put it simply, Workload is nothing but amount of work the system must do in given period of time. In the context of performance testing, its about understanding what sort of load system will be catering in production.

Factors deciding workload

So to define workload, need to decide upon below factor:

      1.       No. of concurrent Users/sessions

      2.       User profiles

      3.       Mix Use cases/Business processes/operations

      4.       Throughput i.e. no of operations or transactions done during load [Transactions per Hour]

      5.       SLAs – in terms of Throughput , response times & resource utilization

Steps to Design Workload Model

      1.       Predicting application usage for future :  Major part of it should be deciding upon the flows/business processes to be tested, their implementation & navigation paths.

      2.       Obtaining Volumes: Need to obtain details of ‘maximum no. of concurrent users’, ‘Mix of business processes’ & ‘maximum no. of transactions/operations predicted.

e.g. lets say we will have 100 as maximum no. of concurrent user operating , majorly doing 5 flows/business processes & during peak hour maximum operations done are 10000.

If the business process mix is: 10%, 20%, 30%, 15%, 15%. Then using this info, we derive below statistics:

Business Process/Use Case	No of concurrent Users	No of operations/transactions to be done during peak hour
Case 1	10	1000
Case 2	20	2000
Case 3	30	3000
Case 4	15	1500
Case 5	15	1500
Total	100	10000

This is simplistic case given for explanation. In reality, percentage of no. of users mapped and no of operations could be different. It depends upon type of user profiles & its usage.

      3.       So , we have no of councurrent users &  no. of operations to be performed per use case. In order to design our tests , need to decide upon pacing & think time. Alternative way is to use – ‘goal oriented scenario’  in  Loadrunner Or ‘constant throughput’ timer in Jmeter.

In order to decide upon it manually, one should know baseline results for particular flow – how much time a business process takes to complete without think time.

e.g. lets say case 1  takes 30 seconds to complete without think time & has 3 think time statements in side script. Lets say if keep think time value to ~10 seconds then with think time , it will need 30 + 30 = 60 seconds for one iteration. And here in our case each user has to perform 1000/10 = 100 iterations in hour.

So with current think time & without pacing , each vuser will do 3600/60 = 60 iterations per hour. In order to target 100 iterations , each iteration should  complete in 3600/100 = 36 seconds. So if we set 2 seconds as think time , then we will have these targets achieved – 30 + (3*2) = 36 seconds.

Its quite common for an application to cater different workloads at different point of time in a day or as per business milestones. Different workload models to should be designed & tests should be carried out to see impact on application performance.

Dealing with correlations having dynamic boundaries

Lest talk about the ways to deal with Loadrunner correlations when the boundaries are not same always.

1. In case there are digits in the Boundaries and which may change every time:

Suppose you have the response data as follows, where VALUE is the string you want to capture, but issue is that the left boundary is changing every time. You get the left boundary as axb, where x ranges between 0 and 9, as follows:
a0b=VALUExyz
a1b=VALUExyz
a2b=VALUExyz
——–
——–
a9b=VALUExyz

You can capture the desired string by putting the following correlation function in place, using the /DIG text flag in combination with LB:

web_reg_save_param(“DynamicCapture”, “LB/DIG=a#b\=”, “RB=rb”, LAST);

The corresponding place, which you expect to be dynamically filled in with a digit, should be replaced by a pound sign ( # ).

2. In case there are letters for which case may change:

If letters are changing case, you can modify the function as below to include the /IC flag:

web_reg_save_param(“DynamicCapture”, “LB/IC/DIG=a#b\=”, “RB/IC=rb”, LAST);

so the left boundary will match to a1b and A1B as well. 

3. In case of multiple digits:

if there are multiple digits, you have to put a pound sign (#) sing for each digit:

A0123b=”VALUE”xyz

web_reg_save_param(“DynamicCapture”, “LB/IC/DIG=a####b\=”, “RB/IC=rb”, LAST);

4.In caes if dynamic digits or letters in the boundaries:

 If there is a case, where you expect a place to be filled in dynamically by a digit or a letter, then modify the function to use /ALNUM instead of /DIG text flag, and use caret sign(^) instead of # :

web_reg_save_param(“DynamicCapture”, “LB/ALNUM=a^b\=”, “RB/IC=rb”, LAST);

5.Options for ALNUM flag:
To deal with the case while, matching alphanumeric dynamic boundaries, there are three versions of /ALNUM flag as –
ALNUMIC to ignore case.
ALNUMLC to match only lower case.
ALNUMUC to match only upper case.