Thursday, April 14, 2016

System Performance monitoring - using vmstat command

One major aspect of monitoring a performance test is to capture system performance metrics and analyze it.
When monitoring machine level or OS level resource utilization, at high level can be categorized in
      ·         CPU Utilization
      ·         Memory Utilization
      ·         I/O stats

There are multiple commands/utilities available in different flavors or Unix/Linux. Let’s check vmstat today!

vmstat – this command displays virtual memory statistics.
e.g.
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd       free          buff         cache       si   so    bi    bo    in   cs us sy id wa st
 0  0 102716 4587628 2528320 18105548    0    0    18   114    4    5  6  1 93  0  0

Processes
r - The number of processes waiting for run time.
     When this number exceeds the number of CPUs on the server, a CPU bottleneck exists, and some tasks are waiting for execution.
b - The average number of kernel threads on the wait queue at one-second intervals. (awaiting resource, awaiting input/output). Kernel threads are placed on the wait queue when scheduled for execution and are waiting for one of their process pages to be paged in.

memory
swpd - the amount of virtual memory used
buff: the amount of memory used as buffers.
cache: the amount of memory used as cache.
inact: the amount of inactive memory. (-a option)
active: the amount of active memory. (-a option)

swap
si: Amount of memory swapped in from disk (/s).
so: Amount of memory swapped to disk (/s).

 Swapping the memory pages to the swap file will be seen in the so (swap out - memory swapped to disk).
  

IO
bi: Blocks received from a block device (blocks/s).
bo: Blocks sent to a block device (blocks/s).

High values for these shows the I/O operations happening.  Those could be related to swapping as well.

System
       in: The number of interrupts per second, including the clock.
       cs: The number of context switches per second.

CPU
       These are percentages of total CPU time.
       us: Time spent running non-kernel code. (user time)
       sy: Time spent running kernel code. (system time)
       id: Time spent idle.
       wa: Time spent waiting for IO.
       st: Time stolen from a virtual machine.

High percentage for ‘wa’ indicates, there is issue with I/O. sometimes it’s classified as "waiting on I/O". A wa value over 40 percent could indicate that the disk subsystem may not be balanced properly, or it may be the result of a disk-intensive workload.

Using with modes:
There are additional modes for vmstat which can be used to get detailed information about specific area:

$ vmstat --help
usage: vmstat [-V] [-n] [delay [count]]
              -V prints version.
              -n causes the headers not to be reprinted regularly.
              -a print inactive/active page stats.
              -d prints disk statistics
              -D prints disk table
              -p prints disk partition statistics
              -s prints vm table
              -m prints slabinfo
              -t add timestamp to output
              -S unit size

Providing sampling time
for capturing the statistics during performance test, we can specify the sampling interval & no of invocations.
e.g.  displays statistics at interval of 1 second for 5 times .
vmstat -t 1 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ ---timestamp---
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 102648 4220480 2537344 18213972    0    0    18   114    7    0  6  1 93  0  0    2016-04-14 06:17:19 EDT
 0  0 102648 4220472 2537344 18213972    0    0     0   741 4468 10644  1  0 99  0  0   2016-04-14 06:17:20 EDT
 0  0 102648 4220472 2537348 18213968    0    0     0    61 4386 11545  1  0 99  0  0   2016-04-14 06:17:21 EDT
 0  0 102648 4220456 2537348 18213972    0    0     0     6 4588 12701  3  0 97  0  0   2016-04-14 06:17:22 EDT
 0  0 102648 4220448 2537348 18213988    0    0     8   248 4522 12081  1  0 98  1  0   2016-04-14 06:17:23 EDT

Can further redirect the statistics to a file and later on open file and analyze it.


e.g. $ vmstat -t 1 5 >> vmstat_output.txt

No comments:

Post a Comment