Resource Utilization Monitoring [Linux]

Let’s check what all metrics we can collect & how these values help us understand the system load.

We will take examples from Linux OS with the help of utilities like NMON (it’s free tool).

Using ‘r’ option in NMON, we see the Linux and CPU details for the machine.

e.g.

Linux: Linux version 2.6.32-431.el6.x86_64 (mockbuild@ca-build44.us.oracle.com)

Build: (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) )

Release  : 2.6.32-431.el6.x86_64

Version  : #1 SMP Wed Nov 20 23:56:07 PST 2013

cpuinfo: model name : Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz

cpuinfo: vendor_id : GenuineIntel

cpuinfo: cpu MHz  : 2599.973

cpuinfo: wp  : yes

# of CPUs: 4

Machine  : x86_64

/etc/*ease[1]: LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-n

/etc/*ease[2]: arch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:print

/etc/*ease[3]: ng-4.0-noarch                                                      

/etc/*ease[4]: Oracle Linux Server release 6.5                                    

lsb_release: Distributor ID:        OracleServer                                   

lsb_release: Description:   Oracle Linux Server release 6.5                       

lsb_release: Release:       6.5                                                   

lsb_release: Codename:      n/a     

    A.      CPU Utilization

First of all, we can check CPU architecture details using command ‘lscpu’.

It gives you details of CPU family, no of CPUs , no of Cores, therads, socktes, speed etc.

e.g. ]$ lscpu

Architecture:          x86_64

CPU op-mode(s):        32-bit, 64-bit

Byte Order:            Little Endian

CPU(s):                4

On-line CPU(s) list:   0-3

Thread(s) per core:    1

Core(s) per socket:    4

Socket(s):             1

NUMA node(s):          1

Vendor ID:             GenuineIntel

CPU family:            6

Model:                 63

Stepping:              2

CPU MHz:               2599.973

BogoMIPS:              5199.94

Hypervisor vendor:     Microsoft

Virtualization type:   full

L1d cache:             32K

L1i cache:             32K

L2 cache:              256K

L3 cache:              25600K

NUMA node0 CPU(s):     0-3

Or you can similar info in ‘proc/cpuinfo’ file. It shows info processor wise. I have copied info for only first processor out of total 4.

e.g. $ cat /proc/cpuinfo

processor       : 0

vendor_id       : GenuineIntel

cpu family      : 6

model           : 63

model name      : Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz

stepping        : 2

cpu MHz         : 2599.973

cache size      : 25600 KB

physical id     : 0

siblings        : 4

core id         : 0

cpu cores       : 4

apicid          : 0

initial apicid  : 0

fpu             : yes

fpu_exception   : yes

cpuid level     : 13

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good unfair_spinlock pni cx16 hypervisor lahf_lm

bogomips        : 5199.94

clflush size    : 64

cache_alignment : 64

address sizes   : 42 bits physical, 48 bits virtual

power management:

Now, once we understand CPU architecture, lets have a look at CPU utilization using NMON utility. For this, on command open it by typing ‘nmon’ and hit ‘c’.

e.g.  here it showing separate stats for each CPU and also showing average stats.

CPU  User%  Sys% Wait% Idle

  1  33.0  11.2   0.0  55.8

  2   7.7   0.0   0.0  92.3

  3  19.9   0.5   0.0  79.6

  4  29.2   3.6   0.0  67.2

---------------------------

Avg  22.4   3.8   0.0  73.7

User% - used by user processes.(usage by your application processes)

Sys% - used by Kernel processes.

Wait% - waiting for I/O to finish. (high percentage can indicate I/O issues)

Idle% - waiting for idle for next tasks to be executed.

   B.      Memory:

If we check memory utilization using NMON or top on Linux, it will always show free  memory percentage as very low, giving you impression that all the memory is being used & you need more memory. But Linux is borrowing unused memory for disk caching. If your applications want more memory, they just take back a chunk that the disk cache borrowed. Disk cache can always be given back to applications immediately.

We can check this using free –m command.

e.g. NMON is showing

                RAM  

x Total MB        80575.8

x Free  MB        1355.7

x Free Percent    1.7%

when I hit free –m

$ free -m

             total       used       free     shared    buffers     cached

Mem:         80575      79218       1357          0       2168      23596

-/+ buffers/cache:      53453      27122

Swap:         4095        122       3973

So here its showing 1357 MB of free memory but out of memory used for caching 26.4 GB memory is free which can be allocated to processes as required.

     C.      I/O

As mentioned in CPU stats ‘wait%’ for CPU indicates waiting on IO resources.

Then we can check I/O stats.

e.g. using ‘d’ option for NMON, we can see busy% , reads and Writes (KB per seconds) per disk and on average.

DiskName Busy  Read WriteKB

sda        0%    0.0    0.0

sda1       0%    0.0    0.0

sdb        0%    0.0    0.0

dm-0       0%    0.0    0.0

dm-1       0%    0.0    0.0

dm-2       0%    0.0    0.0

dm-3       0%    0.0    0.0

dm-4       0%    0.0    0.0

Totals Read-MB/s=0.0      Writes-MB/s=0.0      Transfers/sec=0.0

Using iostat:

e.g. $ iostat –xtc

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           5.93    0.00    0.69    0.08    0.00   93.29

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util

sda               0.02     2.84    0.05    2.36     3.38    41.58    18.60     0.00    0.48   0.15   0.04

sdb               1.76    80.30    1.69   53.15   112.90   831.04    17.21     0.05    0.94   0.13   0.71

dm-0              0.00     0.00    0.01    0.10     0.45     0.80    11.41     0.00    1.99   0.09   0.00

dm-1              0.00     0.00    0.03    0.09     0.21     0.75     8.00     0.00    2.55   0.18   0.00

dm-2              0.00     0.00    0.06    0.08     1.19     0.65    12.71     0.00    1.32   0.54   0.01

dm-3              0.00     0.00    0.49    1.30     5.24    10.39     8.74     0.01    4.85   0.13   0.02

dm-4              0.00     0.00    2.87  131.72   105.80   818.45     6.87     0.57    4.22   0.05   0.68

columns explained:

rrqm/s

The number of read requests merged per second that were queued to the device.

wrqm/s

The number of write requests merged per second that were queued to the device.

r/s

The number of read requests that were issued to the device per second.

w/s

The number of write requests that were issued to the device per second.

rsec/s

The number of sectors read from the device per second.

wsec/s

The number of sectors written to the device per second.

avgrq-sz

The average size (in sectors) of the requests that were issued to the device.

avgqu-sz

The average queue length of the requests that were issued to the device.

await

The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.

svctm

The average service time (in milliseconds) for I/O requests that were issued to the device. Warning! Do not trust this field any more. This field will be removed in a future sysstat version.

%util

Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.