Linux Performance monitoring using SAR

linux

Posted By: admin May 8, 2018

The tool sar is a very powerful tool that can be used to monitor performance of various Linux components such as CPU, Memory, IO, etc in real-time, and also collect data on an on-going basis and save them for later analysis.

The tool sar is part of the sysstat pacakges available in various distrubution of Linux systems such as Redhat, Ubuntu, etc. In this example we will be using Redhat Linux RHEL 7.

  • Install sar using the following command:
    [root@gw16-lap-devops ~]# yum install sysstat -y

 

  • Verify the version of sar:
    [root@gw16-lap-devops ~]# sar -V
    sysstat version 10.1.5
    (C) Sebastien Godard (sysstat <at> orange.fr)

 

  • The following command gives the CPU usage 3 times, within an interval of 1 second each:
    [root@gw16-lap-devops ~]# sar 1 3
    Linux 3.10.0-514.el7.x86_64 (gw16-lap-devops) 05/07/2018 _x86_64_ (1 CPU)
    07:56:58 PM CPU %user %nice %system %iowait %steal %idle
    07:56:59 PM all 84.16 0.00 15.84 0.00 0.00 0.00
    07:57:00 PM all 77.66 0.00 18.09 0.00 0.00 4.26
    07:57:01 PM all 0.00 0.00 0.00 0.00 0.00 100.00
    Average: all 53.56 0.00 11.19 0.00 0.00 35.25

 

Collecting historical data

Since I have sa1 application stored in “/usr/lib64/sa/“, the following command will collect sar data one time with an interval of 1 second, and store it in a file “/var/log/sa/sa<day of the month>“, in binary format. Therefore if you run this command using a cron job every 5 minutes, you can later analyse this data that was gathered every 5 minutes.

[root@gw16-lap-devops ~]# /usr/lib64/sa/sa1 1 1
The content of above command cannot be viewed as normal ASCII file

The following command will collect the daily report once between an interval of 1 second, and store the result in binary format in the file “/var/log/sa/sar<day of the month>”

[root@gw16-lap-devops ~]# /usr/lib64/sa/sa2 1 1
The result of above command is stored in an ASCII file

  • The below command can give information about the host CPU using the “-u” switch, similar to “sar 1 3”. The “%idle” column will show the CPU load:
    [root@gw16-lap-devops ~]# sar -u
    Linux 3.10.0-514.el7.x86_64 (gw16-lap-devops) 05/07/2018 _x86_64_ (1 CPU)
    08:18:04 PM CPU %user %nice %system %iowait %steal %idle
    08:18:26 PM all 0.00 0.00 0.05 0.00 0.00 99.95
    08:19:14 PM all 0.02 0.00 0.02 0.00 0.00 99.96
    08:20:01 PM all 0.04 0.00 0.04 0.00 0.00 99.91
    Average: all 0.03 0.00 0.03 0.00 0.00 99.94

 

  • The below command will display the content of the binary file “/var/log/sa/sa07” which was created by running the “sa1” command earlier:
    [root@gw16-lap-devops ~]# sar -u -f /var/log/sa/sa07
    Linux 3.10.0-514.el7.x86_64 (gw16-lap-devops) 05/07/2018 _x86_64_ (1 CPU)
    08:18:04 PM CPU %user %nice %system %iowait %steal %idle
    08:18:26 PM all 0.00 0.00 0.05 0.00 0.00 99.95
    08:19:14 PM all 0.02 0.00 0.02 0.00 0.00 99.96
    08:20:01 PM all 0.04 0.00 0.04 0.00 0.00 99.91
    08:27:35 PM all 0.01 0.00 0.00 0.00 0.00 99.99
    Average: all 0.01 0.00 0.01 0.00 0.00 99.98
  • The following command is same as the command “sar -u” with additional information.
    [root@gw16-lap-devops ~]# sar -u ALL
  • The following command displays information about all CPUs in a host. Each CPU number is mentioned in each row, in the column CPU, starting from 0 (zero). Below is the report of a system that has just one CPU:

    [root@gw16-lap-devops ~]# sar -P ALL 1 1
    Linux 3.10.0-514.el7.x86_64 (gw16-lap-devops) 05/07/2018 _x86_64_ (1 CPU)
    08:31:48 PM CPU %user %nice %system %iowait %steal %idle
    08:31:49 PM all 0.00 0.00 0.00 0.00 0.00 100.00
    08:31:49 PM 0 0.00 0.00 0.00 0.00 0.00 100.00
    Average: CPU %user %nice %system %iowait %steal %idle
    Average: all 0.00 0.00 0.00 0.00 0.00 100.00
    Average: 0 0.00 0.00 0.00 0.00 0.00 100.00

    In the above example we can replace “-P ALL” with “-P 0”, “-P 1”, etc
  • Monitoring the memory of a system using the “-r” switch. The below command collects memory statistics 3 times within an interval of 1 second:

    [root@gw16-lap-devops ~]# sar -r 1 3
    Linux 3.10.0-514.el7.x86_64 (gw16-lap-devops) 05/08/2018 _x86_64_ (1 CPU)
    10:15:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
    10:15:02 AM 79044 421676 84.21 880 109360 447928 28.91 291028 60380 5232
    10:15:03 AM 265272 235448 47.02 880 110860 260020 16.78 104860 60380 6732
    10:15:04 AM 265272 235448 47.02 880 110860 260020 16.78 104860 60380 6732
    Average: 203196 297524 59.42 880 110360 322656 20.83 166916 60380 6232

 

  • The sar command with “-S” switch is used to get statistics of Swap space.
    [root@gw16-lap-devops ~]# sar -S 1 3
    Linux 3.10.0-514.el7.x86_64 (gw16-lap-devops) 05/08/2018 _x86_64_ (1 CPU)
    10:09:44 AM kbswpfree kbswpused %swpused kbswpcad %swpcad
    10:09:45 AM 1048572 0 0.00 0 0.00
    10:09:46 AM 1048572 0 0.00 0 0.00
    10:09:47 AM 1048572 0 0.00 0 0.00
    Average: 1048572 0 0.00 0 0.00

 

  • The below command read the binary file sa07 and displays the content related to memory
    [root@gw16-lap-devops ~]# sar -r -f /var/log/sa/sa07
    Linux 3.10.0-514.el7.x86_64 (gw16-lap-devops) 05/07/2018 _x86_64_ (1 CPU)
    08:18:04 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
    08:18:26 PM 165488 335232 66.95 0 172376 260456 16.81 84144 147540 8
    08:19:14 PM 165520 335200 66.94 0 172384 260456 16.81 84148 147540 0
    08:20:01 PM 164740 335980 67.10 0 172424 262388 16.94 84620 147540 0
    08:27:35 PM 165468 335252 66.95 0 172392 260456 16.81 84148 147540 0
    08:30:01 PM 164712 336008 67.10 0 172436 262388 16.94 84612 147540 0
    08:40:01 PM 150680 350040 69.91 0 182800 262420 16.94 93464 149068 0
    08:50:01 PM 150548 350172 69.93 0 183016 262420 16.94 93508 149272 0
    09:00:01 PM 150552 350168 69.93 0 183020 262420 16.94 93504 149268 0
    09:10:01 PM 150452 350268 69.95 0 183024 262420 16.94 93512 149252 0
    09:20:01 PM 151060 349660 69.83 0 183004 261880 16.90 93264 149244 0
    09:30:01 PM 143228 357492 71.40 0 191172 261880 16.90 94624 156064 0
    09:40:01 PM 143088 357632 71.42 0 191180 261880 16.90 94616 156068 0
    09:50:01 PM 143008 357712 71.44 0 191188 261880 16.90 94628 156064 0
    Average: 154503 346217 69.14 0 180801 261796 16.90 90215 150154 1

 

  • The sar command with “-b” switch is used to get statistics of I/O activities.
    [root@gw16-lap-devops ~]# sar -b 1 3
    Linux 3.10.0-514.el7.x86_64 (gw16-lap-devops) 05/08/2018 _x86_64_ (1 CPU)
    10:16:31 AM tps rtps wtps bread/s bwrtn/s
    10:16:32 AM 5821.78 0.00 5821.78 0.00 46897.03
    10:16:33 AM 5481.19 0.00 5481.19 0.00 44222.77
    10:16:34 AM 5748.96 0.00 5748.96 0.00 46302.08
    Average: 5682.89 0.00 5682.89 0.00 45798.99

 

  • The sar command with “-q” switch is used to get statistics of average load.
    [root@gw16-lap-devops ~]# sar -q 1 3
    Linux 3.10.0-514.el7.x86_64 (gw16-lap-devops) 05/08/2018 _x86_64_ (1 CPU)
    10:18:16 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
    10:18:17 AM 1 110 2.15 0.65 0.26 3
    10:18:18 AM 2 110 2.15 0.65 0.26 2
    10:18:19 AM 1 110 2.14 0.67 0.27 3
    Average: 1 110 2.15 0.66 0.26 3

 

  • The sar command with “-n” switch is used to get statistics of various network activities.
    sar -n <network activity>The network activity can be one of the following:
    DEV – Displays network devices vital statistics for eth0, eth1, etc.,
    EDEV – Display network device failure statistics
    UDP – Displays UDPv4 network traffic
    NFS – Displays NFS client activities
    ICMP – Displays ICMPv4 network traffic
    EICMP – Displays ICMPv4 network errors
    NFSD – Displays NFS server activities
    SOCK – Displays sockets in use for IPv4
    IP – Displays IPv4 network traffic
    EIP – Displays IPv4 network errors
    TCP – Displays TCPv4 network traffic
    ETCP – Displays TCPv4 network errors
    SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
    ALL – This displays all of the above information. The output will be very long.

    [root@gw16-lap-devops ~]# sar -n DEV 1 3
    Linux 3.10.0-514.el7.x86_64 (gw16-lap-devops) 05/08/2018 _x86_64_ (1 CPU)
    10:23:05 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
    10:23:06 AM ens32 3.19 3.19 0.34 0.36 0.00 0.00 0.00
    10:23:06 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    10:23:06 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
    10:23:07 AM ens32 1.32 1.32 0.08 0.50 0.00 0.00 0.00
    10:23:07 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    10:23:07 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
    10:23:08 AM ens32 1.19 1.19 0.07 0.45 0.00 0.00 0.00
    10:23:08 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
    Average: ens32 1.97 1.97 0.17 0.43 0.00 0.00 0.00
    Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00