The Sysstat Suite of Resource Monitoring Tools

2.5.4. The Sysstat Suite of Resource Monitoring Tools

While the previous tools may be helpful for gaining more insight into system performance over very short time frames, they are of little use beyond providing a snapshot of system resource utilization. In addition, there are aspects of system performance that cannot be easily monitored using such simplistic tools.

Therefore, a more sophisticated tool is necessary. Sysstat is such a tool.

Sysstat contains the following tools related to collecting I/O and CPU statistics:

iostat

Displays an overview of CPU utilization, along with I/O statistics for one or more disk drives.
mpstat

Displays more in-depth CPU statistics.

Sysstat also contains tools that collect system resource utilization data and create daily reports based on that data. These tools are:

sadc

Known as the system activity data collector, sadc collects system resource utilization information and writes it to a file.
sar

Producing reports from the files created by sadc, sar reports can be generated interactively or written to a file for more intensive analysis.

The following sections explore each of these tools in more detail.
2.5.4.1. The iostat command

The iostat command at its most basic provides an overview of CPU and disk I/O statistics:

Linux 2.4.20-1.1931.2.231.2.10.ent (pigdog.example.com) 07/11/2003

avg-cpu: %user %nice %sys %idle
6.11 2.56 2.15 89.18

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev3-0 1.68 15.69 22.42 31175836 44543290

Below the first line (which contains the system’s kernel version and hostname, along with the current date), iostat displays an overview of the system’s average CPU utilization since the last reboot. The CPU utilization report includes the following percentages:

*

Percentage of time spent in user mode (running applications, etc.)
*

Percentage of time spent in user mode (for processes that have altered their scheduling priority using nice(2))
*

Percentage of time spent in kernel mode
*

Percentage of time spent idle

Below the CPU utilization report is the device utilization report. This report contains one line for each active disk device on the system and includes the following information:

*

The device specification, displayed as dev-sequence-number, where is the device’s major number[1], and is a sequence number starting at zero.
*

The number of transfers (or I/O operations) per second.
*

The number of 512-byte blocks read per second.
*

The number of 512-byte blocks written per second.
*

The total number of 512-byte blocks read.
*

The total number of 512-byte block written.

This is just a sample of the information that can be obtained using iostat. For more information, refer to the iostat(1) man page.
2.5.4.2. The mpstat command

The mpstat command at first appears no different from the CPU utilization report produced by iostat:

Linux 2.4.20-1.1931.2.231.2.10.ent (pigdog.example.com) 07/11/2003

07:09:26 PM CPU %user %nice %system %idle intr/s
07:09:26 PM all 6.40 5.84 3.29 84.47 542.47

With the exception of an additional column showing the interrupts per second being handled by the CPU, there is no real difference. However, the situation changes if mpstat’s -P ALL option is used:

Linux 2.4.20-1.1931.2.231.2.10.ent (pigdog.example.com) 07/11/2003

07:13:03 PM CPU %user %nice %system %idle intr/s
07:13:03 PM all 6.40 5.84 3.29 84.47 542.47
07:13:03 PM 0 6.36 5.80 3.29 84.54 542.47
07:13:03 PM 1 6.43 5.87 3.29 84.40 542.47

On multiprocessor systems, mpstat allows the utilization for each CPU to be displayed individually, making it possible to determine how effectively each CPU is being used.
2.5.4.3. The sadc command

As stated earlier, the sadc command collects system utilization data and writes it to a file for later analysis. By default, the data is written to files in the /var/log/sa/ directory. The files are named sa

, where

is the current day’s two-digit date.

sadc is normally run by the sa1 script. This script is periodically invoked by cron via the file sysstat, which is located in /etc/cron.d/. The sa1 script invokes sadc for a single one-second measuring interval. By default, cron runs sa1 every 10 minutes, adding the data collected during each interval to the current /var/log/sa/sa

file.
2.5.4.4. The sar command

The sar command produces system utilization reports based on the data collected by sadc. As configured in Red Hat Enterprise Linux, sar is automatically run to process the files automatically collected by sadc. The report files are written to /var/log/sa/ and are named sar

, where

is the two-digit representations of the previous day’s two-digit date.

sar is normally run by the sa2 script. This script is periodically invoked by cron via the file sysstat, which is located in /etc/cron.d/. By default, cron runs sa2 once a day at 23:53, allowing it to produce a report for the entire day’s data.
2.5.4.4.1. Reading sar Reports

The format of a sar report produced by the default Red Hat Enterprise Linux configuration consists of multiple sections, with each section containing a specific type of data, ordered by the time of day that the data was collected. Since sadc is configured to perform a one-second measurement interval every ten minutes, the default sar reports contain data in ten-minute increments, from 00:00 to 23:50[2].

Each section of the report starts with a heading describing the data contained in the section. The heading is repeated at regular intervals throughout the section, making it easier to interpret the data while paging through the report. Each section ends with a line containing the average of the data reported in that section.

Here is a sample section sar report, with the data from 00:30 through 23:40 removed to save space:

00:00:01 CPU %user %nice %system %idle
00:10:00 all 6.39 1.96 0.66 90.98
00:20:01 all 1.61 3.16 1.09 94.14
…
23:50:01 all 44.07 0.02 0.77 55.14
Average: all 5.80 4.99 2.87 86.34

In this section, CPU utilization information is displayed. This is very similar to the data displayed by iostat.

Other sections may have more than one line’s worth of data per time, as shown by this section generated from CPU utilization data collected on a dual-processor system:

00:00:01 CPU %user %nice %system %idle
00:10:00 0 4.19 1.75 0.70 93.37
00:10:00 1 8.59 2.18 0.63 88.60
00:20:01 0 1.87 3.21 1.14 93.78
00:20:01 1 1.35 3.12 1.04 94.49
…
23:50:01 0 42.84 0.03 0.80 56.33
23:50:01 1 45.29 0.01 0.74 53.95
Average: 0 6.00 5.01 2.74 86.25
Average: 1 5.61 4.97 2.99 86.43

There are a total of seventeen different sections present in reports generated by the default Red Hat Enterprise Linux sar configuration; some are explored in upcoming chapters. For more information about the data contained in each section, refer to the sar(1) man page.

Leave a Reply

Your email address will not be published. Required fields are marked *