Managing system performance

Collecting system activity data on demand

The sar(1M) command can be used either to gather system activity data itself or to extract what has been collected in the daily activity files created by sa1 and sa2.

The sar command has the following formats:

   sar [-P processor_id[, . . . ] | ALL] [-ubdycwaqvtmpgrkAR]


   	[-o file] t [n]
   
   sar [-P processor_id[, . . . ] | ALL] [-ubdycwaqvtmpgrkAR]


   	[-s time]	[-e time] [-i sec] [-f file]

In the first format, sar samples cumulative activity counters in the operating system at intervals specified by n for a time (in seconds) specified by t (t should be 5 seconds or greater). The default value of n is 1. If the -o option is specified, samples are saved in file in binary format.

In the second format, with no sampling interval specified, sar extracts data from a previously recorded file, either the one specified by the -f option or, by default, the standard daily activity file, /var/adm/sa/sardd. The -s and -e options define the starting and ending times for the report. The starting and ending times are of the form hh[:mm[:ss]]. The -i option specifies, in seconds, the intervals to select records. If the -i option is not included, all intervals found in the daily activity file are reported.

You can use the -P option to specify collection of data for particular processors rather than for system-wide activity.

The following options are available with sar:

option report to determine:

-a file access operations amount of inode and directory activity

-b buffer activity the effectiveness of system buffering and the appropriateness of the value of the BUFHWM system parameter

-c system call activity the percentage of read and write system calls

-d disk activity disk slowdowns

-g page-out and memory freeing whether you have a memory shortage

-k kernel memory allocation if you make special use of KMA, if it is too large a portion of total memory

-m interprocess communication the number of messages and semaphore calls

-p page-in and fault activity whether you have a memory shortage

-q queue activity whether you have a memory shortage

-r unused memory whether you have a memory shortage

-t inode or DNLC activity by filesystem type if inode limits should be changed so that more inodes and pages can be kept in memory and reused

-u CPU utilization if there are tuning problems or memory constraints

-v system table status if system tables need to be made larger

-w swapping and switching volume whether you have a memory shortage

-y asynchronous terminal activity determine if there are any bad lines

-A report overall system performance, same as entering all options view overall system performance

-P report system activity for specified processors only view performance of a specific processor or distribution among processors

-R report raw data values view the values of the counters from which metrics are computed, rather than computed values

option	report	to determine:
-a	file access operations	amount of inode and directory activity
-b	buffer activity	the effectiveness of system buffering and the appropriateness of the value of the BUFHWM system parameter
-c	system call activity	the percentage of read and write system calls
-d	disk activity	disk slowdowns
-g	page-out and memory freeing	whether you have a memory shortage
-k	kernel memory allocation	if you make special use of KMA, if it is too large a portion of total memory
-m	interprocess communication	the number of messages and semaphore calls
-p	page-in and fault activity	whether you have a memory shortage
-q	queue activity	whether you have a memory shortage
-r	unused memory	whether you have a memory shortage
-t	inode or DNLC activity by filesystem type	if inode limits should be changed so that more inodes and pages can be kept in memory and reused
-u	CPU utilization	if there are tuning problems or memory constraints
-v	system table status	if system tables need to be made larger
-w	swapping and switching volume	whether you have a memory shortage
-y	asynchronous terminal activity	determine if there are any bad lines
-A	report overall system performance, same as entering all options	view overall system performance
-P	report system activity for specified processors only	view performance of a specific processor or distribution among processors
-R	report raw data values	view the values of the counters from which metrics are computed, rather than computed values

When displaying data, sar fits numbers within their allocated field widths by changing output formats.

if the number exceeds the output field width and it has decimal places, sar drops the decimal places.
if removing decimal places does not make the number fit, sar prints the number in ``shortened exponential notation''. For example, if the number is ``123456578'' and the field width is 6, sar prints ``1.23e7''.

When absolute numbers are mentioned in the following descriptions of the sar command options, they are correct for Intel486/33 processors. Faster processors can support more activity per second.

Checking file access with sar -a

The sar -a option reports on the use of file access operations. The following system routines are reported:

iget/s: Number of s5, ufs, vxfs, and sfs files located by inode entry per second.
namei/s: Number of filesystem path searches per second. If the kernel does not find a directory name in the directory name lookup cache, it will perform an iget operation to get the directory. Hence, most igets are the result of directory name lookup cache misses.
dirbk/s: Number of s5, ufs, vxfs, and sfs directory block reads issued per second.
%dnlc: Hit rate percentage of the directory name lookup cache.
If -R is specified then ``%dnlc'' is replaced by ``dnlc-hits'' and ``dnlc-misses'', the counts of cache hits and misses.

The following is an example of sar -a output.

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 iget/s namei/s dirbk/s %dnlc 01:00:00 0 1 0 97 02:00:00 0 1 0 97 03:00:00 0 1 0 97 04:00:00 0 1 0 97 05:00:00 0 1 0 97 06:00:00 0 1 0 97 07:00:00 17 49 24 62 08:00:00 0 1 0 97 . . . 14:00:00 0 1 0 97 14:20:00 0 3 0 93 14:40:00 0 3 0 90 Average 1 5 1 73

The larger the values reported, the more time the system kernel is spending to access files. The amount of time reflects how heavily programs and applications are using the filesystem(s). The -a option is helpful for understanding how disk-dependent an application system is. It might be useful in tuning your directory name lookup cache (dnlc) size; change the value of the filesystem system parameter DNLCSIZE if %dnlc is often low. (That is, when %dnlc is low other than when a large number of files are accessed for the first time in a while.)

Checking buffer activity with sar -b

The buffer caches filesystem meta-data about inodes, indirect blocks, super-blocks and so on. The -b option reports on the following buffer activities:

bread/s: Average number of physical block (512 bytes each) reads into the system buffers from the disk per second. (Note that the system buffers are only used for inodes, indirect blocks, and other filesystem internal information. Filesystem data blocks go through the paging system.)
lread/s: Average number of logical reads from system buffers per second.
%rcache: Fraction of logical reads found in the system buffers (100 percent minus the ratio of breads to lreads).
bwrit/s: Average number of physical writes from the system buffers to disk per second.
lwrit/s: Average number of logical writes to system buffers per second.
%wcache: Fraction of logical writes found in the system buffers (100 percent minus the ratio of bwrit/s to lwrit/s).
pread/s: Average number of physical read requests per second.
pwrit/s: Average number of physical write requests per second.

If %rcache falls below 90, or if %wcache falls below 60, it might be possible to improve performance by increasing the buffer space by adjusting the buffer cache system parameter BUFHWM. Check again when running the new system to be sure they are still acceptable values.

If the -R option is specified, the %rcache and %wcache columns are not displayed.

The following is an example of sar -b output:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrite/s 01:00:00 0 1 100 0 0 2 0 0 02:00:00 0 1 100 0 0 2 0 0 03:00:01 0 0 100 0 0 3 0 0 04:00:00 0 1 100 0 0 2 0 0 05:00:00 0 1 100 0 0 2 0 0 06:00:00 0 1 100 0 0 2 0 0 07:00:01 1 41 97 1 1 32 0 0 08:00:00 0 1 100 0 0 2 0 0 . . . 14:00:00 0 1 100 0 0 1 0 0 14:20:00 0 1 100 1 1 18 0 0 14:40:00 0 2 96 1 1 11 0 0 Average 0 3 97 0 1 9 0 0

Checking system calls with sar -c

The -c option reports on system calls in the following categories:

scall/s: All types of system calls per second (generally about 30 per second on a busy four- to six-user system).
sread/s: read system calls per second.
swrit/s: write system calls per second.
fork/s: fork system calls per second (about 0.5 per second on a four- to six-user system). This number increases if shell scripts are running.
lwpcr/s: lwpcreate system calls made per second.
exec/s: exec system calls per second. [If (exec/s) / (fork/s) is greater than 3, look for inefficient PATH variables. If the ratio is good, there still might be inefficient PATH variables. Shells use either exec(2) or access(2) to search the PATH. For example, while sh(1) uses access, other shells may not.]
rchar/s: Characters (bytes) transferred by read system calls per second.
wchar/s: Characters (bytes) transferred by write system calls per second.

Typically, reads plus writes account for about half the total system calls, although the percentage varies greatly with the activities being performed by the system.

This is an example of sar -c output:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 scall/s sread/s swrit/s fork/s lwpcr/s exec/s rchar/s wchar/s 01:00:00 14 1 0 0.04 0.00 0.01 777 47 02:00:00 14 1 0 0.04 0.00 0.01 771 46 03:00:01 14 1 0 0.04 0.00 0.01 778 47 04:00:00 14 1 0 0.04 0.00 0.01 777 47 05:00:00 14 1 0 0.04 0.00 0.01 796 47 06:00:00 14 1 0 0.04 0.00 0.01 780 47 07:00:01 237 13 6 2.51 0.00 2.80 4858 471 08:00:00 14 1 0 0.04 0.00 0.01 779 47 . . . 14:00:00 14 1 0 0.04 0.00 0.01 771 52 14:20:00 31 2 1 0.09 0.00 0.06 1081 389 14:40:00 23 2 2 0.10 0.00 0.08 4550 418 Average 31 2 1 0.22 0.00 0.21 1215 109

Checking disk activity with sar -d

The sar -d option reports the activities of disk devices.

device: Name of the disk device(s) monitored.
%busy: Percentage of time the device spent servicing a transfer request.
avque: The average number of requests outstanding during the monitored period (measured only when the queue was occupied).
r+w/s: Number of read and write transfers to the device per second.
blks/s: Number of 512-byte blocks transferred to the device per second.
avwait: Average time, in milliseconds, that transfer requests wait idle in the queue (measured only when the queue is occupied).
avserv: Average time, in milliseconds, for a transfer request to be completed by the device. (For disks, this time includes seek, rotational latency, and data transfer times.)

When the -R option is specified, the columns avque, avwait, and avserv are not displayed. An additional column, busy, the total time the disk was active, is displayed.

The following is an example of sar -d output:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 device %busy avque r+w/s blks/s avwait avserv 01:00:00 sd011 0 1.3 0 7 3.7 12.9 02:00:00 sd011 0 1.3 0 6 3.6 12.6 03:00:01 sd011 0 1.3 0 6 3.2 12.6 04:00:00 sd011 0 1.3 0 7 3.7 13.1 05:00:00 sd011 0 1.4 0 6 4.5 12.5 06:00:00 sd011 0 1.3 0 7 3.6 13.1 07:00:01 sd011 3 1.7 3 41 6.9 10.5 08:00:00 sd011 0 1.6 0 6 6.2 11.1 08:00:00 sd011 0 1.6 0 6 6.2 11.1 . . . 14:00:00 sd011 0 1.5 0 7 6.0 11.4 14:20:00 sd011 1 1.6 1 15 7.0 11.6 14:40:00 sd011 1 1.8 1 21 8.7 10.9 Average sd011 0 1.5 0 10 5.8 11.5

Note that queue lengths and wait and service times are measured when there is something in the queue. If %busy is small, large queues and service times probably represent the periodic sync efforts by the system to ensure altered blocks are written to the disk in a timely fashion.

Checking page-out and memory freeing activity with sar -g

The sar -g option reports page-out and memory freeing activities as follows:

pgout/s: The number of times per second that the system performed page-out operations.
ppgout/s: The number of pages that are paged out per second. (A single page-out operation may involve the paging out of multiple pages.)
vfree/s: The number of virtual pages per second that are placed on the free list.
pfree/s: The number of physical pages per second that are placed on the free list by recycling policies of the kernel. If this value is greater than 5, more memory may improve performance. (This is the same as rclm/s, previously reported by option -p.)
vscan/s: The number of virtual pages per second scanned by recycling policies of the kernel. If this value is greater than 5, the system is spending a lot of time checking for free memory. More memory might improve performance.

The following is an example of sar -g output:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 pgout/s ppgout/s vfree/s pfree/s vscan/s 01:00:00 0.00 0.00 0.01 0.01 0.94 02:00:00 0.00 0.00 0.01 0.01 1.12 03:00:01 0.00 0.00 0.01 0.01 0.95 04:00:00 0.00 0.00 0.01 0.01 0.80 05:00:00 0.00 0.00 0.01 0.01 1.23 06:00:00 0.00 0.00 0.02 0.02 1.14 07:00:01 0.15 0.15 1.52 0.92 12.72 08:00:00 0.00 0.00 0.00 0.00 0.00 . . . 14:00:00 0.00 0.00 0.01 0.01 1.03 14:20:00 0.37 0.37 0.77 0.52 11.91 14:40:00 0.14 0.14 0.52 0.22 10.38 15:00:00 0.00 0.00 0.25 0.06 2.74 Average 0.03 0.03 0.20 0.10 2.58

The sar -g report is a good indicator of whether more memory will improve the performance of your system. If the values of vfree/s, pfree/s and vscan/s are high, or the attach rate and vflt rates from sar -p are high, then you probably have a memory shortage. sar -p, sar -u, sar -r, and sar -w are also good memory shortage indicators.

Checking kernel memory allocation activity with sar -k

The -k option reports on the following activities of the Kernel Memory Allocator (KMA).

The following information is displayed for each memory pool:

size: The size of buffers in the memory pool, or Ovsz for the oversize pool.
mem: The amount of memory in bytes KMA has for the pool.
alloc: The number of bytes allocated from this pool.
succ: The amount of memory requested by KMA customers and successfully allocated. This might be less than the alloc column since the buffers are predetermined sizes.
fail: The number of requests that were not satisfied (failed).

The KMA allows a kernel subsystem to allocate and free memory as needed. Rather than statically allocating the maximum amount of memory it is expected to require under peak load, the KMA divides requests for memory into many fixed sized pools and one oversize pool. The pools are sized from 16 bytes to 8K bytes in powers of 2. There are also pools with special sizes for system tables that have large numbers of entries whose sizes do not fit well in powers of 2. The oversized requests are satisfied by allocating memory from the system page allocator.

If your system is being used to write drivers or STREAMS that use KMA resources, then sar -k will likely prove useful; otherwise, you will probably not need the information it provides. However, if the total KMA represents a large portion of total memory, it might be an indication that the system is not properly tuned or that a locally written driver or STREAMS module has a memory leak. Any driver or module that uses KMA resources but does not specifically return the resources before it exits can create a memory leak. A memory leak will cause the amount of memory allocated by KMA to increase over time. Thus, if the alloc fields of sar -k increase steadily over time, there might be a memory leak. Another indication of a memory leak is failed requests. If many requests fail, then it is likely that a memory leak has caused KMA to be unable to reserve and allocate memory.

If it appears that a memory leak has occurred, check any locally written drivers or STREAMS that may have requested memory from KMA and have not returned it.

Checking interprocess communication with sar -m

The sar -m option reports interprocess communication activities. Message and semaphore calls are reported as follows:

msg/s: Number of message operations (sends and receives) per second.
sema/s: Number of semaphore operations per second.

An example of sar -m output follows:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 msg sema 01:00:00 0.00 0.00 02:00:00 0.00 0.00 03:00:01 0.00 0.00 04:00:00 0.00 0.00 05:00:00 0.00 0.00 06:00:00 0.00 0.00 07:00:01 0.00 0.00 08:00:00 0.00 0.00 . . . 14:00:00 0.00 0.00 14:20:00 0.00 0.35 14:40:00 0.00 0.06 15:00:00 0.00 0.00 Average 0.00 0.02

These figures will usually be zero (0.00) unless you are running applications that use messages or semaphores.

Checking page-in activity with sar -p

The sar -p option reports page-in activity, which includes protection and validation faults.

atch/s: The number of page faults per second that are satisfied by reclaiming a page currently in memory (attaches per second). Instances of this include reclaiming a page from the free list or using an instance of the page that is already mapped in some other process address space.
atfree/s: The number of page faults per second that are satisfied by a page on the free list.
atmiss/s: The number of page faults per second not fulfilled by a page in memory.
pgin/s: This field reports the number of page-in requests per second.
ppgin/s: This field reports the number of pages paged in per second. (A single page-in request, such as a software lock request (described under slock/s) or a request against a filesystem with a large block size, may involve the paging in of multiple pages.)
pflt/s: The number of page faults from protection errors per second. Instances of protection faults are invalid access to a page and ``copy-on-writes.'' Generally, this number consists primarily of copy-on-writes.
vflt/s: The number of address translation page faults per second. These are known as validity faults and occur when a valid page is not present in memory.
slock/s: This field reports the number of faults per second caused by software lock (or softlock) requests requiring physical I/O. An example of the occurrence of a softlock request is the transfer of data from a disk to memory. To ensure that the page that is to receive the data is not claimed and used by another process, it is locked by the system software.

The following is an example of sar -p output:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 atch/s atfree/s atmiss/s pgin/s ppgin/s pflt/s vflt/s slock/s 01:00:00 5.85 1.43 0.05 0.01 0.01 1.02 3.28 0.00 02:00:00 5.81 1.44 0.05 0.00 0.00 1.01 3.27 0.00 03:00:01 5.84 1.49 0.05 0.00 0.00 1.03 3.26 0.00 04:00:00 5.88 1.52 0.05 0.00 0.00 1.03 3.30 0.00 05:00:00 5.92 1.48 0.05 0.00 0.01 1.04 3.32 0.00 06:00:00 5.89 1.49 0.05 0.00 0.00 1.04 3.29 0.00 07:00:01 85.65 2.92 3.91 0.70 1.10 66.11 28.16 0.00 08:00:00 5.85 1.50 0.05 0.00 0.00 1.02 3.29 0.00 . . . 14:00:00 5.83 1.44 0.05 0.00 0.00 1.00 3.28 0.00 14:20:00 18.87 2.01 0.20 0.09 0.23 5.11 4.68 0.00 14:40:00 8.34 2.41 0.27 0.17 1.09 1.80 4.89 0.00 15:00:00 7.52 2.10 0.12 0.06 0.11 1.59 3.80 0.00 Average 12.18 1.70 0.33 0.07 0.14 5.70 5.17 0.00

If vflt/s becomes much higher than 50, look at sar -g to determine if there is a memory shortage or if the inode free list is page bound. (See sar -t for more details.) In addition, sar -u, sar -w, and sar -r can help verify whether memory is a bottleneck. In particular, a high attach rate and a high vflt rate indicates that memory is scarce.

Checking queue activity with sar -q

The sar -q option reports the average queue length while the queue is occupied and the percentage of time the queue is occupied.

prunq: The number of processes in memory and runnable, and queued for a specific processor.
%prunocc: The percentage of time processor private run queue is occupied.
runq: The number of processes waiting in memory to run. Typically, this should be less than 3 for a uniprocessor system. It might be somewhat higher on a multiprocessing system. Consistently higher values mean you are CPU-bound.
%runocc: The percentage of time the run queue is occupied.
swpq: The average number of processes in the swap queue when there were processes in the queue. If there were no processes in the swap queue, this field is blank.
%swpocc: The percentage of time during the sample that there were processes in the swap queue. If there were no processes in the swap queue, this field is blank.

If the -P option has not been entered to specify one or more processors, then the prunq and %prunocc columns are blank.

The following is an example of sar -q output:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 prunq %prunocc runq %runocc swpq %swpocc 01:00:00 1.5 0 2.0 100 02:00:00 2.8 0 2.0 100 03:00:01 3.0 0 2.0 100 04:00:00 1.2 0 2.0 100 05:00:00 1.5 0 2.1 100 06:00:00 1.5 0 2.1 100 07:00:01 1.4 3 2.1 100 1.0 0 08:00:00 2.5 0 2.0 100 . . . 14:00:00 2.0 100 14:20:00 1.5 0 2.0 100 14:40:00 6.0 0 2.2 100 15:00:00 2.0 0 2.7 100 Average 1.7 0 2.0 100 1.0 0

If %runocc is greater than 90 percent and runq-sz is greater than 3, the CPU might be heavily loaded and response may be degraded. (See sar -u.) In this case, additional CPU capacity may be required to obtain acceptable system response. If sar -p shows a large number of validity faults and sar -g shows high page-out activity, then more memory may be required.

swpq-sz and %swpocc are blank unless there are processes swapped out that are now ready to run. A swpq-sz greater than ``2'' usually indicates a memory shortage. sar -w reports the number of processes swapped in and out. The swpq-sz reports how many of those were ready to run as opposed to those waiting for an event.

Checking unused memory with sar -r

The -r option records the number of memory pages and swap file disk blocks that are currently unused.

freemem: Average number of pages of physical memory available to user processes over the intervals sampled by the command.
freeswap: Number of 512-byte disk blocks available for page swapping.

The following is an example of sar -r output:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 freemem freeswap 01:00:00 3703 22865 02:00:00 3704 22865 03:00:01 3703 22866 04:00:00 3702 22859 05:00:00 3703 22865 06:00:00 3705 22863 07:00:01 3700 22610 08:00:00 3835 22709 . . . 14:00:00 3704 22570 14:20:00 3754 22520 14:40:00 3803 22463 15:00:00 3819 22412 Average 3755 22725

Checking filesystem usage with sar -t

The -t option reports the inode usage by filesystem type. This report shows when inode limits need to be changed. The -t is a companion to the -v option, which reports the sum of all inodes in the system, as well as other system table usage.

filesystem: Filesystem type (s5, sfs/ufs, and vxfs).
inodes inuse: The current number of inode table entries being used by processes.
alloc: The current number of existing inode table entries. (This number is the sum of the number of inodes in use and the number of free inodes.)
limit: The maximum limit of inodes that can be allocated by the kernel. (This number is a soft upper limit and varies by filesystem. For the s5 filesystem, the number is statically allocated. For the ufs, sfs, and vxfs filesystems, the inodes are dynamically allocated by the kernel. alloc may slightly exceed limit.)
fail: The number of failures that occur when you're trying to give an inode to a process between sampling points. (Failures can occur when the limit is exceeded or when memory for inodes is unavailable.)
%ipf: The percentage of inodes with which reusable pages were associated that were taken off the free list by iget. These pages are flushed and cannot be reclaimed by processes. Thus, this is the percentage of igets with page flushes. If this value is greater than 20 percent, then the free list of inodes is considered page-bound and the number of inodes may need to be increased.

If the -R option is specified, the %ipf column is not displayed. Instead, the columns ipage and inopage, (equal to the counts of inodes with and without reusable pages, respectively) are displayed.

The following is an example of sar -t output:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 file system inodes inuse alloc limit fail %ipf 01:00:00 s5 0 0 0 0 100 sfs/ufs 0 0 0 0 100 vxfs 1009 1010 5125 0 0 other 0 0 0 0 100

02:00:00 s5 0 0 0 0 100 sfs/ufs 0 0 0 0 100 vxfs 1009 1010 5125 0 0 other 0 0 0 0 100 . . . 14:40:00 s5 0 0 0 0 100 sfs/ufs 0 0 0 0 100 vxfs 890 911 5125 0 30 other 0 0 0 0 100

15:00:00 s5 0 0 0 0 100 sfs/ufs 0 0 0 0 100 vxfs 879 880 5125 0 10 other 0 0 0 0 100

Average s5 0 0 0 0 100 sfs/ufs 0 0 0 0 100 vxfs 426 429 5125 0 3 other 0 0 0 0 100

Because many filesystems allocate inodes dynamically, the number of inodes alloced will often be less than the limit. Then, no inodes are recycled during a sample; therefore, making the %ipf value meaningless, because of division by zero. If the inodes in use number is consistently at the limit and the value of %ipf is higher than 20, then raising the limit may enhance system performance. As long as memory is available (see the -p, -g, -w, -q, and -u options), increasing inode limits will help keep inodes and pages in the system so they can be used again.

Checking CPU use with sar -u

The sar -u option lists CPU use. At any given moment the processor is either busy or idle. When busy, the processor is in either user or system mode, or is handling interrupts. When idle, the processor is either waiting for input or output completion or ``sitting still'' with no work to do. sar -u lists the percentage of time that the processor is in system mode, in user mode, waiting for input or output completion, interrupt handling or idle.

%usr: Percentage of time running in user mode
%sys: Percentage of time running in system mode
%wio: Percentage of time idle with some process waiting for block I/O
%idle: Percentage of time otherwise idle
%intr: Percentage of time spent in interrupt handling

The following is an example of sar -u output:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 %usr %sys %wio %idle %intr 01:00:00 0 0 0 100 0 02:00:00 0 0 0 100 0 03:00:01 0 0 0 100 0 04:00:00 0 0 0 100 0 05:00:00 0 0 0 100 0 06:00:00 0 0 0 100 0 07:00:01 3 5 2 90 0 08:00:00 0 0 0 100 0 . . . 14:00:00 0 0 0 100 0 14:20:00 0 1 0 99 0 14:40:00 2 0 0 97 0 15:00:00 0 0 0 99 0 Average 0 1 0 99 0

In typical timesharing use, %sys is usually a little higher than %usr. If %sys is more that double %usr, this may indicate tuning problems or that the system is being asked to do an unusually large amount of work. ps and kernel profiling can help identify the problems. In special applications, either of these may be larger than the other without anything being abnormal. A high %wio generally means a disk slowdown has occurred. A high %idle, with degraded response time, may mean memory constraints are present; time spent waiting for memory is attributed to %idle.

Checking system table status with sar -v

The -v option reports the status of the process, lightweight processes (LWPs), inode, file, and shared memory record table. From this report, you can determine when the system tables need to be updated.

proc-sz: Number of process table entries currently being used/allocated in the kernel.
inod-sz: Number of inode table entries currently being used/allocated in the kernel. (See the sar -t inuse and limit fields for a breakdown by filesystem type.)
file-sz: Number of file table entries currently being used in the kernel. The sz is given as 0 because space is allocated dynamically for the file table.
fail: Number of times a table has overflowed or memory for a resource was unavailable (reported for the three tables listed above). See the sar -t fail field for a breakdown by filesystem type.
lock-sz: Number of shared memory record table entries currently being used/allocated in the kernel. The sz is given as 0 because space is allocated dynamically for the shared memory record table.

The following is an example of sar -v output:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 proc-sz fail lwp fail inod-sz fail file fail lock 01:00:00 70/400 0 131 0 1009/5125 0 453 0 11 02:00:00 70/400 0 131 0 1009/5125 0 453 0 11 03:00:01 70/400 0 131 0 0/5125 0 458 0 11 04:00:00 70/400 0 131 0 1011/5125 0 453 0 11 05:00:00 70/400 0 131 0 998/5125 0 453 0 11 06:00:00 74/400 0 135 0 1002/5125 0 453 0 11 07:00:01 70/400 0 131 0 0/5125 0 460 0 11 08:00:00 70/400 0 131 0 0/5125 0 454 0 11 . . . 14:00:00 74/400 0 135 0 804/5125 0 461 0 11 14:20:00 73/400 0 134 0 840/5125 0 461 0 11 14:40:00 74/400 0 135 0 890/5125 0 461 0 11 15:00:00 74/400 0 135 0 879/5125 0 462 0 11 15:20:01 74/400 0 135 0 0/5125 0 468 0 11 Average 71/400 0 132 0 412/5125 0 457 0 11

This example shows that all tables are large enough to have no overflows. If the values in these tables never exceed those shown here, you can reduce the size of the tables as a way of saving space in main memory. The inod-sz values are the total of s5, ufs, sfs, and vxfs inodes and their maximums. sar -t breaks down the inode usage by filesystem type.

Checking swapping and switching volume with sar -w

The -w option reports swapping and switching activity.

The following are some target values and observations:

swpin/s: Number of transfers into memory (swap-ins) per second.
pswin/s: Number of pages transferred for swap-ins (including initial loading of some programs) per second.
swpot/s: Number of transfers from memory to the disk swap area (swap-outs) per second. If greater than 1, you may need to increase memory or decrease the amount of buffer space.
pswot/s: Number of pages transferred for swap-outs per second.
vpswout/s: Number of virtual pages transferred for swap-outs per second. This may be higher than pswot/s if some physical pages were being used by multiple processes.
pswch/s: Process switches per second. This should be 60 to 90 on a busy four- to six-user system.

The following is an example of sar -w output:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 swpin/s pswin/s swpot/s pswot/s vpswout/s pswch/s 01:00:00 0.00 0.0 0.00 0.0 0.0 12 02:00:00 0.00 0.0 0.00 0.0 0.0 12 03:00:01 0.00 0.0 0.00 0.0 0.0 12 04:00:00 0.00 0.0 0.00 0.0 0.0 12 05:00:00 0.00 0.0 0.00 0.0 0.0 12 06:00:00 0.00 0.0 0.00 0.0 0.0 12 07:00:01 0.00 0.0 0.00 0.1 0.2 41 08:00:00 0.00 0.0 0.00 0.0 0.0 12 . . . 14:00:00 0.00 0.0 0.00 0.0 0.0 11 14:20:00 0.00 0.0 0.00 0.0 0.0 15 14:40:00 0.00 0.0 0.00 0.0 0.0 13 15:00:00 0.00 0.0 0.00 0.0 0.0 12 15:20:01 0.00 0.0 0.00 0.0 0.0 14 Average 0.00 0.0 0.00 0.0 0.0 14

This example shows that because no swapping is occurring, there is no memory shortage for the currently active users.

Checking terminal activity with sar -y

The -y option monitors asynchronous terminal device activities. If there is a lot of terminal I/O, you can use this report to determine if there are any bad lines. If you are not using serial lines the report will show zeros for all activities. The activities recorded are defined as follows:

rawch/s: Input characters (raw queue) per second.
canch/s: Input characters processed by canon (canonical queue) per second.
outch/s: Output characters (output queue) per second.
rcvin/s: Receiver hardware interrupts per second.
xmtin/s: Transmitter hardware interrupts per second.
mdmin/s: Modem interrupts per second.

An example of sar -y output follows:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s 01:00:00 0 0 0 0 0 0 02:00:00 0 0 0 0 0 0 03:00:01 0 0 0 0 0 0 04:00:00 0 0 0 0 0 0 05:00:00 0 0 0 0 0 0 06:00:00 0 0 0 0 0 0 07:00:01 0 0 0 0 0 0 08:00:00 0 0 0 0 0 0 . . . 14:00:00 0 0 0 0 0 0 14:20:00 1 0 18 0 0 0 14:40:00 1 1 75 0 0 0 15:00:00 0 0 27 0 0 0 15:20:01 0 0 43 0 0 0 Average 0 0 7 0 0 0

The number of modem interrupts per second (mdmin/s) should be close to 0, and the receive and transmit interrupts per second (xmtin/s and rcvin/s) should be less than or equal to the number of incoming or outgoing characters, respectively. If this is not the case, check for bad lines.

Checking overall system performance with sar -A

The sar -A option provides a view of overall system performance. Use it to get a more global perspective. If data from more than one time slice is shown, the report includes averages.

Checking raw system performance data with sar -R

The sar -R option provides a report of raw data values. Using this option, sar displays the values of the counters from which metrics are computed, rather than computed values. For example, when used with -u, the number of clock ticks are displayed rather than a percentage.

When using the -R option, % prefixes and /s suffixes are not displayed in column headings. For example, instead of %name or name/s, the heading becomes name. Some tables are displayed differently; the layout of tables reporting the ratio of two counters is changed. Columns that are computed from more than one counter are not displayed. If one, or both, of these counters is not already output in another column, then a new column is added for it. This changes the tables for the options -b, -d, -a, -t. At the end of the report averages are not computed; instead, the totals from which the averages are derived are displayed.

An example of sar -R output follows:

UnixWare holly 5 7.1.0 i386 04/22/99

00:00:00 usr sys wio idle intr 01:00:00 332 684 460 358421 88 02:00:00 330 662 490 358598 85 03:00:01 299 690 343 358652 81 04:00:00 299 707 423 358464 80 05:00:00 287 722 496 358588 89 06:00:00 278 713 392 358504 81 07:00:01 11767 16541 7938 323096 937 08:00:00 261 723 342 358566 75 . . . 14:00:00 91 255 141 119499 34 14:20:00 535 696 420 118264 120 14:40:00 1950 495 489 116910 103 15:00:00 123 370 292 119167 71 15:20:01 523 759 415 118232 108 15:40:00 1119 1209 573 117074 170 16:00:00 395 538 296 118737 60 Average 20994 31295 17230 5.69e6 2881

Next topic: Reporting application turnaround with timex
Previous topic: Collecting system activity data automatically

© 2004 The SCO Group, Inc. All rights reserved.
UnixWare 7 Release 7.1.4 - 22 April 2004