To totally unlock this section you need to Log-in
Monitoring Disk I/O on Linux systems is one of the most common task to follow when analyzing the current performance status of a system and trying troubleshoot/optimize an existing environment.
The following article will introduce, with some details, several command line tools (on both RHEL/Centos and Debian/Ubuntu environments) that could be used for this purpose.
iostat - Report Disk IO Statistics
isotat is part of sysstat rpm. You can install sysstat using yum or any other tool depending upon your environment.
# yum install sysstat
iostat summarises per-disk I/O statistics, providing metrics for IOPS, throughput, I/O request times, and utilization. It can be executed by any user, and is typically the first command used to monitor disk io performance and investigate disk IO statistics and issues at the command line.
iostat provides many options for customizing the output. A useful combination is -dxz 1, to show disk utilization only (-d), extended columns (-x), skipping devices with zero metrics (-z), and per-second output.
An alternative command to show disk usage stats is iostat -y 5:
Each report, every 5 seconds, include the CPU stats and the disk stats. The CPU stats is a break up of where CPU time was spent during the interval. The disk stats includes the number of I/O requests per second (tps), the rate of read and write (kB_read/s and kB_write/s) and the amount of data read and written (kB_read and kB_wrtn).
The -y argument instructs iostat to discard the first report which are the stats since boot and are rarely useful. The “5” in the command line specifies the interval in seconds. The CPU stats can be omitted by including the -d flag (as in the previous command), although practically it is useful to have it there.
dstat
dstat is a little more user-friendly version of iostat, and can show much more information than just disk bandwidth. Here is dstat in action, showing cpu and disk stats:
The above output is shown when using the following command
dstat -cd --disk-util --disk-tps
The command-line flags include -c for CPU stats, -d for disk stats, --disk-util for disk utilization and --disk-tps for disk transactions (I/O requests) per second.
The dstat utility is usually available for installation, on both Debian and RedHat and relative alternative distros, with the following common commands:
apt-get install dstat (Debian/Ubuntu) yum install dstat (RedHat/Centos/Fedora) dnf install dstat (RedHat/Centos/Fedora)
NOTE: on RedHat/Centos/Fedora systems the dstat utility is included into the pcp-system-tools package (you can always check it by using the dnf provides dstat).
A small set of options available in dstat are the following:
-c, –cpu enable cpu stats -C 0,3,total include cpu0, cpu3 and total -d, –disk enable disk stats -D total,hda include hda and total -g, –page enable page stats -i, –int enable interrupt stats -I 5,eth2 include int5 and interrupt used by eth2 -l, –load enable load stats -m, –mem enable memory stats -n, –net enable network stats -N eth1,total include eth1 and total -p, –proc enable process stats -r, –io enable io stats (I/O requests completed) -s, –swap enable swap stats -S swap1,total include swap1 and total -t, –time enable time/date output -T, –epoch enable time counter (seconds since epoch) -y, –sys enable system stats
ioping
ioping is a quick and dirty storage volume latency checker. It is useful for checking if the elevated disk times that you’re seeing are because of a degradation of the underlying virtual disk / network / hardware.
Low numbers (<1ms) and low variance in the numbers are indicators of a healthy storage volume.
-c count - Stop after count requests. -i interval - Set time between requests to interval (1s). -l speed - Set speed limit in bytes per second. Set interval to request-size / speed. -t time - Minimal valid request time (0us). Too fast requests are ignored in statistics. -T time - Maximum valid request time. Too slow requests are ignored in statistics. -s size - Request size (4k). -S wsize - Working set size (1m for directory, whole size for file or device). -o offset - Starting offset in the file/device (0). -w deadline - Stop after deadline time passed. -p period - Print raw statistics for every period requests (see format below). -P period - Print raw statistics for every period in time. -A - Use asynchronous I/O (syscalls io_submit(2), io_submit(2), etc). -B - Batch mode. Be quiet and print final statistics in raw format. -C - Use cached I/O. Suppress cache invalidation via posix_fadvise(2)) before read and fdatasync(2) after each write. -D - Use direct I/O (see O_DIRECT in open(2)). -L - Use sequential operations rather than random. This also sets default request size to 256k (as in -s 256k). -R - Disk seek rate test. This option suppress human-readable output for each request (as -q), sets default interval to zero (-i 0), stops measurement after 3 seconds (-w 3) and increases default working set size to 64m (-S 64m). Working set (-S) should be increased accordingly if disk has huge cache. -W - Use writes rather than reads. Safe for directory target. Write I/O gives more reliable results for systems where non-cached reads are not supported or cached at some level. Might be *DANGEROUS* for file/device: it will shred your data. In this case should be repeated tree times (-WWW). -G - Alternate read and write requests. -Y - Use sync I/O (see O_SYNC in open(2)). -y - Use data sync I/O (see O_DSYNC in open(2)). -k - Keep and reuse temporary working file "ioping.tmp" (only for directory target). -q - Suppress periodical human-readable output. -h - Display help message and exit. -v - Display version and exit.
For options that expect time argument (-i, -P and -w), default is seconds, unless you specify one of the following suffixes (case-insensitive):
us, usec - microseconds (a millionth of a second, 1 / 1 000 000) ms, msec - milliseconds (a thousandth of a second, 1 / 1 000) s, sec - seconds m, min - minutes h, hour - hours
For options that expect "size" argument (-s, -S and -o), default is bytes, unless you specify one of the following suffixes (case-insensitive):
sector - disk sectors (a sector is always 512). KiB, k, kb - kilobytes (1 024 bytes) page - memory pages (a page is always 4KiB). MiB, m, mb - megabytes (1 048 576 bytes) GiB, g, gb - gigabytes (1 073 741 824 bytes) TiB, t, tb - terabytes (1 099 511 627 776 bytes)
For options that expect "number" argument (-p and -c) you can optionally specify one of the following suffixes (case-insensitive):
k - kilo (thousands, 1 000) m - mega (millions, 1 000 000) g - giga (billions, 1 000 000 000) t - tera (trillions, 1 000 000 000 000)
Exit Codes
Returns 0 upon success. The following error codes are defined:
1 - Invalid usage (error in arguments). 2 - Error during preparation stage. 3 - Error during runtime.
vmstat - Report virtual memory statistics
The vmstat command (short for virtual memory statistics) is a built-in monitoring utility in Linux. The command is used to obtain information about memory, system processes, paging, interrupts, block I/O, disk, and CPU scheduling. Users can observe system activity virtually in real-time by specifying a sampling period.
vmstat is another monitoring tool which is part of procps-ng rpm (the same package is also available on Ubuntu/Debian systems by using yum/dnf utilities). It is most likely possible that procps-ng is installed by default on your Linux node or else you can also install it manually using yum (in this example on RHEL/CentOS, but a similar approach can be used on Ubuntu/Debian systems):
# yum install procps-ng
By default, vmstat produces output once. However, if you want vmstat to automatically update its output, then you can do that by specifying a numeric value as input to the command. This numeric value acts as delay (in seconds) after which the output is updated. For example:
# vmstat 3
vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity. The following example represents a usage of vmstat to monitor disk IO performance in Linux using -d for 1 second with 1 second interval.
# vmstat -d 1 1
To get summary disk IO statistics about disk activity:
# vmstat -D 1 1 6 disks 3 partitions 1492064 total reads 12447 merged reads 20407898 read sectors 5711511 milli reading 374572389 writes 3460667 merged writes 48208708608 written sectors 7759736862 milli writing 0 inprogress IO 47247 milli spent IO
The basic output of the vmstat command displays system information in six sections.
1. procs – Process Statistics
r – Active process count. b – Sleeping process count.
2. memory – Memory statistics
swpd – Total virtual memory. The swap space is initially unoccupied. However, the kernel starts using swap space as the system’s physical memory reaches its limit. free – Total free memory. buff – Total memory temporarily used as a data buffer. cache – Total cache memory.
3. swap – Swap space Statistics
si – The rate of swapping-in memory from disk. so – The rate of swapping-out memory to disk.
4. io – Input/Output Statistics
bi – Blocks received from a block device per second. bo – Blocks sent to a block device per second.
5. system – Scheduling statistics
in – The number of system interrupts. cs – The number of context switches per second.
6. cpu – CPU Statistics
us – The percentage of CPU time spent on non-kernel processes. sy – The percentage of CPU time spent on kernel processes. id – The percentage of idle CPU. wa – The percentage of CPU time spent waiting for Input/Output. st – The percentage of CPU time stolen by a virtual machine.
iotop - Monitor disk IO Speed
iotop or Input/Output top is a command in Linux which is used to display and monitor the disk IO usage details and even gets a table of existing IO utilization by the process. It is designed in python and needs kernel modules for its execution. It is used by system administrators to trace the specific process that may be causing a high disk I/O read/writes. His output is very similar, as layout, to top utility.
iotop specialises is part of iotop rpm. You can install iotop using yum or any other tool depending upon your environment (apt or apt-get on Ubuntu/Debian).
# yum install iotop # apt-get install iotop
iotop watches disk I/O usage information output by the Linux kernel (requires 2.6.20 or later) and displays a table of current I/O usage by processes or threads on the system.
With --onlyswitch iotop will only show processes or threads actually doing I/O, instead of showing all processes or threads so you can check and monitor disk IO performance.
Another example is to show processes that are actually doing IO:
# iotop -o
To display a specific process (specifying a valid PID of a running process/thread):
# iotop -p 10989
Another useful option is -t, to add a timestamp to each line of the output:
sudo iotop -t
nmon - Monitor System Stats
nmon application can be installed from the distribution's standard repository (for Ubuntu/Debian, using apt-get install nmon).
nmon is not available in the default repository of RHEL/CentOS. You can install if from the EPEL repository. To install the entire EPEL repo on RHEL/CentOS 7:
# rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
And to install EPEL repo on CentOS/RHEL 8:
# rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
Next you can install nmon using yum command (in this example on RHEL/CentOS, but a similar approach can be used on Ubuntu/Debian systems):
# yum install nmon
Alternatively you can also install nmon tool manually:
# rpm -Uvh https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/n/nmon-16g-3.el7.x86_64.rpm
Another quicker way to install nnmon on RHEL/CentOS systems is the following:
dnf install epel-release # to install nmon dnf install nmon
nmon can display the CPU, memory, network, disks (mini graphs or numbers), file systems, NFS, top processes, resources (Linux version & processors) and on Power micro-partition information. When using nmon via a terminal session you can see the performance data directly on the screen and updated every second.
Execute nmon from the terminal:
# nmon
To only display disk usage statistics press D.
The tool also includes the ability to increase and decrease the speed of updates. By hitting the - key on your keyboard you will speed up the screen updates and, conversely, the + key will slow them down.
We can add network and memory stats to the mix by hitting the n key followed by the m key (on your keyboard). The window will add those to the terminal console.
If we want to remove any added statistics from the nmon tool view, press the key again (that we used to include the stats); for example, to remove memory stats, we can press the m key again, or the d key to remove disk usage stats.
To quit nmon, hit the q key and you will be returned to your bash prompt.
There is also the possibility to capture data to a file. The following example show how nmon will collect fourty rounds of information every 30 seconds. To do this, you would issue the command:
nmon -f -s 60 -c 40
After issuing the above command we will find a file in the current working directory with the extension .nmon. Open that file to view the collected data.
Finally, we can also create a simple cron job to schedule a regular dump of nmon-collected data (which could be handy for troubleshooting a recurring issue). A simple solution for this would be to create a bash script with a definition similar to the following:
#! /bin/sh nmon -f -s 60 -c 30
Now we need to save the file and give it executable permissions with chmod u+x nmon.sh, for example. Next we can define the cron job in crontab for editing with the command crontab -e and entering something similar:
30 11 * * * ~/nmon.sh
Save and close crontab. The above cron job will run every day at 11:30 AM. Modify that to fit your needs and you have an easy solution for troubleshooting an issue occurring on your Linux data center machines.
ADDITIONAL NOTE: for some automation purposes it is possible also to use nmonchart to convert nnmon capture file to html file (using a simple command like nmonchart <nmon-file> <output-file>.html).
atop - Advanced System & Process Monitor
We can again install atop using yum (assuming we had installed EPEL repo) or alternatively we can manually install atop to check and monitor disk IO performance in Linux. On Ubuntu/Debian systems atop can be easily installed by simply using apt-get install atop.
# rpm -Uvh https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/a/atop-2.4.0-4.el7.x86_64.rpm
The program atop is an interactive monitor to view the load on a Linux system. We can use atop to monitor disk IO by process. It shows the occupation of the most critical hardware resources (from a performance point of view) on system level, i.e. cpu, memory, disk and network.
# atop
Now press Shift + d for disk activity. An alternative way to show directly disk activity is by using atop -d command.
We can see that disk utilization of the each process is show separately. Each process disk metrics have the following available fields (columns):
RRDSK: shows read disk. WRDSK: shows write disk. DSK: shows general disk usage. CMD: shows related command.
A quick review of the available of atop command is shown below (visible by using the atop -h command):
Usage: atop [-flags] [interval [samples]] or Usage: atop -w file [-S] [-a] [interval [samples]] atop -r [file] [-b hh:mm] [-e hh:mm] [-flags] generic flags: -V show version information -a show or log all processes (i.s.o. active processes only) -R calculate proportional set size (PSS) per process -P generate parseable output for specified label(s) -L alternate line length (default 80) in case of non-screen output -f show fixed number of lines with system statistics -F suppress sorting of system resources -G suppress exited processes in output -l show limited number of lines for certain resources -y show individual threads -1 show average-per-second i.s.o. total values -x no colors in case of high occupation -g show general process-info (default) -m show memory-related process-info -d show disk-related process-info -n show network-related process-info -s show scheduling-related process-info -v show various process-info (ppid, user/group, date/time) -c show command line per process -o show own defined process-info -u show cumulated process-info per user -p show cumulated process-info per program (i.e. same name) -j show cumulated process-info per container -C sort processes in order of cpu consumption (default) -M sort processes in order of memory consumption -D sort processes in order of disk activity -N sort processes in order of network activity -E sort processes in order of GPU activity -A sort processes in order of most active resource (auto mode) specific flags for raw logfiles: -w write raw data to file (compressed) -r read raw data from file (compressed) special file: y[y...] for yesterday (repeated) -S finish atop automatically before midnight (i.s.o. #samples) -b begin showing data from specified time -e finish showing data after specified time interval: number of seconds (minimum 0) samples: number of intervals (minimum 1) If the interval-value is zero, a new sample can be forced manually by sending signal USR1 (kill -USR1 pid_atop) or with the keystroke 't' in interactive mode.
collectl - Collects data that describes the current system status
collectl is a very nice feature rich command-line utility that can be used to collect performance data that describes the current system status. Unlike most of the other monitoring tools, collectl does not focus in a limited number of system metrics, instead it can gather information on many different types of system resources such as cpu, disk, memory, network, sockets, tcp, inodes, infiniband, lustre, memory, nfs, processes, quadrics, slabs and buddyinfo.
collectl main project page can be found at http://collectl.sourceforge.net/index.html.
Ubuntu/Debian and the likes have Collectl available in the default repositories, so just apt it.
$ sudo apt-get install collectl
Fedora/CentOS too have it in the repos, so grab it with yum.
$ yum install collectl
collectl will collect data that describes the current system status. We can use collectl to check and monitor disk IO performance in Linux. The following command reports CPU and disk IO statistics. Here c and d represent CPU and Disk.
# collectl -sCD
Another option that provides extended information is the "--verbose" option. It expands the summary to include more information but is not identical to using D.
$ collectl -sd --verbose
If we want to filder/include only sdX disks (sda, sdb, etc.) we can use the following approach (The D in -sD option will show you even more details about the Disk usage):
$ collectl -sD --dskfilt sd