Best Linux Performance Monitoring and Debugging Tools

1. SAR
Using sar utility you can do two things: 1) Monitor system real time performance (CPU, Memory, I/O,
etc) 2) Collect performance data in the background on an on-going basis and do analysis on the historical
data to identify bottlenecks.

Sar is part of the sysstat package. The following are some of the things you can do using sar utility.

Collective CPU usage

Individual CPU statistics

Memory used and available

Swap space used and available

Overall I/O activities of the system

Individual device I/O activities

Context switch statistics

Run queue and load average data

Network statistics

Report sar data from a specific time

and lot more..

The following sar command will display the system CPU statistics 3 times (with 1 second interval).

The following “sar -b” command reports I/O statistics. “1 3” indicates that the sar -b will be executed for
every 1 second for a total of 3 times.

$ sar -b 1 3
Linux 2.6.18-194.el5PAE (dev-db) 03/26/2011 _i686_ (8 CPU)

01:56:28 PM tps rtps wtps bread/s bwrtn/s

01:56:29 PM 346.00 264.00 82.00 2208.00 768.00
01:56:30 PM 100.00 36.00 64.00 304.00 816.00
01:56:31 PM 282.83 32.32 250.51 258.59 2537.37
Average: 242.81 111.04 131.77 925.75 1369.90
Iostat
iostat reports CPU, disk I/O, and NFS statistics. The following are some of iostat command examples.

Iostat without any argument displays information about the CPU usage, and I/O statistics about all the
partitions on the system as shown below.

$ iostat
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011

avg-cpu: %user %nice %system %iowait %steal %idle

5.68 0.00 0.52 2.03 0.00 91.76

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

sda 194.72 1096.66 1598.70 2719068704 3963827344
sda1 178.20 773.45 1329.09 1917686794 3295354888
sda2 16.51 323.19 269.61 801326686 668472456
sdb 371.31 945.97 1073.33 2345452365 2661206408
sdb1 371.31 945.95 1073.33 2345396901 2661206408
sdc 408.03 207.05 972.42 513364213 2411023092
sdc1 408.03 207.03 972.42 513308749 2411023092
By default iostat displays I/O data for all the disks available in the system. To view statistics for a specific
device (For example, /dev/sda), use the option -p as shown below.

$ iostat -p sda
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011

avg-cpu: %user %nice %system %iowait %steal %idle

5.68 0.00 0.52 2.03 0.00 91.76

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

sda 194.69 1096.51 1598.48 2719069928 3963829584
sda2 336.38 27.17 54.00 67365064 133905080
sda1 821.89 0.69 243.53 1720833 603892838
Mpstat
mpstat reports processors statistics. The following are some of mpstat command examples.

Option -A, displays all the information that can be displayed by the mpstat command as shown below.
This is really equivalent to “mpstat -I ALL -u -P ALL” command.

$ mpstat -A
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 _x86_64_
(4 CPU)

10:26:34 PM CPU %usr %nice %sys %iowait %irq %soft %steal

%guest %idle
10:26:34 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 99.99
10:26:34 PM 0 0.01 0.00 0.01 0.01 0.00 0.00 0.00
0.00 99.98
10:26:34 PM 1 0.00 0.00 0.01 0.00 0.00 0.00 0.00
0.00 99.98
10:26:34 PM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
10:26:34 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00

10:26:34 PM CPU intr/s

10:26:34 PM all 36.51
10:26:34 PM 0 0.00
10:26:34 PM 1 0.00
10:26:34 PM 2 0.04
10:26:34 PM 3 0.00

10:26:34 PM CPU 0/s 1/s 8/s 9/s 12/s 14/s 15/s

16/s 19/s 20/s 21/s 33/s NMI/s LOC/s SPU/s PMI/s PND/s
RES/s CAL/s TLB/s TRM/s THR/s MCE/s MCP/s ERR/s MIS/s
10:26:34 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 7.47 0.00 0.00 0.00
0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:26:34 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 4.90 0.00 0.00 0.00
0.00 0.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:26:34 PM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.04 0.00 0.00 0.00 0.00 0.00 3.32 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:26:34 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.

mpstat Option -P ALL, displays all the individual CPUs (or Cores) along with its statistics as shown below.

$ mpstat -P ALL
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db) 07/09/2011 _x86_64_
(4 CPU)

10:28:04 PM CPU %usr %nice %sys %iowait %irq %soft %steal

%guest %idle
10:28:04 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 99.99
10:28:04 PM 0 0.01 0.00 0.01 0.01 0.00 0.00 0.00
0.00 99.98
10:28:04 PM 1 0.00 0.00 0.01 0.00 0.00 0.00 0.00
0.00 99.98
10:28:04 PM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
10:28:04 PM 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 1

Vmstat
vmstat reports virtual memory statistics. The following are some of vmstat command examples.

vmstat by default will display the memory usage (including swap) as shown below.

$ vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu--
----
r b swpd free buff cache si so bi bo in cs us sy id wa
st
0 0 305416 260688 29160 2356920 2 2 4 1 0 0 6 1 92
2 0
To execute vmstat every 2 seconds for 10 times, do the following. After
executing 10 times, it will stop automatically.
$ vmstat 2 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu--
---
r b swpd free buff cache si so bi bo in cs us sy id wa
st
1 0 0 537144 182736 6789320 0 0 0 0 1 1 0 0 100
0 0
0 0 0 537004 182736 6789320 0 0 0 0 50 32 0 0 100
0 0
..
iostat and vmstat are part of the sar utility. You should install sysstat package to get iostat and vmstat
working.

PS Command
Process is a running instance of a program. Linux is a multitasking operating system, which means that
more than one process can be active at once. Use ps command to find out what processes are running
on your system.

ps command also give you lot of additional information about the running process which will help you
identify any performance bottlenecks on your system.

The following are few ps command examples.

Use -u option to display the process that belongs to a specific username. When you have multiple
username, separate them using a comma. The example below displays all the process that are owned by
user oracle.

UID PID PPID C STIME TTY TIME CMD

oracle 571 564 0 09:57 ? 00:00:00 sshd: oracle@pts/0

oracle 580 571 0 09:57 pts/0 00:00:00 -bash

oracle 885 881 0 10:00 ? 00:00:00 /bin/sh /home/oracle/one.sh

oracle 902 885 0 10:00 ? 00:00:00 /bin/sh /home/oracle/one.sh

oracle 904 902 7 10:00 ? 00:00:04 rman target /

oracle 918 904 3 10:00 ? 00:00:01 oracleTEST
(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=b

eq)))

oracle 920 904 0 10:00 ? 00:00:00 oracleTEST

(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=b

eq)))

oracle 929 904 26 10:00 ? 00:00:13 oracleTEST

(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=b

eq)))

oracle 930 904 10 10:00 ? 00:00:05 oracleTEST

(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=b

eq)))

oracle 932 904 10 10:00 ? 00:00:05 oracleTEST

(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=b

eq)))

oracle 946 1 0 10:00 ? 00:00:00 ora_p000_TEST

oracle 948 1 0 10:00 ? 00:00:00 ora_p001_TEST

oracle 950 1 0 10:00 ? 00:00:00 ora_p002_TEST

oracle 952 1 0 10:00 ? 00:00:00 ora_p003_TEST

oracle 1010 23779 0 10:00 ? 00:00:00 FNDCRM

apps/ZG918FB6FD632F59E18960F4B98044CF074594C51C5

328260A85782F2C0E61E9A4CD60CCD6B3C66D5A343B90B5F35ED54D FND FNDCRM N 10 c LOCK Y

DRSITE 4318941

oracle 1013 1 8 10:00 ? 00:00:00 oracleTEST (LOCAL=NO)

oracle 2580 2575 0 Dec07 ? 00:00:02 /usr/libexec/gnome-session-binary --session gnome-class

ic

oracle 2589 1 0 Dec07 ? 00:00:00 dbus-launch --sh-syntax --exit-with-session

oracle 2590 1 0 Dec07 ? 00:00:00 /usr/bin/dbus-daemon --fork --print-pid 5 --print-addre

ss 7 --session

oracle 2633 1 0 Dec07 ? 00:00:00 /usr/libexec/imsettings-daemon

oracle 2637 1 0 Dec07 ? 00:00:00 /usr/libexec/gvfsd

oracle 2725 2580 0 Dec07 ? 00:00:01 /usr/bin/ssh-agent /bin/sh -c exec -l /bin/bash -c "env

GNOME_SHELL_SESSION_MODE=classic gnome-session --session gnome-classic"

oracle 2771 1 0 Dec07 ? 00:00:00 /usr/libexec/at-spi-bus-launcher

oracle 2776 2771 0 Dec07 ? 00:00:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/

at-spi2/accessibility.conf --nofork --print-address 3

oracle 2778 1 0 Dec07 ? 00:00:00 /usr/libexec/at-spi2-registryd --use-gnome-session

oracle 2836 1 0 Dec07 ? 00:00:00 /usr/bin/gnome-keyring-daemon --start --components=secr

ets

oracle 2847 2580 0 Dec07 ? 00:20:04 /usr/bin/gnome-shell

oracle 2894 1 0 Dec07 ? 00:00:00 /usr/bin/pulseaudio --start --log-target=syslog

oracle 2936 2847 0 Dec07 ? 00:00:04 ibus-daemon --xim --panel disable

oracle 2940 2936 0 Dec07 ? 00:00:00 /usr/libexec/ibus-dconf

oracle 2942 1 0 Dec07 ? 00:00:00 /usr/libexec/ibus-x11 --kill-daemon

oracle 2945 1 0 Dec07 ? 00:00:00 /usr/libexec/ibus-portaloracle 2956 1 0 Dec07 ?

00:00:00 /usr/libexec/xdg-permission-store

oracle 2961 1 0 Dec07 ? 00:00:00 /usr/libexec/gnome-shell-calendar-server

oracle 2966 1 0 Dec07 ? 00:00:00 /usr/libexec/evolution-source-registry

oracle 2975 1 0 Dec07 ? 00:00:00 /usr/libexec/goa-daemon

oracle 2984 1 0 Dec07 ? 00:00:00 /usr/libexec/dconf-service

oracle 2995 1 0 Dec07 ? 00:00:01 /usr/libexec/mission-control-5

oracle 2999 1 0 Dec07 ? 00:00:14 /usr/libexec/gvfs-udisks2-volume-monitor

oracle 3008 1 0 Dec07 ? 00:00:00 /usr/libexec/gvfs-gphoto2-volume-monitor

oracle 3013 1 0 Dec07 ? 00:00:00 /usr/libexec/gvfs-goa-volume-monitor

oracle 3022 1 0 Dec07 ? 00:00:14 /usr/libexec/goa-identity-service

oracle 3066 1 0 Dec07 ? 00:00:00 /usr/libexec/gvfs-mtp-volume-monitor

oracle 3071 1 0 Dec07 ? 00:00:00 /usr/libexec/gvfs-afc-volume-monitor

oracle 3083 2580 0 Dec07 ? 00:00:02 /usr/libexec/gsd-power

oracle 3084 2580 0 Dec07 ? 00:00:00 /usr/libexec/gsd-print-notifications

oracle 3085 2580 0 Dec07 ? 00:00:00 /usr/libexec/gsd-rfkill

oracle 3086 2580 0 Dec07 ? 00:00:00 /usr/libexec/gsd-screensaver-proxy

oracle 3088 2580 0 Dec07 ? 00:00:00 /usr/libexec/gsd-sharing

oracle 3090 2580 0 Dec07 ? 00:00:00 /usr/libexec/gsd-smartcard

oracle 3094 2580 0 Dec07 ? 00:00:00 /usr/libexec/gsd-sound

oracle 3097 2580 0 Dec07 ? 00:00:00 /usr/libexec/gsd-wacom

oracle 3100 2580 0 Dec07 ? 00:00:00 /usr/libexec/gsd-xsettings

oracle 3104 2580 0 Dec07 ? 00:00:00 /usr/libexec/gsd-account

oracle 22557 1 0 Dec11 ? 00:00:19 ora_psp0_TEST

oracle 22559 1 0 Dec11 ? 00:00:18 ora_vktm_TEST

oracle 22564 1 0 Dec11 ? 00:00:04 ora_gen0_TEST

oracle 22566 1 0 Dec11 ? 00:00:04 ora_mman_TEST

oracle 22570 1 0 Dec11 ? 00:00:04 ora_diag_TEST

oracle 22572 1 0 Dec11 ? 00:02:24 ora_dbrm_TEST

oracle 22574 1 0 Dec11 ? 00:00:01 ora_vkrm_TEST

oracle 22576 1 0 Dec11 ? 00:03:47 ora_dia0_TEST

oracle 22578 1 0 Dec11 ? 00:00:11 ora_dbw0_TEST

oracle 22580 1 0 Dec11 ? 00:00:11 ora_dbw1_TEST

oracle 22582 1 0 Dec11 ? 00:00:21 ora_lgwr_TEST

oracle 22584 1 0 Dec11 ? 00:00:39 ora_ckpt_TEST

oracle 22586 1 0 Dec11 ? 00:00:11 ora_lg00_TEST

oracle 22588 1 0 Dec11 ? 00:00:07 ora_smon_TEST

oracle 22590 1 0 Dec11 ? 00:00:01 ora_lg01_TEST

oracle 22592 1 0 Dec11 ? 00:00:01 ora_reco_TEST

oracle 22594 1 0 Dec11 ? 00:00:03 ora_lreg_TEST

oracle 22596 1 0 Dec11 ? 00:00:02 ora_pxmn_TEST

oracle 22598 1 0 Dec11 ? 00:00:41 ora_mmon_TEST

oracle 22600 1 0 Dec11 ? 00:02:43 ora_mmnl_TEST

oracle 22642 1 0 Dec11 ? 00:00:01 ora_tmon_TEST

oracle 22644 1 0 Dec11 ? 00:00:01 ora_arc0_TEST

The example below display the process Id and commands in a hierarchy. –forest is an argument to ps
command which displays ASCII art of process tree. From this tree, we can identify which is the parent
process and the child processes it forked in a recursive manner.
ps -e -o pid,args --forest

Free
Free command displays information about the physical (RAM) and swap memory of your system.

In the example below, the total physical memory on this system is 1GB. The values displayed below are
in KB.

# free
total used free shared buffers cached
Mem: 1034624 1006696 27928 0 174136 615892
-/+ buffers/cache: 216668 817956
Swap: 2031608 0 2031608

The following example will display the total memory on your system including RAM and Swap.

In the following command:

option m displays the values in MB

option t displays the “Total” line, which is sum of physical and swap memory values

option o is to hide the buffers/cache line from the above example.

# free -mto
total used free shared buffers cached
Mem: 1010 983 27 0 170
601
Swap: 1983 0 1983
Total: 2994 983 2011

TOP
Top command displays all the running process in the system ordered by certain columns. This displays
the information real-time.

You can kill a process without exiting from top. Once you’ve located a process that needs to be killed,
press “k” which will ask for the process id, and signal to send. If you have the privilege to kill that
particular PID, it will get killed successfully.
PID to kill: 1309
Kill PID 1309 with signal [15]:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1309 geek 23 0 2483m 1.7g 27m S 0 21.8 45:31.32 gagent
1882 geek 25 0 2485m 1.7g 26m S 0 21.7 22:38.97 gagent
5136 root 16 0 38040 14m 9836 S 0 0.2 0:00.39 nautilus

Use top -u to display a specific user processes only in the top command output.

$ top -u geek

While unix top command is running, press u which will ask for username as shown below.

Which user (blank for all): geek

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1309 geek 23 0 2483m 1.7g 27m S 0 21.8 45:31.32 gagent
1882 geek 25 0 2485m 1.7g 26m S 0 21.7 22:38.97 gagent

Pmap
pmap command displays the memory map of a given process. You need to pass the pid as an argument
to the pmap command.

The following example displays the memory map of the current bash shell. In this example, 5732 is the
PID of the bash shell.

$ pmap 5732
5732: -bash
00393000 104K r-x-- /lib/ld-2.5.so
003b1000 1272K r-x-- /lib/libc-2.5.so
00520000 8K r-x-- /lib/libdl-2.5.so
0053f000 12K r-x-- /lib/libtermcap.so.2.0.8
0084d000 76K r-x-- /lib/libnsl-2.5.so
00c57000 32K r-x-- /lib/libnss_nis-2.5.so
00c8d000 36K r-x-- /lib/libnss_files-2.5.so
b7d6c000 2048K r---- /usr/lib/locale/locale-archive
bfd10000 84K rw--- [ stack ]
total 4796K
pmap -x gives some additional information about the memory maps.

$ pmap -x 5732
5732: -bash
Address Kbytes RSS Anon Locked Mode Mapping
00393000 104 - - - r-x-- ld-2.5.so
003b1000 1272 - - - r-x-- libc-2.5.so
00520000 8 - - - r-x-- libdl-2.5.so
0053f000 12 - - - r-x-- libtermcap.so.2.0.8
0084d000 76 - - - r-x-- libnsl-2.5.so
00c57000 32 - - - r-x-- libnss_nis-2.5.so
00c8d000 36 - - - r-x-- libnss_files-2.5.so
b7d6c000 2048 - - - r---- locale-archive
bfd10000 84 - - - rw--- [ stack ]
-------- ------- ------- ------- -------
total kB 4796 - - -

To display the device information of the process maps use ‘pamp -d pid’.

Netstat
Netstat command displays various network related information such as network connections, routing
tables, interface statistics, masquerade connections, multicast memberships etc.,

The following are some netstat command examples.

List all ports (both listening and non listening) using netstat -a as shown below.

# netstat -a | more
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:30037 *:* LISTEN
udp 0 0 *:bootpc *:*

Active UNIX domain sockets (servers and established)

Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 6135 /tmp/.X11-unix/X0
unix 2 [ ACC ] STREAM LISTENING 5140
/var/run/acpid.socket

Use the following netstat command to find out on which port a program is running.

# netstat -ap | grep ssh

(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 1 0 dev-db:ssh 101.174.100.22:39213
CLOSE_WAIT -
tcp 1 0 dev-db:ssh 101.174.100.22:57643
CLOSE_WAIT -

Use the following netstat command to find out which process is using a particular port.

# netstat -an | grep ':80'

Lsof
Lsof stands for ls open files, which will list all the open files in the system. The open files include network
connection, devices and directories. The output of the lsof command will have the following columns:

COMMAND process name.

PID process ID

USER Username

FD file descriptor

TYPE node type of the file

DEVICE device number

SIZE file size

NODE node number

NAME full path of the file name.

To view all open files of the system, execute the lsof command without any parameter as shown below.

# lsof | more
COMMAND PID USER FD TYPE DEVICE SIZE NODE
NAME
init 1 root cwd DIR 8,1 4096 2 /
init 1 root rtd DIR 8,1 4096 2 /
init 1 root txt REG 8,1 32684 983101
/sbin/init
init 1 root mem REG 8,1 106397 166798
/lib/ld-2.3.4.so
init 1 root mem REG 8,1 1454802 166799
/lib/tls/libc-2.3.4.so
init 1 root mem REG 8,1 53736 163964
/lib/libsepol.so.1
init 1 root mem REG 8,1 56328 166811
/lib/libselinux.so.1
init 1 root 10u FIFO 0,13 972
/dev/initctl
migration 2 root cwd DIR 8,1 4096 2 /
skipped..
To view open files by a specific user, use lsof -u option to display all the files opened by a specific user.

# lsof -u ramesh
vi 7190 ramesh txt REG 8,1 474608 475196 /bin/vi
sshd 7163 ramesh 3u IPv6 15088263 TCP dev-db:ssh->abc-
12-12-12-12.
To list users of a particular file, use lsof as shown below. In this example, it displays all users who are
currently using vi.

# lsof /bin/vi
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
vi 7258 root txt REG 8,1 474608 475196 /bin/vi
vi 7300 ramesh txt REG 8,1 474608 475196 /bin/vi

w and uptime
While monitoring system performance, w command will hlep to know who is logged on to the system.

$ w
09:35:06 up 21 days, 23:28, 2 users, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root tty1 :0 24Oct11 21days 1:05 1:05 /usr/bin/Xorg :0
-nr -verbose
ramesh pts/0 192.168.1.10 Mon14 0.00s 15.55s 0.26s sshd: localuser
[priv]
john pts/0 192.168.1.11 Mon07 0.00s 19.05s 0.20s sshd: localuser
[priv]
jason pts/0 192.168.1.12 Mon07 0.00s 21.15s 0.16s sshd: localuser
[priv]

For each and every user who is logged on, it displays the following info:

Username

tty info

Remote host ip-address

Login time of the user

How long the user has been idle

JCPU and PCUP

The command of the current process the user is executing

Line 1 of the w command output is similar to the uptime command output. It displays the following:

Current time

How long the system has been up and running

Total number of users who are currently logged on the system

Load average for the last 1, 5 and 15 minutes

If you want only the uptime information, use the uptime command.

$ uptime
09:35:02 up 106 days, 28 min, 2 users, load average: 0.08, 0.11, 0.05

Please note that both w and uptime command gets the information from the /var/run/utmp data file.

/proc
/proc is a virtual file system. For example, if you do ls -l /proc/stat, you’ll notice that it has a size of 0
bytes, but if you do “cat /proc/stat”, you’ll see some content inside the file.

Do a ls -l /proc, and you’ll see lot of directories with just numbers. These numbers represents the
process ids, the files inside this numbered directory corresponds to the process with that particular PID.

The following are the important files located under each numbered directory (for each process):
cmdline – command line of the command.

environ – environment variables.

fd – Contains the file descriptors which is linked to the appropriate files.

limits – Contains the information about the specific limits to the process.

mounts – mount related information

The following are the important links under each numbered directory (for each process):

cwd – Link to current working directory of the process.

exe – Link to executable of the process.

root – Link to the root directory of the process.

Socket Statistics – SS
ss stands for socket statistics. This displays information that are similar to netstat command.

To display all listening sockets, do ss -l as shown below.

$ ss -l
Recv-Q Send-Q Local Address:Port Peer Address:Port
0 100 :::8009 :::*
0 128 :::sunrpc :::*
0 100 :::webcache :::*
0 128 :::ssh :::*
0 64 :::nrpe :::*

The following displays only the established connection.

$ ss -o state established
Recv-Q Send-Q Local Address:Port Peer Address:Port
0 52 192.168.1.10:ssh 192.168.2.11:55969 timer:(on,414ms,0)
The following displays socket summary statistics. This displays the total number of sockets broken down
by the type.

$ ss -s
Total: 688 (kernel 721)
TCP: 16 (estab 1, closed 0, orphaned 0, synrecv 0, timewait 0/0), ports 11
Transport Total IP IPv6
* 721 - -
RAW 0 0 0
UDP 13 10 3
TCP 16 7 9
INET 29 17 12
FRAG 0 0 0