Performance triage and troubleshooting

A repeatable method for a slow server: load, memory, CPU, disk and I/O, then syscall-level evidence.

A method before a tool

• For each resource — CPU, memory, disk, network — ask three questions: how utilised is it, how saturated (is work queuing), and is it producing errors?
• Work from the outside in: whole machine first, then the process, then the threads, then the syscalls.
• Measure once, change one thing, measure again. Tuning two variables at once teaches you nothing.
• Write down what changed before the slowness started: a deployment, a config edit, a traffic shift, a cron job.
• Prefer short samples of the real workload over a single instantaneous reading.

uptime                 # load average over 1, 5 and 15 minutes
free -h                # memory, buffers/cache and swap
df -h /                # space; df -i for inodes
dmesg -T | tail -30    # kernel messages: OOM kills, I/O errors, link flaps
systemctl --failed

The main tools and what each answers

sudo apt install sysstat          # provides iostat, pidstat and sar
vmstat 1 5
iostat -xz 1 3
sudo iotop -oPa
pidstat -u -r -d 1 5

# read the columns that matter
# vmstat: r = runnable, b = blocked, si/so = swap in/out,
#         us/sy/id = cpu split, wa = waiting on I/O
# iostat: %util near 100 and high await = the device is the bottleneck

• Non-zero si/so in vmstat means the box is swapping and memory is the real problem.
• High wa with idle CPU points at storage or a network filesystem, not at your code.
• Sustained %util near 100 with a growing queue is a saturated disk; a short spike is normal.

Going deeper

# what is this process doing, right now?
sudo strace -f -tt -T -p 1234 2>&1 | tail -40
sudo strace -c -p 1234                # syscall summary with time and errors
ltrace -p 1234                        # library calls, when you need them

sudo lsof -p 1234 | wc -l             # open files and sockets
sudo ls -l /proc/1234/fd | wc -l
cat /proc/1234/status | grep -E 'Threads|VmRSS|FDSize'
cat /proc/1234/limits | grep 'open files'

ulimit -n                             # soft limit for this shell
ulimit -Hn                            # hard limit
# 'Too many open files' means the limit, not the disk

# raise the limit for a service, not for the whole machine
# /etc/systemd/system/app.service.d/limits.conf
[Service]
LimitNOFILE=65535

sudo systemctl daemon-reload && sudo systemctl restart app

# a container needs it too - it inherits the daemon's limit
docker run --ulimit nofile=65535:65535 app:1.0

• strace slows the target dramatically. Use -f to follow threads, -T for time in each call, and stop as soon as you have your answer.
• An EAGAIN, EMFILE or ECONNRESET in a syscall trace is often the entire diagnosis.
• When the app is CPU-bound with no obvious hot function, sample it with perf top or a profiler instead of staring at top.

FAQ

Related

Disks, filesystems and mounts Networking: addresses, DNS and firewalls

Last refreshed 2026-09-18.