Monitoring with htop

You have started a job. Is it actually doing anything? Is it using the memory you expected? Is somebody else already using the whole machine? htop answers all three, and it is the tool to reach for before any of the others in this section.

$ htop

If it is not installed, top is always there and shows the same information in a less friendly way. On a machine where you can install things, or inside a conda environment, htop is worth having.

The screen splits in two. Along the top are meters for the machine as a whole, and below is the list of running processes, refreshing every couple of seconds.

A word on cores first, since the whole display depends on it. A modern machine has several independent processors, called cores, and can genuinely run that many things at once. Sixteen cores means sixteen jobs at full speed. A program that uses only one is single-threaded, which most simple tools are; one that spreads itself over several is multithreaded. Find out how many your machine has with:

$ nproc

The meters at the top. One bar per core, then memory, then swap. The core bars tell you how busy the machine is: on a sixteen-core server, one full bar and fifteen empty ones means a single-threaded job, which is normal, and all sixteen full means either your parallel work or somebody else's. The memory bar is the one to watch most closely. When it fills and swap starts filling too, the machine is about to become extremely slow for everybody, because it has started shuffling memory out to disk.

The process list. The columns that matter:

• USER - who owns it. On a shared server most of what you see is not yours.
• RES - resident memory, the actual RAM this process is using right now. This is the memory number to trust.
• VIRT - virtual memory, which is usually an alarmingly large number and rarely means anything. Ignore it.
• CPU% - how much processor it is using. Note that 100% means one core fully busy, not the whole machine, so a well-behaved multithreaded tool can legitimately show 800%.
• MEM% - the same as RES, as a share of the machine's total.
• TIME+ - how much processor time it has consumed since it started, which is not the same as how long ago you launched it.

Making it useful. The list is long and mostly other people's. A few keys fix that:

• u then your username - show only your own processes. This is the first thing to do.
• F6 or > - choose what to sort by. Sorting by RES or by CPU% puts the heavy things at the top.
• F5 or t - tree view, which groups a process with the ones it started. Useful when a script has launched a tool and you want to see which is which.
• F4 or \ - filter the list by a piece of text, such as the name of your tool.
• q - quit. As ever, worth knowing before you go in.

What you are looking for. A few patterns are worth recognising:

If your job is at 100% CPU and its memory is steady, it is working normally. If it is at 0% CPU, it is waiting for something, usually reading or writing a file, and if it stays there the disk is your bottleneck rather than the processor. If RES is climbing steadily and never levels off, the job is consuming more and more memory, and on a shared machine it will eventually be killed, or take the machine down with it. Notice that early rather than at three in the morning.

Killing something. First, the reassurance: you can only kill your own processes. Selecting somebody else's and trying simply fails, which is the permissions system doing its job, so you cannot destroy a colleague's week by mistyping.

Select a process with the arrow keys and press F9, or k if your keyboard's function keys are taken by something else, which they usually are on a Mac. htop offers a list of signals; the default, SIGTERM, is the polite one that asks the program to stop and clean up. Only use SIGKILL if that does nothing, since it gives the program no chance to finish writing its output. The killing processes lesson later in this section covers what those signals actually are.