Disk space

Sequencing data is large, intermediate files are larger, and shared servers run out of space with great regularity. Running out is not a tidy failure either: jobs die partway through, files end up truncated, and on a full home directory even logging in can start misbehaving. Knowing two commands prevents most of it.

df shows how full each filesystem is. The -h flag makes the numbers human readable, which you always want:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   47G  1.2G  98% /
/dev/sdb1       9.1T  4.2T  4.5T  49% /data

Each line is a separate pool of storage. The one to look at is the row whose "Mounted on" path your files are actually under. In the example above, the home directories are on a nearly full 50 GB disk while /data has terabytes free, and that is a very common arrangement: a small system disk and a big data volume. Writing a large output into your home directory when /data is right there is the usual way people run into trouble.

To ask about a particular place rather than reading the whole table:

$ df -h .

The dot means the directory you are standing in, so this answers "how much room is there where I am?".

du shows how much space things are taking up. On its own it walks every subdirectory and produces an unreadable wall of output, so it is almost always used with -s for summary and -h for human readable:

$ du -sh
14G     .

That is the total for the current directory. The genuinely useful form asks about everything in the current directory, one line each:

$ du -sh *
2.1G    alignments
8.6G    raw_reads
1.2G    results
14M     scripts

Now you can see where it went. When a directory has a lot in it, sort the answer, using the -h flag from the sort lesson, which understands G and M:

$ du -sh * | sort -h

The biggest offender ends up at the bottom, next to your prompt, which is exactly where you want it. This one line is the answer to "why is my quota full" nine times out of ten.

One catch: * does not match names beginning with a dot, so hidden directories are invisible to that command, and some of the largest things in your home directory are hidden. ~/.conda and ~/.cache are the usual culprits. If the numbers do not add up to what du -sh reports for the whole directory, that is why. Include them with:

$ du -sh -- .[!.]* * 2>/dev/null | sort -h

The 2>/dev/null is there because if a directory happens to have no hidden entries, the shell leaves the pattern unexpanded and du complains about a file literally called .[!.]* . Sending that grumble to /dev/null keeps the output clean.

A warning worth having: du on a large directory tree takes a while, because it really does look at everything. On a network filesystem it can take minutes. That is normal, and Ctrl-C stops it.

Quotas. On many shared servers you are not limited by the disk being full but by a quota, a per-user limit set by the administrators. If you get "Disk quota exceeded" while df cheerfully reports terabytes free, that is what has happened. Check yours with:

$ quota -s

If that command is not available or reports nothing, your site may use a different tool, and it is worth asking early rather than discovering the limit mid-analysis.

Habits that keep you out of trouble:

• Find out where your big data is supposed to live — usually a project, scratch or data area, not your home directory — and work there from the start. Moving a terabyte afterwards is slow and dull.
• Delete intermediate files once you no longer need them. Alignment pipelines in particular leave enormous ones behind.
• Keep data compressed. Most bioinformatics tools read .gz files directly, so decompressing is usually unnecessary.
• Check df -h before launching something large, not after it fails at 90% complete.
• Remember that conda environments are not small, as the previous lesson mentioned. A few of them will quietly eat a home directory quota.