Running many jobs

You know how to run a command on every file in a directory:

for f in *.fastq
do
    mytool "$f" > "$(basename "$f" .fastq).out"
done

That loop runs them one after another. With four samples, fine. With four hundred, each taking ten minutes, that is nearly three days of the server doing one thing at a time while fifteen of its sixteen processors sit idle.

The saving grace is that these jobs usually do not depend on each other. Sample 7 does not need sample 6 to finish first. Work like that is called embarrassingly parallel, and it is most of what bioinformatics does.

GNU parallel runs the same command on many inputs at once. In its simplest form it looks like a loop turned inside out:

$ parallel -j 4 mytool {} ::: *.fastq

The {} is where each input gets substituted, and ::: separates the command from the list of things to run it on. -j is how many to run at once.

Do not leave -j out. Without it parallel helps itself to one job per processor core, which on a shared machine means taking the entire thing, and there is a whole section on that below.

It can also read its inputs from another command, which is how you feed it something more selective than a wildcard:

$ ls *.fastq | parallel -j 4 mytool {}

There are modifiers for building output names, and {.} is the one you will want most, since it gives the input with its extension removed:

$ parallel -j 4 'mytool {} > {.}.out' ::: *.fastq

Note the quotes. Without them the shell would apply the redirect once, to parallel itself, rather than inside each job.

Two things worth doing before you trust a parallel command. --dry-run prints what it would run without running any of it, which is the same "echo before you act" habit from the loops lesson:

$ parallel --dry-run mytool {} ::: *.fastq

And start with -j 2 on a couple of files rather than -j 32 on all of them. Which brings us to the thing that will make you unpopular.

Do not saturate a shared machine. You are not the only person on the server. Take every core with -j 32 and everyone else's work crawls, including the interactive session of whoever is trying to work out why the machine got slow.

On a server without a job scheduler, which is the situation for most of this course, there is nothing stopping you doing this, so the restraint has to come from you. In practice:

$ nproc

That tells you how many cores the machine has. -j 4 is almost always polite and is a good default until you know the machine better. Never use all of the cores nproc reports.

And remember that memory runs out before cores do: sixteen copies of a tool that each want 8 GB need 128 GB, and when that is not there the machine starts swapping and becomes unusable for everybody. The next section, Jobs and Processes, covers a tool called htop that shows you what the machine is doing before you add to it.

Larger shared systems solve this with a job scheduler, most commonly Slurm. Instead of running work yourself, you describe what it needs and submit it to a queue, and the scheduler decides when and where it runs so that the machine is shared fairly. You do not need it here, but you will meet it the moment you move to a cluster, and it is worth knowing the word.

If parallel is not installed, xargs is on every machine and does a cruder version of the same thing:

$ ls *.fastq | xargs -n 1 -P 4 mytool

-P 4 is the number of jobs at once and -n 1 means one input per invocation.

This is as far as one command over many files will take you, and for a great deal of work that is far enough. When it stops being enough, the next lesson is about what to reach for instead.

One last thing before you set any of this going. Work at this scale takes longer than your connection will stay up, so it needs to survive you closing your laptop. The next section covers screen, which is how you do that; until you have read it, do not start a long run and walk away.