Other languages and programs

Everything you have written so far has been a shell script: a list of commands in a file, run by bash. The shell is very good at one particular job, which is gluing other programs together — take these files, run that tool on each one, feed the output into the next thing. It is not good at arithmetic, statistics, or anything with much structure to it.

For that you use a real programming language, and in bioinformatics that usually means Python or R. You may also meet Perl, which is older and less fashionable but still underneath a lot of working bioinformatics software.

A rough division of labour: R for statistics and plots, Python for general programming and data wrangling, and the shell for orchestrating the tools that do the heavy lifting. Plenty of people argue about the boundaries. The useful thing to know is that they are all just programs on the server, and everything you have learned about paths, permissions and redirection applies to all of them equally.

Interactive versus a script. This is the distinction worth being clear about, because it confuses people for a long time.

Run a language with no file and you get an interactive session, a prompt of its own where you type one line at a time and see the result immediately:

$ python3
Python 3.12.3
>>> 2 + 2
4
>>> exit()
$ 

R does the same thing if you type R, and RStudio is a comfortable graphical wrapper around exactly that: a place to poke at your data, try something, look at a plot, adjust it. Note that the >>> above is Python's prompt, not the shell's $. Beginners regularly type shell commands like ls at a Python prompt and get an error, or type python code at the shell and get command not found. If you are confused about why something will not run, look at the prompt and ask which program you are actually talking to. Ctrl-D gets you out of most of them and back to the shell.

Interactive work is excellent for exploring and hopeless for remembering. Nobody can reconstruct what you did last Tuesday from an interactive session. So once you know what you want, you put the same lines in a file and run it as a script:

$ python3 analysis.py
$ Rscript analysis.R

Now it is repeatable, it can go in version control, a colleague can run it, and you can run it over a hundred samples from a shell loop without sitting there. That progression — try it interactively, then write it down as a script — is how most analysis actually gets built.

You can give these scripts a shebang exactly like your bash scripts, and then run them by name:

#!/usr/bin/env python3

print("hello")

$ chmod +x analysis.py
$ ./analysis.py

The #!/usr/bin/env python3 form is worth preferring over writing out a full path such as /usr/bin/python3, because it finds whichever python3 the shell would find if you typed it yourself. Once you start installing your own software, in the Your Environment section later, that matters: it means your script uses your python rather than the system's.

Compiled programs. Python, R, Perl and bash are all interpreted: another program reads your file and does what it says, every time you run it. Languages like C, C++ and Rust work differently. They are compiled: a compiler translates the source code once, ahead of time, into a file of machine instructions that the processor runs directly.

That file is a binary, and it is what most of the commands you have been using actually are. ls, grep and sort are compiled C programs. The file command, from the command line section, will tell you:

$ file /bin/ls
/bin/ls: ELF 64-bit LSB pie executable, x86-64, dynamically linked

"ELF executable" means a compiled binary. Compare it with one of your own scripts, where file says something like "Python script" or "Bourne-Again shell script" instead: it has looked at the shebang and told you which language it is.

The practical consequences for you:

• Compiled programs are typically much faster, which is why aligners and variant callers are written in C or C++ rather than Python.
• You cannot read one. Opening a binary with cat or nano gives you rubbish and may leave your terminal in a mess (run reset if that happens). A script is just text, so you can always read it and see what it does.
• A binary is built for a particular kind of machine. A binary compiled for Linux on x86 will not run on your Mac, whereas the same Python script runs anywhere that has Python.
• Some software is distributed as source code rather than as a ready-made binary, and has to be compiled before you can run it. That is a job, and it is exactly the pain that package managers spare you by shipping things already built.

None of this changes how you run them. Whether a command is a shell script, a Python script or a compiled binary, you type its name and it needs the execute bit set. That is the whole point of the design: from the outside, they all look the same. The Your Environment section later explains how the shell finds them.