sed (Stream Editor)

grep finds lines. sed changes them. It reads its input a line at a time, applies an instruction to each one, and prints the result, which makes it the tool for "replace this with that in a whole file" without opening an editor.

The instruction you will use ninety per cent of the time is s, for substitute:

$ sed 's/chr1/chromosome1/' variants.txt

Read it as: substitute, then the thing to look for, then what to replace it with, separated by slashes. Every line that contains chr1 comes out with chromosome1 instead.

Two things to know immediately.

It does not change the file. sed prints the modified text to standard output and leaves your file exactly as it was. That is a feature, not a limitation: you get to look at the result before committing to it. To keep it, redirect to a new file, which you already know how to do:

$ sed 's/chr1/chromosome1/' variants.txt > renamed.txt

Never redirect onto the file you are reading. sed 's/a/b/' file.txt > file.txt empties the file before sed gets to read it, and your data is gone.

It replaces the first match on each line only. Add a g on the end, for global, to get all of them:

$ echo "one two one two" | sed 's/one/1/'
1 two one two

$ echo "one two one two" | sed 's/one/1/g'
1 two 1 two

The slash is not sacred. Whatever character comes after the s is the separator, so when your text contains slashes, pick something else and avoid a thicket of backslashes:

$ sed 's|/home/pete|/data/pete|' paths.txt

It takes regular expressions, the ones from the regex lesson, which is where it gets its power:

$ sed 's/^chr//' variants.txt          strip a leading chr from every line
$ sed 's/[0-9]//g' messy.txt           remove every digit

A few other instructions worth knowing:

$ sed -n '5p' file.txt        print only line 5 (-n means print nothing else)
$ sed -n '10,20p' file.txt    print lines 10 to 20
$ sed '1d' file.txt           delete the first line, useful for dropping a header
$ sed '/^#/d' file.txt        delete every line starting with #, dropping comments

That last pair are genuinely useful on data files, where a header or comment lines get in the way of everything downstream.

Finally, -i edits the file in place, changing it for real:

$ sed -i 's/chr1/chromosome1/g' variants.txt

Treat that with respect. There is no undo, and if your pattern was wrong you have now quietly damaged your data. Run it without -i first, look at the output, and only then add the flag. If the file matters, -i.bak keeps the original as variants.txt.bak, which costs nothing. Note there is no space after -i, unlike most flags you have met:

$ sed -i.bak 's/chr1/chromosome1/g' variants.txt