uniq (Unique)

The uniq (unique) command is another useful tool for parsing text. Say you pulled the condition column out of your sample table and ended up with a file full of repeats:

conditions.txt
control
control
treated
treated
treated
untreated

To collapse the repeats:

$ uniq conditions.txt
control
treated
untreated

Counting how often each one occurs is usually more useful, and that is -c:

$ uniq -c conditions.txt
      2 control
      3 treated
      1 untreated

-u keeps only the lines that appear exactly once:

$ uniq -u conditions.txt
untreated

and -d keeps only the ones that are repeated:

$ uniq -d conditions.txt
control
treated

Now the catch, and it is the thing to remember about uniq: it only compares each line with the one directly above it. Duplicates that are not next to each other are invisible to it.

conditions.txt
control
treated
control
treated
untreated

$ uniq conditions.txt
control
treated
control
treated
untreated

Nothing was removed, because no two identical lines were adjacent. This catches people out constantly, and the symptom is a count that is quietly too high.

The fix is to sort first, so that identical lines end up together:

$ sort conditions.txt | uniq
control
treated
untreated

sort | uniq -c is one of the most useful pairs in the whole shell. It answers "how many of each?" for anything you can get onto separate lines:

$ sort conditions.txt | uniq -c
      2 control
      2 treated
      1 untreated

Add a numeric reverse sort on the end and you have a frequency table, most common first:

$ sort conditions.txt | uniq -c | sort -nr

That pipeline is worth memorising. You will use it on log files, on column output from cut, and on anything else where the question is "what is in here, and how much of it".