awk

cut pulls out columns. awk pulls out columns, and also does arithmetic on them, filters rows by a condition, and adds things up. It is where you go when cut is not quite enough, which happens quickly with real data.

awk splits every line into fields and names them $1, $2 and so on, with $0 meaning the whole line. The program goes in single quotes, in curly braces:

samples.tsv
sample1   control   12000
sample2   treated   9500
sample3   control   8000

$ awk '{print $1}' samples.tsv
sample1
sample2
sample3

Print several fields, in whatever order you like, which is the thing cut cannot do:

$ awk '{print $3, $1}' samples.tsv
12000 sample1
9500 sample2
8000 sample3

The comma puts a space between them. Note that these dollar signs are awk's field numbers, not shell variables. That is exactly why the program is in single quotes: it stops the shell trying to expand $1 before awk ever sees it.

Whitespace is handled sensibly. Unlike cut, awk treats any run of spaces or tabs as one separator, so it works on data lined up with spaces. For anything else, -F sets the separator:

$ awk -F "," '{print $2}' samples.csv

Filtering. Put a condition before the braces and only matching lines are processed:

$ awk '$3 > 10000 {print $1}' samples.tsv
sample1

Read it as: for lines where field 3 is greater than 10000, print field 1. The single quotes are doing double duty here: as well as protecting $3 from the shell, they stop that > being read as a redirect. Inside the quotes it is awk's greater-than, and no file gets overwritten. Conditions can be numeric comparisons or regular expressions, and with no braces at all awk prints the whole matching line, which makes it a grep that understands columns:

$ awk '$2 == "control"' samples.tsv
sample1   control   12000
sample3   control   8000

Arithmetic and totals. awk keeps variables between lines, so you can accumulate. END runs once, after the last line:

$ awk '{sum += $3} END {print sum}' samples.tsv
29500

Three things happen there that are unlike the shell. awk variables need no declaration, an unset one starts at zero, and += means "add this to what is already there". So sum accumulates down the file, and END prints it once at the bottom. Note also that awk's own variables have no dollar sign: sum is a variable, while $3 is a field.

NR is the current line number and NF is the number of fields on this line. Both are more useful than they look:

$ awk 'NR > 1' data.tsv              skip the header line
$ awk '{print NF}' data.tsv          how many columns does each row have?
$ awk 'NF != 3' data.tsv             show rows that do not have three columns

That last one is a small gift. Malformed rows in a large file are miserable to find by eye, and this pulls them straight out.

awk is a complete programming language and this is a fraction of it. But the handful above covers most of what people actually use it for day to day, and between grep, sed and awk you can do a great deal to a text file without ever opening it.