Text Processing

Most day-to-day Linux administration comes down to text: log lines, config files, command output. These tools are what turn a wall of text into just the part you need - see Globbing and Regex for the pattern syntax they share, and Pipes and Redirection for how they chain together.

grep: finding lines that match

grep "error" app.log              # lines containing "error"
grep -i "error" app.log           # case-insensitive
grep -v "debug" app.log           # invert: lines NOT containing "debug"
grep -c "error" app.log           # count matching lines, don't print them
grep -n "error" app.log           # show line numbers
grep -r "TODO" src/               # recurse into a directory
grep -A 3 -B 1 "panic" app.log    # 3 lines after, 1 line before each match

-E switches to extended regex (equivalent to egrep); without it, grep uses basic regex, where +, ?, and | need escaping.

sed: stream editor for substitution

sed reads input line by line and applies an editing command to each, most commonly a find-and-replace:

sed 's/foo/bar/' file.txt         # replace first "foo" per line with "bar"
sed 's/foo/bar/g' file.txt        # replace every occurrence per line
sed -i 's/foo/bar/g' file.txt     # edit the file in place
sed -i.bak 's/foo/bar/g' file.txt # in place, keeping a .bak backup first
sed -n '10,20p' file.txt          # print only lines 10-20
sed '/^#/d' file.txt              # delete lines starting with #

-i with no argument overwrites the file with no backup - on systems where that matters, always test a sed command without -i first to see what it would do before committing to it.

awk: field-based processing

awk splits each input line into fields (whitespace-delimited by default) and lets you act on specific ones - it's a small programming language, but most real-world use is one-liners:

awk '{print $1}' access.log            # print just the first field
awk -F: '{print $1}' /etc/passwd       # use : as the field separator
awk '{print $NF}' file.txt             # print the last field ($NF = field count)
awk '$3 > 100 {print $1, $3}' data.txt # print fields 1 and 3 where field 3 > 100
awk '{sum += $2} END {print sum}' data.txt  # sum a column

ps aux --sort=-%mem | awk '{print $4, $11}' | head -5

%MEM COMMAND
8.2  /usr/lib/firefox/firefox
3.1  /usr/bin/gnome-shell

cut, sort, and uniq: smaller, focused tools

Not everything needs awk's full field logic:

cut -d: -f1 /etc/passwd       # field 1, splitting on :
cut -c1-10 file.txt           # characters 1 through 10 of each line

sort file.txt                 # alphabetical
sort -n file.txt              # numeric (alphabetical sort orders "10" before "2")
sort -r file.txt              # reverse
sort -k2 -n data.txt          # sort by the 2nd whitespace-delimited field, numerically

uniq file.txt                 # remove adjacent duplicate lines (input must be sorted first)
sort file.txt | uniq -c       # count occurrences of each unique line
sort file.txt | uniq -c | sort -rn   # ...sorted by count, most frequent first

uniq only collapses adjacent duplicates, which is why it's almost always paired with sort first.

wc: counting

wc -l file.txt      # line count
wc -w file.txt       # word count
wc -c file.txt       # byte count
grep -c "error" app.log   # often clearer than `grep "error" app.log | wc -l`

Putting it together

The real value is chaining these into a pipeline, each tool doing one narrow job:

cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

Reads as: extract the first field (client IP) from every line, then find the 10 most frequent values - a quick way to answer "who's hitting this server the most" without writing a dedicated script.