Pipes and Redirection

Every process starts with three open file descriptors: stdin (0, input), stdout (1, normal output), and stderr (2, error output). Redirection and pipes both work by rewiring these descriptors - redirection sends a descriptor to a file, a pipe sends it to another process.

Redirecting to and from files

command > output.txt     # stdout to a file, overwriting it
command >> output.txt    # stdout to a file, appending
command < input.txt      # stdin from a file
command 2> errors.txt    # stderr only
command > all.txt 2>&1   # stdout AND stderr, both to the same file
command &> all.txt       # Bash shorthand for the line above

Order matters with 2>&1: it means "point fd 2 at wherever fd 1 currently points," so it has to come after > redirects stdout, or it'll still point at the terminal.

command > all.txt 2>&1   # correct: stderr follows stdout into all.txt
command 2>&1 > all.txt   # wrong: stderr still goes to the terminal

Discard output entirely by redirecting to /dev/null, a special device file that silently accepts and discards anything written to it:

noisy-command > /dev/null 2>&1

Pipes

A pipe (|) connects one command's stdout directly to the next command's stdin, without ever touching disk:

ps aux | grep nginx | grep -v grep

www-data  1284  0.0  0.3  55120  9876 ?  S  09:14  0:00 nginx: worker process
root      1281  0.0  0.2  55120  6544 ?  Ss 09:14  0:00 nginx: master process

Chains like this are the standard Unix pattern: each tool (ps, grep, and text-processing tools like sed and awk) does one thing, and pipes compose them into something none of them do alone. See Job Control for what happens to the processes in a pipeline once it's running.

tee: split a stream in two

tee writes its input to a file and passes it through to stdout unchanged - useful when you want to both see output live and save it:

long-build.sh | tee build.log

xargs: turning output into arguments

Pipes pass data as a stream, but many commands expect arguments, not stdin. xargs bridges the two by reading stdin and building a command line from it:

find . -name "*.tmp" | xargs rm

Be careful with filenames containing spaces or newlines - use -print0 and xargs -0 for filenames that might not be simple:

find . -name "*.tmp" -print0 | xargs -0 rm

Heredocs and here-strings

A heredoc feeds multi-line input to a command's stdin, useful in scripts:

cat <<EOF > config.txt
host=localhost
port=8080
EOF

A here-string does the same for a single line, more concisely:

grep "port" <<< "$config_variable"

See Shell Scripting for using these inside scripts, and Environment Variables for how values like $config_variable get set in the first place.