Pipes, Redirection & Composition

Every program has three streams

• stdin — input (keyboard by default)
• stdout — normal output (terminal by default)
• stderr — error messages (terminal by default)

The shell lets you redirect any of these. THIS IS THE WHOLE POINT of Unix.

Redirection

echo "hello" > greet.txt        # write stdout to a file (overwrites)
echo "world" >> greet.txt       # append
sort < names.txt                # read stdin from a file
python script.py 2> errors.log  # redirect stderr (file descriptor 2)
python script.py > out.log 2>&1 # both stdout and stderr to one file

Pipes — the killer feature

A pipe (`|`) takes the stdout of one command and feeds it as stdin to the next. You compose tiny tools into pipelines that do what huge GUI apps do — but scriptable.

cat access.log | grep "404" | wc -l
# Find every line containing "404" and count them

ls -lh | sort -k 5 -h | tail -n 5
# 5 largest files in the current directory

ps aux | grep node | grep -v grep | awk '{print $2}' | xargs kill
# Find every node process and kill it (sledgehammer; use carefully)

Useful tools to pipe through

• grep PATTERN — keep lines matching a regex
• wc -l — count lines
• sort — sort alphabetically (or `-n` numerically, `-r` reverse, `-u` unique)
• uniq -c — collapse repeats and count
• head / tail — first or last N lines
• cut -d',' -f3 — extract column 3 of comma-separated input
• sed 's/old/new/g' — search and replace
• awk '{print $2}' — print column 2 of each line
• jq '.users[].name' — query JSON