Lesson 05 beginner

Text Processing and Pipes

How Linux commands pass text through stdin, stdout and stderr, how to redirect and pipe it, and how grep, sort, uniq, cut, tr, sed, awk, xargs and jq turn a raw nginx log into answers in one line.

Most of what a server tells you arrives as text: access logs, error logs, config files, command output, JSON from an API. When the Orbit API starts returning errors at 9 a.m., nobody opens a spreadsheet. You type a short chain of commands, each doing one small job, and thirty seconds later you know which IP is hammering the server, which endpoint is slow and how many 5xx errors happened in the last few minutes.

In this lesson you learn how text flows between commands, how to send it to files or throw it away, and the dozen tools that filter, count and reshape it. You finish by building real one-liners on an nginx access log. A container or WSL is enough for everything here; you do not need the VM.

Three streams: stdin, stdout and stderr

Every process starts with three open channels, each with a number called a file descriptor:

NumberNameDefaultUsed for
0stdinYour keyboardInput the program reads
1stdoutYour terminalNormal output
2stderrYour terminalErrors and warnings

Both stdout and stderr land on your screen by default, so they look identical. They are not, and the difference shows the moment you redirect:

$ ls /etc/hostname /etc/nope > found.txt
ls: cannot access '/etc/nope': No such file or directory
$ cat found.txt
/etc/hostname

The good result went into the file; the error still reached your screen because it travels on stream 2. That separation is deliberate: a script can save its data and still show you what went wrong.

Redirection

Redirection tells the shell where a stream should go before the command starts. The command itself never knows.

SyntaxEffect
cmd > filestdout to file, truncating it first
cmd >> filestdout appended to file
cmd 2> filestderr to file
cmd > file 2>&1stdout to file, then stderr to wherever stdout points
cmd &> fileBash shorthand for both streams to one file
cmd < filefile becomes stdin
cmd 2> /dev/nulldiscard errors (/dev/null swallows everything written to it)

2>&1 means “make stream 2 point where stream 1 points right now“, and the shell reads redirections from left to right. So these two lines differ:

$ ./deploy.sh > deploy.log 2>&1
$ ./deploy.sh 2>&1 > deploy.log

The first sends both streams to deploy.log. In the second, stderr was copied to the terminal before stdout moved to the file. When in doubt in Bash, write &> and the question disappears. In portable sh scripts, use the first form.

Warning: > truncates the file before the command even runs. sort notes.txt > notes.txt leaves you with an empty file. Write to a new file and rename it, or use sort -o notes.txt notes.txt, which is safe.

Pipes and tee

The pipe | connects the stdout of one command to the stdin of the next, as in ls /etc | wc -l. No temporary files, and all commands run at the same time, so a pipe can process a 10 GB log with almost no memory. Pipes carry only stdout; to send errors down the pipe too, write cmd 2>&1 | less (Bash also accepts cmd |& less).

tee copies its input to a file and to stdout, like a T-junction: ./orbit-healthcheck.sh | tee healthcheck.txt shows the result and keeps a copy. It also solves a classic trap. sudo echo "LOG_LEVEL=debug" >> /etc/orbit/orbit.env fails with “Permission denied”, because the redirection is done by your unprivileged shell, not by sudo. Let a root process write the file instead:

$ echo "LOG_LEVEL=debug" | sudo tee -a /etc/orbit/orbit.env > /dev/null

-a appends instead of overwriting, and > /dev/null hides the copy tee would print.

Set up a practice log

If you skipped ahead, or are in a fresh container, create the files used in the rest of this lesson. Orbit’s nginx uses the standard combined log format plus one extra field at the end: the request time in seconds ($request_time).

mkdir -p ~/lab && cd ~/lab
cat > access.log <<'EOF'
203.0.113.10 - - [03/Nov/2026:09:14:02 +0000] "GET /api/notes HTTP/1.1" 200 1532 "-" "Mozilla/5.0 (X11; Linux x86_64)" 0.042
203.0.113.10 - - [03/Nov/2026:09:14:03 +0000] "POST /api/notes HTTP/1.1" 201 211 "-" "Mozilla/5.0 (X11; Linux x86_64)" 0.118
198.51.100.7 - - [03/Nov/2026:09:14:05 +0000] "GET /api/notes/42 HTTP/1.1" 404 64 "-" "curl/8.14.1" 0.006
192.0.2.55 - - [03/Nov/2026:09:14:09 +0000] "GET /wp-login.php HTTP/1.1" 404 162 "-" "Go-http-client/1.1" 0.001
203.0.113.10 - - [03/Nov/2026:09:14:11 +0000] "GET /api/search?q=linux HTTP/1.1" 200 4810 "-" "Mozilla/5.0 (X11; Linux x86_64)" 1.874
198.51.100.7 - - [03/Nov/2026:09:14:12 +0000] "GET /api/notes HTTP/1.1" 502 157 "-" "curl/8.14.1" 3.001
192.0.2.55 - - [03/Nov/2026:09:14:15 +0000] "GET /.env HTTP/1.1" 404 162 "-" "Go-http-client/1.1" 0.001
203.0.113.10 - - [03/Nov/2026:09:14:20 +0000] "GET /api/notes HTTP/1.1" 200 1532 "-" "Mozilla/5.0 (X11; Linux x86_64)" 0.039
203.0.113.24 - - [03/Nov/2026:09:14:21 +0000] "GET / HTTP/1.1" 200 3120 "-" "Mozilla/5.0" 0.004
198.51.100.7 - - [03/Nov/2026:09:14:25 +0000] "GET /api/notes HTTP/1.1" 504 167 "-" "curl/8.14.1" 60.002
203.0.113.24 - - [03/Nov/2026:09:14:30 +0000] "DELETE /api/notes/7 HTTP/1.1" 204 0 "-" "Mozilla/5.0" 0.051
192.0.2.55 - - [03/Nov/2026:09:14:31 +0000] "GET /admin HTTP/1.1" 404 162 "-" "Go-http-client/1.1" 0.001
EOF

Split on spaces, the useful fields are: $1 client IP, $4 time, $6 method (with a leading quote), $7 path, $9 status, $10 bytes and the last field, request time. The user agent contains spaces, so its field count varies; that is why you will reach the request time as “the last field” rather than by number.

Reading files: cat, less, head and tail

  • cat file prints a whole file. Fine for short files, useless for a 2 GB log.
  • less file pages through it: /text searches, n jumps to the next match, G goes to the end, q quits.
  • head -n 5 file shows the first five lines, tail -n 20 file the last twenty, and tail -n +2 file everything from line 2 (skipping a CSV header).

For live logs, tail -f keeps the file open and prints new lines as they arrive. There is a catch: logrotate renames access.log at night and nginx starts a new file. tail -f keeps watching the old, renamed file and goes silent. tail -F follows the name and reopens the new file, so sudo tail -F /var/log/nginx/access.log is the one to use on servers. For services that log to the journal instead of files, journalctl -u orbit-api -f is the equivalent; lesson 18 covers it.

Searching with grep

grep prints the lines that match a pattern. The flags you will use every week:

FlagMeaningExample
-iIgnore casegrep -i error app.log
-vInvert: lines that do not matchgrep -v '^#' nginx.conf
-nShow line numbersgrep -n listen nginx.conf
-cCount matching linesgrep -c '" 404 ' access.log
-rSearch a directory recursivelygrep -rn ORBIT_ /etc/orbit
-EExtended regex (+, ?, alternation)grep -E '50[234]' access.log
-oPrint only the matched partgrep -oE '^[0-9.]+' access.log
-FFixed string, no regexgrep -F '[error]' error.log
-A n, -B n, -C nn lines after, before, or around each matchgrep -C 3 Traceback api.log

The -c example searches for '" 404 ' (quote, space, 404, space) rather than just 404, so a response that happens to be 404 bytes long is not counted; on the practice log it prints 4. Alternation in action:

$ grep -E 'wp-login|\.env|/admin' access.log | cut -d' ' -f1,7
192.0.2.55 /wp-login.php
192.0.2.55 /.env
192.0.2.55 /admin

This finds a bot probing for well-known files; | inside the quotes is the regex “or”, not a pipe, and \. means a literal dot.

Tip: egrep and fgrep are obsolete and recent GNU grep prints a warning when you use them. Write grep -E and grep -F. For searching large code trees, rg (ripgrep) is a fast modern alternative, but grep is on every server.

Cutting, sorting and counting

cut

cut extracts columns: -d sets the delimiter (one character) and -f picks fields, so cut -d: -f1,7 /etc/passwd prints every account with its login shell. It treats every single delimiter as a boundary, so two spaces in a row create an empty field. For whitespace-aligned output such as ps or df, use awk or squeeze the spaces first with tr -s ' '.

sort, uniq and wc

sort orders lines alphabetically by default. -n sorts numerically (10 after 9), -r reverses, -k 2 sorts by the second whitespace-separated field (-k 2,2n: only that field, numerically), -h understands human sizes like 512K and 1.5G (try du -sh /var/log/* | sort -h) and -u drops duplicates.

uniq collapses adjacent identical lines, and uniq -c prefixes each with a count. Adjacent is the key word: always sort before uniq. Finally, wc -l counts lines (-w words, -c bytes), the end of countless “how many?” pipelines.

tr

tr translates, squeezes or deletes single characters from stdin. tr 'a-z' 'A-Z' upper-cases text, tr -s ' ' squeezes runs of spaces into one, and tr -d '\r' < win.txt > unix.txt strips the carriage returns Windows editors add, a frequent cause of “bad interpreter” errors in scripts.

Editing streams with sed

sed (stream editor) changes text as it flows past. Its most common command is substitution: sed 's/80/8080/' replaces the first match on each line, and s/80/8080/g every match. It can also select lines by number or pattern:

CommandWhat it does
sed -n '5,10p' filePrint only lines 5 to 10 (-n turns off automatic printing)
sed -n '$p' filePrint the last line
sed '/^#/d' fileDelete comment lines
sed -n '/09:14:10/,/09:14:25/p' access.logPrint from the first line matching one pattern to the next matching the other

To change a file in place, use -i. On a config file, always add a suffix so sed keeps a backup:

$ sudo sed -i.bak 's/^LOG_LEVEL=info$/LOG_LEVEL=debug/' /etc/orbit/orbit.env
$ diff /etc/orbit/orbit.env.bak /etc/orbit/orbit.env
< LOG_LEVEL=info
> LOG_LEVEL=debug

Preview first by running the command without -i, and check afterwards with diff (output shortened here). Anchors (^ start, $ end of line) keep the change from hitting lines you did not mean.

Warning: sed -i behaves differently on macOS (BSD sed), which requires an argument: sed -i '' 's/a/b/' file. Scripts that run on both should use sed -i.bak and delete the backup afterwards.

Processing columns with awk

awk reads input line by line, splits each line into fields ($1, $2 and so on; $NF is the last field and NF the number of fields), and runs pattern { action } rules on it. With no pattern, the action runs on every line; with no action, matching lines are printed.

$ awk '$9 >= 500' access.log | cut -d' ' -f1,7,9
198.51.100.7 /api/notes 502
198.51.100.7 /api/notes 504
$ awk '{bytes += $10} END {print bytes}' access.log
12079
$ awk '{t += $NF; n++} END {printf "%.3f s average over %d requests\n", t/n, n}' access.log
5.428 s average over 12 requests

The first prints matching lines (a pattern with no action); the last two use a variable and the special END block, which runs after the final line: that is how awk sums and averages. Note how one 60-second timeout drags the average to 5.4 seconds while most requests took milliseconds. Averages hide outliers; the slowest-requests list below shows them.

Use -F to change the separator, for example awk -F: '$3 >= 1000 {print $1}' /etc/passwd lists human accounts, a preview of lesson 7.

Tip: Ubuntu installs mawk as awk, Rocky Linux uses GNU gawk, and Alpine uses BusyBox awk. Everything in this lesson works in all three; gawk-only extras such as strftime do not.

Turning output into arguments with xargs

Some commands do not read stdin; they want file names as arguments (rm, chmod, sed -i). xargs reads items from stdin and appends them to a command:

$ grep -rlZ 'orbit-db-01' /etc/orbit | xargs -0 -r echo sudo sed -i.bak 's/orbit-db-01/orbit-db-02/'
sudo sed -i.bak s/orbit-db-01/orbit-db-02/ /etc/orbit/orbit.env

grep -l lists matching file names and -Z ends each with a null byte; xargs -0 splits on those bytes, so names with spaces survive. -r runs nothing if the input is empty. The echo in front is a dry run that prints the command instead of running it; remove it when the output looks right. -n 1 passes one argument per command and -P 4 runs four in parallel.

JSON logs with jq

Modern apps, including Orbit’s API, often log one JSON object per line. Install the processor with sudo apt install jq, then create a small sample:

cat > ~/lab/api.log <<'EOF'
{"ts":"2026-11-03T09:14:12Z","level":"error","path":"/api/notes","status":502,"ms":3001,"msg":"redis timeout"}
{"ts":"2026-11-03T09:14:20Z","level":"info","path":"/api/notes","status":200,"ms":39,"msg":"ok"}
{"ts":"2026-11-03T09:14:25Z","level":"error","path":"/api/notes","status":504,"ms":60002,"msg":"upstream timeout"}
EOF
$ jq -r 'select(.level == "error") | "\(.ts) \(.status) \(.msg)"' ~/lab/api.log
2026-11-03T09:14:12Z 502 redis timeout
2026-11-03T09:14:25Z 504 upstream timeout

select() filters objects and -r prints raw strings without quotes; jq -c 'select(.ms > 1000) | {path, ms}' would print compact one-line JSON objects instead. Never parse JSON with grep and sed: field order and spacing can change at any release, and jq does not care.

One-liners for the Orbit access log

Put the pieces together. On the server, replace access.log with /var/log/nginx/access.log and prefix sudo.

# Top client IPs
$ awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -n 3
      4 203.0.113.10
      3 198.51.100.7
      3 192.0.2.55
# Requests per status code
$ awk '{print $9}' access.log | sort | uniq -c | sort -rn
      4 404
      4 200
      1 504
      1 502
      1 204
      1 201
# Three slowest requests: seconds, path, status
$ awk '{print $NF, $7, $9}' access.log | sort -rn | head -n 3
60.002 /api/notes 504
3.001 /api/notes 502
1.874 /api/search?q=linux 200
# How many 5xx in the last 1000 requests
$ tail -n 1000 access.log | awk '$9 >= 500 {n++} END {print n+0}'
2

Read each from left to right: pick a column, sort so duplicates sit together, count, sort by the count, keep the top. That pattern answers almost any “which, how many, how slow” question. n+0 prints 0 instead of an empty line when nothing matched.

Practice lab

It is Monday morning and the Orbit on-call channel reports “the API feels slow and some calls fail”. Use ~/lab/access.log (or a real nginx log on your lab VM) to write a short incident note:

  1. Count how many requests failed with a 5xx status and list their paths.
  2. Find the three slowest requests and the client that made each one.
  3. Find which IP sent the most 404s and what paths it asked for.
  4. Save the answers to ~/lab/incident.txt while also seeing them on screen, then print only the msg of error entries in ~/lab/api.log.
Show a solution

awk '$9 >= 500 {print $9, $7}' access.log lists the two failures (502 and 504 on /api/notes). awk '{print $NF, $1, $7}' access.log | sort -rn | head -n 3 shows the slow requests and their clients; the 60-second one is a 504 timeout. awk '$9 == 404 {print $1}' access.log | sort | uniq -c | sort -rn | head -n 1 names 192.0.2.55, and awk '$1 == "192.0.2.55" {print $7}' access.log shows it probing /wp-login.php, /.env and /admin: a scanner, not a user. Wrap any of these as { cmd1; cmd2; } | tee ~/lab/incident.txt to see and save the output together. Finally, jq -r 'select(.level == "error") | .msg' ~/lab/api.log prints the two error messages.

Common mistakes

  • Using uniq without sort. It only merges neighbours, so counts come out split and wrong.
  • Redirecting into the file you are reading. cmd file > file empties the file before cmd reads it.
  • Combining sudo with a redirect. The redirection runs as you, not as root. Use sudo tee or sudo tee -a.
  • Following rotated logs with tail -f. It silently follows the old file. Use tail -F.

Exercise

You add a line to Orbit’s config with sudo echo “FEATURE_SEARCH=on” >> /etc/orbit/orbit.env and get “Permission denied”, even though sudo worked a second ago. What is going on?

  1. The file is locked by the running API
  2. sudo cannot run echo because it is a shell builtin
  3. The >> redirection is performed by your own unprivileged shell before sudo runs; use echo … | sudo tee -a /etc/orbit/orbit.env
  4. You need sudo -i because >> requires a login shell
Show answer

Answer: C. Redirections are set up by the shell that parses the line, and that shell runs as you. sudo only elevates the echo. Piping into sudo tee -a makes a root process open the file for appending.

Exercise

A nightly job runs ./orbit-backup.sh 2>&1 > /var/log/orbit/backup.log, yet the error messages never appear in the log file. What is the fix?

  1. Use >> instead of > so the errors are appended
  2. Swap the order to ./orbit-backup.sh > /var/log/orbit/backup.log 2>&1
  3. Add 2> /dev/null so errors are flushed
  4. Pipe through tee because redirection cannot capture stderr
Show answer

Answer: B. Redirections apply left to right. 2>&1 first copies stderr to where stdout points at that moment (the terminal or cron’s mail), and only then is stdout moved to the file. Redirect stdout first, then duplicate stderr onto it.

Exercise

You count requests per IP with awk ‘{print $1}’ access.log | uniq -c and the same IP appears on dozens of separate lines with small counts. Why?

  1. awk splits IPv4 addresses on the dots
  2. uniq -c only counts the first 1000 lines
  3. The log has Windows line endings
  4. uniq only merges adjacent duplicates, so the input must be sorted first: awk ‘{print $1}’ access.log | sort | uniq -c
Show answer

Answer: D. uniq compares each line with the previous one only. Sorting groups identical values together so uniq -c can count each one once; add sort -rn afterwards to rank them.

Key takeaways

  • Every process has stdin (0), stdout (1) and stderr (2); redirect them with >, >>, 2>, 2>&1 (order matters) or &>, and discard with /dev/null.
  • Pipes connect small tools into a stream; tee saves a copy, and sudo tee is how you write root-owned files.
  • grep finds lines, cut and awk pick columns, sort, uniq -c and wc count, tr and sed transform.
  • The pattern awk | sort | uniq -c | sort -rn | head answers most “who, what, how many” questions about a log.
  • Use tail -F for rotated logs, sed -i.bak for safe edits, xargs -0 for file lists and jq for JSON.

FAQ

What is the difference between > and >> in Linux?

> creates the file or truncates it to zero bytes before writing, so its old content is gone. >> creates the file if needed and appends to the end. Use >> for logs and anything you want to keep; in Bash, set -o noclobber makes > refuse to overwrite an existing file as a safety net.

What does 2>&1 mean?

It duplicates file descriptor 2 (stderr) onto whatever file descriptor 1 (stdout) currently points to. Combined with a redirect, as in cmd > out.log 2>&1, both normal output and errors end up in the same file. The order matters because the shell processes redirections from left to right.

Should I use grep, sed or awk?

Use grep to find lines, sed to change text in lines (substitutions, deleting or printing ranges) and awk when you think in columns or need arithmetic such as sums, averages and comparisons. They overlap, and a pipeline often uses all three. When the logic grows beyond one or two lines, a short Python script is clearer.

How do I follow a log file in real time?

Use tail -F /path/to/file, which keeps printing new lines and survives log rotation. Combine it with grep, for example tail -F access.log | grep --line-buffered '" 5', so filtered lines appear immediately. For systemd services, journalctl -u name -f does the same for the journal.

How do I search for text in all files in a directory?

Run grep -rn 'pattern' /path. -r recurses into subdirectories and -n shows line numbers; add -i to ignore case, -l to list only file names, and --include='*.conf' to limit the search to some file types. Send errors about unreadable files away with 2>/dev/null.

↑ ↓ navigate Enter open Esc close