Why this matters
Security work is mostly reading data the computer already wrote down: authentication logs, firewall logs, web-server access logs, CSV exports from a SIEM. These files are huge — a busy server writes tens of thousands of lines a day — so you never read them by hand. Instead you build a pipeline: a chain of small text tools where each one does one job and passes its output to the next. One well-built line like grep "Failed password" auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn turns a wall of noise into a ranked list of the IP addresses hammering your server. Don't worry about the pieces yet — every tool in that line gets its own section below, and you'll be able to build this exact pipeline yourself by the end.
This sheet is the sequel to the terminal basics sheet. The core idea is simple: each tool reads text and writes text, so you can snap them together with the pipe symbol | like LEGO bricks. Learn six of them — grep, cut, sort, uniq, wc, plus a taste of awk and sed — and you can answer almost any "how many" or "which one is worst" question about a log file.
The pipe: chaining tools together
A pipe (|) takes the output of the command on its left and feeds it as the input of the command on its right. Nothing is saved to disk in between — the text just flows through. Redirection (>, >>, <) connects a command to a file instead of to another command. Together these let you assemble a pipeline and then park the result in a file.
The plumbing that connects everything else. The tools in these examples — grep, sort, uniq, wc, and so on — each get their own section below; for now just watch how | and > move text from one command to the next.
| Command | What it does | Example |
|---|---|---|
| cmd1 | cmd2 | | (pipe) - send cmd1's output straight into cmd2 as its input, with nothing saved to disk in between | grep Failed auth.log | wc -l |
| cmd > file | > (redirect) - send output into a file, OVERWRITING whatever was there | sort ips.txt > sorted.txt |
| cmd >> file | >> (append redirect) - add output to the END of a file instead of overwriting it | grep Failed auth.log >> failures.txt |
| cmd < file | < (input redirect) - feed a file in as the command's standard input | sort < ips.txt |
| cmd 2> file | 2> (error redirect) - send only error messages (stderr) to a file, keeping them out of the pipe | curl example.com 2> errors.txt |
| cmd 2>&1 | 2>&1 - merge the error stream into the normal output stream so both flow down the same pipe | make 2>&1 | grep -i error |
| cmd | tee file | tee - named after the T-shaped pipe fitting: copy the stream to a file AND pass it on down the pipe | grep Failed auth.log | tee failed.txt | wc -l |
| head -n 5 / tail -n 5 | head / tail - both plain English words: print only the first / last 5 lines, perfect for peeking mid-pipeline | sort -rn counts.txt | head -n 5 |
Build one stage at a time
Don't write the whole pipeline in one go. Add one | stage, pipe it into head, look at the output, and only then add the next stage. You catch a wrong delimiter or field number immediately instead of debugging a five-stage monster.
> truncates before it runs
> empties the target file the instant you press Enter, before the command produces anything. Never write sort file.txt > file.txt — the shell blanks file.txt first and you lose your data. Write to a new name, then rename.
grep — find the lines you care about
grep prints every line that matches a pattern and drops the rest. It's almost always the first stage of a pipeline — you narrow 50,000 lines down to the 1,400 that mention a failed login before you do anything else. The pattern can be plain text or a regular expression.
| Flag | What it does | Example |
|---|---|---|
| grep PATTERN file | Global Regular Expression Print - print every line of the file that matches PATTERN and drop the rest | grep "Failed password" auth.log |
| -i | ignore case - match Error, error, and ERROR alike | grep -i error syslog |
| -v | invert - print the lines that do NOT match | grep -v " 200 " access.log |
| -c | count - print how many LINES matched (not total occurrences) | grep -c "Failed" auth.log |
| -n | line number - prefix each match with its line number in the file | grep -n root /etc/passwd |
| -r | recursive - search every file under a directory, walking subfolders | grep -r "api_key" ./src |
| -o | only-matching - print just the matched text, one match per line | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' auth.log |
| -E | extended regex - enable + ? | ( ) { } without backslashes | grep -E "warn|error" syslog |
| -w | word - match whole words only (root, not chroot) | grep -w root /etc/passwd |
| -l | list files - print only the NAMES of files that contain a match, not the matching lines | grep -rl "api_key" ./src |
| -q | quiet - print nothing and just set the exit status (found / not found), handy inside scripts | grep -q root /etc/passwd && echo present |
| -A / -B / -C N | after / before / context - show N lines After / Before / around each match | grep -C2 segfault kern.log |
-c counts lines, not hits
grep -c counts how many LINES matched, even if a line contains the pattern three times. To count total occurrences, use grep -o PATTERN file | wc -l — -o puts each hit on its own line first.
cut — pull out columns and fields
Log lines and CSV rows are made of fields separated by a delimiter — a comma in a CSV, a colon in /etc/passwd, a space in a log. cut slices out the field(s) you want and throws the rest away. Its default delimiter is a TAB, so for anything else you must set -d.
| Option | What it does | Example |
|---|---|---|
| cut -f LIST file | cut - a plain English word (not an acronym): slice out selected fields or columns from each line and throw the rest away | cut -d, -f1,2 access.csv |
| -d CHAR | delimiter - set the field separator (default is a TAB) | cut -d, -f2 access.csv |
| -f N | fields - keep field number N | cut -d: -f1 /etc/passwd |
| -f N,M | fields (list) - keep several specific fields | cut -d, -f1,4 access.csv |
| -f N-M | fields (range) - keep a contiguous range of fields | cut -d, -f2-4 access.csv |
| -c N-M | characters - keep character positions N through M (position, not fields) | cut -c1-19 auth.log |
| -s | separated only - skip lines that contain no delimiter at all instead of passing them through whole | cut -s -d, -f2 access.csv |
| --complement -f N | complement - keep everything EXCEPT the listed field(s) | cut --complement -d, -f1 access.csv |
cut splits on ONE character only
cut treats each single delimiter character as a break, so two spaces in a row make an empty field. Log lines often have runs of spaces — when column positions drift, reach for awk (below), which collapses whitespace automatically.
sort & uniq — count and rank
This pair is the heart of log analysis. sort puts lines in order; uniq collapses repeats. The classic combo sort | uniq -c | sort -rn turns a raw column of values into a frequency table ranked from most to least common — exactly how you find the noisiest IP, the most-hit URL, or the busiest user.
sort — put lines in order.
| Option | What it does | Example |
|---|---|---|
| sort file | sort - a plain English word: put lines in order, alphabetically (dictionary order) by default | sort names.txt |
| -n | numeric - sort by numeric value so 9 comes before 10 (not after 100) | sort -n sizes.txt |
| -r | reverse - flip the order so the largest / last comes first | sort -rn counts.txt |
| -u | unique - sort AND drop duplicate lines in one step | sort -u ips.txt |
| -f | fold case - treat upper- and lower-case as equal while comparing | sort -f users.txt |
| -t CHAR -k N | field separator / key - sort by column N using CHAR as the delimiter | sort -t, -k4 -n access.csv |
| -h | human-numeric - understand size suffixes like 2K, 5M, 1G when sorting | sort -h sizes.txt |
uniq — collapse ADJACENT repeated lines.
| Option | What it does | Example |
|---|---|---|
| uniq file | unique (shortened) - collapse ADJACENT identical lines into one (sort first so repeats sit next to each other) | sort ips.txt | uniq |
| -c | count - prefix each line with how many times it repeated | sort ips.txt | uniq -c |
| -d | duplicates - show only the lines that appear more than once | sort ips.txt | uniq -d |
| -u | unique-only - show only the lines that appear exactly once | sort ips.txt | uniq -u |
| -i | ignore case - treat lines that differ only in case as the same | sort users.txt | uniq -ci |
| -f N | skip fields - ignore the first N fields when deciding whether lines match | sort log.txt | uniq -f1 -c |
uniq only sees its neighbors — always sort first
uniq compares each line only to the one directly above it, so it misses duplicates that aren't next to each other. On unsorted data it silently under-counts. The rule is unbreakable: **sort before uniq -c, every single time.**
wc — how many?
wc (word count) answers the raw "how much" questions. In pipelines you'll use wc -l constantly to count the lines that survived your filters — the number of failed logins, blocked packets, or matching rows.
| Option | What it does | Example |
|---|---|---|
| wc file | Word Count - print the line, word, and byte counts of a file (three numbers, in that order) | wc auth.log |
| -l | lines - count only the number of lines | wc -l auth.log |
| -w | words - count whitespace-separated tokens | wc -w report.txt |
| -c | bytes - count the number of bytes | wc -c payload.bin |
| -m | chars - count characters (differs from -c on multi-byte UTF-8) | wc -m notes.txt |
| -L | longest line - print the length of the longest line, in characters | wc -L access.log |
| ... | wc -l | lines through a pipe - count the lines flowing through a pipe (the most common pipeline ending) | grep Failed auth.log | wc -l |
-c vs -m
wc -c counts bytes and wc -m counts characters. They match for plain ASCII but differ when a file has accented letters or emoji, since those take several bytes each. For file size you want -c; for "how many letters" you want -m.
awk & sed — a light touch
awk and sed are small programming languages, but you only need a few tricks. **awk** shines when you want a specific field but the columns drift — it splits on runs of whitespace and gives you $1, $2, and so on, plus NF (number of fields on the line) and NR (current line number). **sed** edits streams of text — its bread and butter is find-and-replace.
awk — field extraction and quick filters.
| Snippet | What it does | Example |
|---|---|---|
| awk 'program' file | named after its authors Aho, Weinberger & Kernighan - a pattern-scanning language that splits each line on whitespace into fields $1, $2, and so on, and runs your program on every line | awk '{print $0}' access.log |
| awk '{print $1}' | $1 = first field - print the first whitespace-separated field of each line | awk '{print $1}' access.log |
| awk -F, '{print $2}' | -F = field separator - use a comma as the delimiter, then print field 2 | awk -F, '{print $2}' access.csv |
| awk '{print $(NF-3)}' | NF = number of fields - count fields from the END so $NF is the last field and $(NF-1) the one before it | awk '{print $(NF-3)}' auth.log |
| awk '$9 == 404' | pattern/condition - print only the lines where field 9 equals 404 | awk '$9 == 404' access.log |
| awk '{s += $1} END {print s}' | END block - after the last line is read, print the running total of column 1 | awk '{s += $1} END {print s}' bytes.txt |
| awk 'NR >= 5 && NR <= 10' | NR = record (line) number - print only a chosen range of line numbers | awk 'NR >= 5 && NR <= 10' log.txt |
sed — stream editing (mostly search-and-replace).
| Snippet | What it does | Example |
|---|---|---|
| sed 'script' file | Stream EDitor - apply an editing script to text as it streams past. By default it prints the edited result and leaves the original file untouched | sed 's/foo/bar/' notes.txt |
| sed 's/old/new/' | s = substitute - replace the FIRST 'old' on each line with 'new' | sed 's/http:/https:/' urls.txt |
| sed 's/old/new/g' | g = global flag - replace ALL occurrences on each line, not just the first | sed 's/,/ /g' data.csv |
| sed -n '5,10p' | -n + p = suppress + print - print only lines 5 through 10 (-n mutes the default output, p prints the chosen range) | sed -n '5,10p' auth.log |
| sed '1d' | d = delete - drop the first line (for example a CSV header) | sed '1d' access.csv |
| sed -i 's/a/b/g' file | -i = in place - rewrite the file itself, so the changes are permanent with no undo | sed -i 's/DEBUG/INFO/g' app.log |
sed -i rewrites the real file
Without -i, sed just prints the edited text to the screen and leaves the file untouched — safe to experiment with. Add -i and it overwrites the original with no undo. Always run the command WITHOUT -i first, confirm the output looks right, then add -i.
Building a pipeline stage by stage
Here's the whole workflow on a real question: which IP addresses are generating the most failed SSH logins? A flood of failures from one address is the classic fingerprint of a brute-force or password-spray attempt, so this is a report a defender runs constantly. Watch how each stage adds exactly one idea, and how we check the output before adding the next |.
Why awk beat cut here
The two log formats — Failed password for root versus Failed password for invalid user admin — put the IP in different numbered columns. A fixed cut -d' ' -f13 would grab the wrong word on half the lines. awk '{print $(NF-3)}' counts backward from the end, and the IP always sits 3 fields from the end — right before the trailing port NNNNN ssh2 — so it stays correct in both cases. When column positions drift, count from the end.
About the addresses in these examples
The IPs here (203.0.113.x, 198.51.100.x, 192.0.2.x) come from ranges RFC 5737 reserves for documentation. They're intentionally fake so nothing in this sheet points at a real host — use them freely in your own practice files.
Try it yourself
auth.logDownload auth.log: https://restedteacher.com/practice-files/cybersecurity/auth.log
A 38-line sample SSH authentication log — Failed and Accepted password lines, in the exact two Failed formats this sheet teaches. Every IP address comes from a range reserved for documentation, so nothing in the file points at a real computer.
Get to the file first
Your browser saves the file to your Downloads folder. Move it into a labs folder and go there, using commands from the terminal basics sheet:
mkdir -p ~/labs && mv ~/Downloads/auth.log ~/labs/ && cd ~/labs
Then run ls and confirm you can see auth.log. On WSL, your Windows Downloads folder is /mnt/c/Users/<your-name>/Downloads instead of ~/Downloads.
Using auth.log, write ONE command for each question, building it one pipe at a time and checking with head after each stage:
1. Count the failures. How many lines contain Failed password? Answer with grep piped into wc -l, then confirm the same number with grep -c.
2. Rank the noisiest source IPs. Reproduce the marquee pipeline from this sheet — filter to the Failed password lines, pull the IP with awk '{print $(NF-3)}', then sort | uniq -c | sort -rn — to rank every source IP from most failures to least. Which IP comes out on top, and how many failures does it have?
3. Find the most-targeted username. In Failed password for <user> from ... (with or without the invalid user prefix), the username always sits two fields before the IP, no matter which of the two formats the line uses. Adjust the awk field index from question 2 to pull the username instead, then sort | uniq -c | sort -rn to rank the accounts attackers tried hardest. Which username was targeted most, and how many times?
Everything here is read-only log analysis — you're inspecting records a defender already collected, not touching any live system. That's the daily job of a SOC analyst: ask a question, build a pipeline, read the answer.