Stage 2 · Tools
Text Processing
cut, sort, uniq, tr
Extract columns, sort output, count occurrences, and translate characters — the utility belt.
cut — Column Extraction
cut extracts columns (fields) or characters from each line. It is simpler than awk but faster for basic column extraction. Use cut when you just need specific fields without processing logic.
# Extract fields by delimiter
echo "alice:1000:home" | cut -d: -f1 # alice
echo "alice:1000:home" | cut -d: -f1,3 # alice:home
# Extract by character position
echo "hello world" | cut -c1-5 # hello
# Extract bytes
echo "abcde" | cut -b1-3 # abc
# Multiple delimiters (cut does not support this — use awk)
# For multiple delimiters, use awk -F'[:/]' insteadcut is optimized for simple field extraction. It is faster than awk for basic operations but cannot handle complex conditions or calculations. Use it when awk's power is not needed.
For simple CSV extraction, cut is faster than awk: cut -d, -f2 data.csv extracts the second column. For CSV with quoted fields containing commas, use awk or a dedicated tool.
sort — Ordering Data
sort orders lines alphabetically, numerically, by field, or by various other criteria. It is essential for preparing data for uniq, diff, and other tools.
# Alphabetical sort
sort names.txt
# Reverse sort
sort -r names.txt
# Numeric sort (important!)
sort -n numbers.txt
# Sort by specific field
sort -t: -k3 /etc/passwd # Sort by 3rd field (UID)
# Sort by multiple fields
sort -t, -k1,1 -k2,2n data.csv # Sort by column 1, then column 2 numerically
# Remove duplicates while sorting
sort -u file.txt
# Sort with case insensitivity
sort -f names.txt
# Human-readable numbers (1K, 2M, 3G)
sort -h sizes.txtThe -n flag is critical — without it, sort uses alphabetical ordering where 10 comes before 2. The -t flag sets the field delimiter. -k specifies which fields to sort by.
Without -n, sort treats numbers as strings: 1, 10, 2, 20 becomes 1, 10, 2, 20 (alphabetical). With -n: 1, 2, 10, 20 (numeric). This is the most common sort mistake.
uniq — Deduplication and Counting
uniq removes or reports duplicate lines. It only detects adjacent duplicates, so you almost always pipe sort into uniq first. The -c flag counts occurrences.
# Remove adjacent duplicates
sort file.txt | uniq
# Count occurrences (most common use)
sort file.txt | uniq -c
# Sort by frequency (most frequent first)
sort file.txt | uniq -c | sort -rn
# Show only duplicates
sort file.txt | uniq -d
# Show only unique lines (no duplicates)
sort file.txt | uniq -u
# Ignore first N fields
uniq -f 2 file.txtuniq -c with sort -rn is the standard pattern for frequency analysis. sort file | uniq -c | sort -rn gives you a frequency count sorted from most to least common.
tr — Character Translation
tr translates or deletes individual characters. It works on streams — it does not understand lines or fields. Use it for character-level transformations.
# Translate characters
echo "Hello World" | tr '[:upper:]' '[:lower:]' # hello world
echo "hello world" | tr ' ' '_' # hello_world
# Delete characters
echo "abc123def456" | tr -d '[:alpha:]' # 123456
echo "abc123def456" | tr -d '[:digit:]' # abcdef
# Squeeze repeated characters
echo "hello world" | tr -s ' ' # hello world
# Complement (translate everything except)
echo "abc123" | tr -cd '[:digit:]' # 123
# Translate newlines to spaces
cat file.txt | tr '\n' ' 'tr works character by character, not word by word. tr 'abc' 'xyz' replaces a with x, b with y, c with z. Character classes like [:upper:], [:lower:], [:digit:] are portable.
Combining Utilities
These utilities shine when combined in pipelines. Each tool does one thing well — together they form powerful data processing pipelines.
# Top 10 most frequent visitors
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
# Extract and sort unique IPs
grep -oE "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" log.txt | sort -u
# Count unique error types
grep "ERROR" app.log | awk -F: '{print $2}' | sort | uniq -c | sort -rn
# Clean and normalize text
cat input.txt | tr '[:upper:]' '[:lower:]' | tr -cs '[:alnum:]' '\n' | sort -u
# Find duplicate lines in config files
sort config.txt | uniq -dPipeline composition is the Unix philosophy in action. Each tool is simple and focused. Combined, they handle complex data processing tasks that would require hundreds of lines in other languages.
grep (filter) -> sed/awk (transform) -> sort (order) -> uniq (deduplicate) -> cut/paste (restructure) -> wc (count). This is the fundamental pattern for text processing in shell scripts.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.