Stage 2 · Tools

Text Processing

awk — Data Processing

Field splitting, patterns, actions, BEGIN/END blocks, and arrays for structured data.

8 min readShell Scripting & AutomationTools

awk Fundamentals

awk is a complete programming language designed for text processing. It reads input line by line, splits each line into fields, and executes patterns and actions. awk is the most powerful text processing tool in the Unix toolkit.

awk command structure


# Basic syntax
awk 'pattern { action }' file.txt

# Print specific field (column)
awk '{ print $2 }' file.txt

# Print with custom delimiter
awk -F: '{ print $1 }' /etc/passwd

# Multiple actions
awk '{ print $1, $3 }' file.txt

# Complex action block
awk '{
    name = $1
    count = $2
    total = name " has " count " items"
    print total
}' data.txt

awk processes each line of input. $1 is the first field, $2 is the second, and $0 is the entire line. Fields are separated by whitespace by default, or by the character specified with -F.

Fields and Separators

awk automatically splits each line into fields based on the field separator. You can change the separator and even modify it between lines.

Field separator examples


# Default: whitespace
echo "hello world foo" | awk '{ print $2 }'   # world

# Colon separator (common for /etc files)
awk -F: '{ print $1, $3 }' /etc/passwd

# Comma separator (CSV)
awk -F, '{ print $2 }' data.csv

# Multiple character separator
awk -F'[:/]' '{ print $1 }' /etc/passwd

# Change separator within awk
awk 'BEGIN { FS = ":" } { print $1 }' /etc/passwd

FS (field separator) controls how awk splits lines. You can set it with -F on the command line or assign it in a BEGIN block. RS (record separator) controls what defines a line — default is newline.

Patterns and Actions

awk patterns work like grep — they filter which lines get processed. Actions are the code that runs for matching lines. You can use patterns without actions (prints the line) or actions without patterns (applies to all lines).

Pattern types


# Regex match
awk '/ERROR/ { print $0 }' logfile.txt

# Field comparison
awk '$3 > 100 { print $1, $3 }' data.txt

# String equality
awk '$1 == "admin" { print $0 }' users.txt

# Range patterns
awk '/START/,/END/ { print }' file.txt

# Compound conditions
awk '$3 > 100 && $2 == "critical" { print }' data.txt

# Negation
awk '!/^#/ { print }' config.txt  # Skip comments

Patterns in awk support regex, field comparisons, range patterns, and logical operators. The pattern is evaluated for each line — if true, the action block runs.

BEGIN and END Blocks

BEGIN runs before processing any input. END runs after all input is processed. The main pattern/action block runs for each line. This three-phase structure is awk's most powerful pattern.

BEGIN and END in practice


# Print CSV header and process data
awk 'BEGIN { print "Name,Score" } { print $1 "," $2 }' scores.txt

# Calculate average
awk 'BEGIN { sum=0; count=0 }
     { sum += $2; count++ }
     END { print "Average:", sum/count }' scores.txt

# Find max value
awk 'BEGIN { max=0 } $2 > max { max=$2; name=$1 }
     END { print "Max:", name, max }' scores.txt

# Process all lines with summary
awk '
BEGIN { print "=== Report ===" }
/ERROR/ { errors++ }
/WARN/ { warnings++ }
END {
    print "Errors:", errors+0
    print "Warnings:", warnings+0
}' logfile.txt

BEGIN blocks are for initialization — setting variables, printing headers. END blocks are for summaries — computing averages, printing totals. The main block processes each line.

Uninitialized variables are 0 (numeric) or empty (string)

In awk, using a variable before assigning it gives 0 for numeric contexts and empty string for string contexts. This is useful: errors++ works even if errors was never initialized.

Built-in Variables

awk provides several built-in variables that give you information about the current state of processing.

Variable	Description	Example
NR	Current line number (global)	NR == 5 (line 5)
NF	Number of fields in current line	$NF (last field)
FS	Field separator (input)	FS = ","
OFS	Output field separator	OFS = ","
RS	Record separator (input)	RS = "" (paragraph mode)
FILENAME	Name of current file	FILENAME

Built-in variables in action


# Print line numbers
awk '{ print NR, $0 }' file.txt

# Print last field of each line
awk '{ print $NF }' file.txt

# Print number of fields
awk '{ print NF }' file.txt

# Use OFS for output formatting
awk -F, 'BEGIN { OFS=":" } { $1=$1; print }' data.csv

# Process multiple files
awk '{ print FILENAME ":" $0 }' file1.txt file2.txt

NR counts lines across all files. FNR counts lines within the current file. NF gives the field count — $NF is the last field, $(NF-1) is the second-to-last.

Arrays in awk

awk has associative arrays (like Bash declare -A). They are indexed by strings, not numbers. This makes them perfect for counting, grouping, and aggregating data.

awk arrays


# Count occurrences of each value
awk '{ count[$1]++ } END { for (k in count) print k, count[k] }' data.txt

# Group lines by field
awk '{ groups[$1] = groups[$1] " " $0 }' data.txt

# Find unique values
awk '!seen[$0]++ { print }' file.txt

# Build a lookup table
awk -F: '{ users[$1] = $3 } END { for (u in users) print u, users[u] }' /etc/passwd

# Sum values per category
awk -F, '{ totals[$1] += $3 }
END { for (cat in totals) print cat, totals[cat] }' sales.csv

awk arrays are always associative. array[key]++ increments a counter. !seen[$0]++ prints only the first occurrence of each line. The for (k in array) loop iterates over all keys.

Use awk for quick reports

awk is perfect for one-liner reports: awk '{ sum += $1 } END { print sum }' numbers.txt calculates a sum. These mini-programs are faster and more readable than equivalent shell loops.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.