HowToCodePage

Text processing: grep, sed and awk in scripts

grep options and exit status, sed substitutions and in-place edits, awk fields and patterns, cut, sort and uniq, and choosing the right tool for the job.

grep: searching and exit status

grep -c 'ERROR' app.log            # count matching lines
grep -n 'ERROR' app.log | head     # line numbers
grep -r --include='*.py' -n 'TODO' src/     # recursive, filtered by glob
grep -E 'warn|error|fatal' app.log          # extended regex: alternation
grep -F 'a.b.c' config.txt                  # fixed string, no regex at all
grep -v '^#' config | grep -v '^[[:space:]]*$'   # drop comments and blanks
grep -oE '[0-9]{4}-[0-9]{2}-[0-9]{2}' log | sort -u

if grep -q 'READY' startup.log; then
    echo "service reported ready"
else
    echo "service did not become ready" >&2
    exit 1
fi
Exit statusMeaning
0At least one line matched
1No line matched — not an error, an answer
2Something went wrong: bad option, unreadable file

Under set -e, a 1 from grep aborts the script even though "no match" is a legitimate result. Put the call in an if or append || true to say so explicitly.

sed: substitutions and edits

sed -n '10,20p' file              # print only a range
sed '/^$/d' file                   # delete blank lines
sed 's/foo/bar/' file              # first match per line
sed 's/foo/bar/g' file             # every match
sed -E 's/([0-9]+)ms/\1 ms/' file  # capture group, with -E
sed -e 's/a/b/' -e 's/c/d/' file   # several expressions
sed -i.bak 's/localhost/db.internal/g' config.ini   # edit in place, keep a backup
sed '1i # managed by automation' file > out        # insert a line at the top
  • -E gives you (), + and | without backslashes; back up references as \\1 in the replacement.
  • -i takes an optional suffix: -i.bak writes file.bak and is the only way to run an in-place edit and still recover from it.
  • Change the delimiter when the data contains slashes: sed 's|/var/app|/srv/app|g'.
  • sed is line-oriented. Anything that needs to compare one line with another belongs in awk.

awk, cut, sort and choosing

awk '{ print $1, $3 }' access.log          # select columns
awk -F, '{ sum += $3 } END { print sum }' data.csv
awk '$9 >= 500 { print $7 }' access.log    # filter by a field
awk -F: 'NR > 1 && $3 >= 1000 { print $1 }' /etc/passwd
awk '/ERROR/ { errors++ } END { print errors + 0 }' app.log

cut -d, -f1,3 data.csv                      # fixed columns, fast
sort -t, -k3,3nr data.csv                   # numeric, descending, third field
sort file | uniq -c | sort -rn | head       # top repeated lines
tr -s ' ' '\t' < file                      # squeeze spaces into tabs
TaskBest toolWhy
Does this pattern appear?grep -qExit status only, no parsing
Rewrite text on matching linessedLine-oriented by design
Sum, filter or regroup columnsawkReal fields, numbers and state
Extract fixed columns from well-formed datacutSimple and very fast
Rank or de-duplicatesort + uniqStreaming and predictable
Structured data (JSON, YAML)jq / yqAwk on JSON breaks on the first escaped quote
⚠️
sed -i and sort -o are not portable in the same way across GNU and BSD userland: BSD sed -i requires a suffix argument, so sed -i 's/a/b/' f fails on macOS. Write to a temporary file and move it, or branch on uname.

FAQ

Which tool should I learn first?
grep for finding, awk for summarising, sed for editing. Most real pipelines are grep to narrow, awk to reduce, sort to order.
Why is my awk command failing inside a script?
The shell is expanding $1 and $9 before awk sees them. Put the program in single quotes, or escape the dollars and use double quotes when you need shell variables inside.

Files, redirection and here-documents Parameter expansion and string manipulation

Last refreshed 2026-09-18.