Linux text pipelines: turn six log lines into two useful counts

The log has six lines, but six is not the answer you need. You want to know which routes returned server errors, and how often each one failed. Two failures belong to the same route. Somewhere between reading the file and printing a total, those two need to meet.

A Linux pipeline connects small programs so one can work on the text another produces. That makes this question manageable: select the server-error rows, keep their route names, then count matching names. Each intermediate result is small enough to read.

Use the disposable Ubuntu virtual machine for this synthetic exercise. The commands below create a practice file in ~/linux-basics. Each line has exactly four space-separated fields: time, method, route and numeric status. Real web-server logs may use quite different formats; this one is deliberately simple.

mkdir -p ~/linux-basics
cd ~/linux-basics
cat > requests.log <<'LOG'
14:00:01 GET /exports 200
14:00:02 GET /exports/7 503
14:00:03 GET /health 200
14:00:04 GET /exports/8 504
14:00:05 GET /exports/7 503
14:00:06 GET /missing 404
LOG

The status column decides which rows belong

Start with cat requests.log to see the whole file. Then give Awk, a text-processing program, a condition to apply to each line:

awk '$4 >= 500 && $4 < 600 {print}' requests.log

Awk splits these lines at whitespace, and $4 names the fourth field: the status. The condition keeps values from 500 through 599. For this fixture, that means three rows, two for /exports/7 and one for /exports/8.

Notice what the command does not do. It does not hunt for the characters “500” somewhere in the line. That search would miss a 503 and might find an unrelated number in a route. Here, both the place we look and the range we accept are part of the question.

The 404 stays outside the selection because it is a client error, not a server error. It may still deserve investigation. It simply does not belong in this particular count.

Two copies of the same route need to meet

The selected rows still contain times, methods and statuses. Now that status has done its job, keep only the route, then watch what each added program changes:

awk '$4 >= 500 && $4 < 600 {print $3}' requests.log
awk '$4 >= 500 && $4 < 600 {print $3}' requests.log | sort
awk '$4 >= 500 && $4 < 600 {print $3}' requests.log | sort | uniq -c

The first line prints field three. Reading downward, the result is /exports/7, /exports/8, and /exports/7 again. Add sort, and the two copies of /exports/7 become neighbors. Add uniq -c, and those neighboring equal lines become a count: two for /exports/7, one for /exports/8.

The sort is doing more than making the output tidy. Without it, uniq sees three separate runs of one line because the other route sits between the duplicates. It counts neighboring equal lines, not every appearance of a name wherever it happens to be.

The | character makes this a pipeline. It sends a program's standard output, its ordinary result stream, into the next program's standard input. Once Awk prints just the routes, the status column is gone. That explains the order: select by status while you still have it, then discard it. The next program receives the text you passed along, not a memory of the original four columns.

Error messages usually travel separately, through standard error to the terminal. If the file cannot be read, one program can complain while the later programs finish with no rows. Bash normally reports the last command's status for a pipeline; set -o pipefail makes an earlier failure affect that status too. Read the error output before deciding an empty result means there were no server errors.

The totals get their own file

Once the intermediate steps make sense, keep the result beside the log:

awk '$4 >= 500 && $4 < 600 {print $3}' requests.log | sort | uniq -c > server-errors.txt
cat server-errors.txt
wc -l requests.log

The > sends the counts to server-errors.txt, while wc -l requests.log counts the original lines. Use a different output filename from the input. The shell can truncate, or empty, an output file before the reader starts. Redirecting this command into requests.log could therefore erase the very rows you meant to count.

The small fixture makes a useful prediction possible. Append 14:00:07 GET /exports/8 502 as a new line in the practice log and repeat the pipeline. Both export routes should now count two: 502 passes the same condition as 504. Add another /health row with status 200 instead, and the file gets longer while both error counts stay where they were.

There is one more comparison worth making in this disposable exercise: deliberately misspell the input filename. The error describes a file that could not be read. A genuine empty selection describes a file that was read but had no matching rows. Either can leave an empty output file, so the file alone does not tell the whole story.

When a line has more to say

Our calculation works because every row is well formed and none of its four fields contains a space. A JSON log, quoted text, a multiline event or a different timestamp format needs a parser that understands that structure. Moving to a larger field number will not repair a line that was split incorrectly in the first place.

Keep the raw log, command and expected counts together. The two totals now have an explanation you can follow back through every step: which rows stayed, which columns disappeared and why the repeated route was counted twice. When another line arrives, you can predict what it should change before asking the computer.

Sources

The manuals for GNU Awk, Coreutils and Bash describe the field selection, sorting, redirection and pipeline-status behavior used here.