Python logging and exit status in a scheduled job

In a hypothetical scheduled counting job, the program catches an exception, writes ERROR to its log and exits with zero. The person reading the log sees a failed job. The scheduler sees success and never takes its failure route. A more detailed error message would not make those two accounts agree.

The program has two audiences. Python's logging module gives the investigator levels, handlers and exception details. The scheduler needs the process result: zero when this count completed, nonzero when it failed. Writing ERROR is not the same operation as setting that result.

One exception record, one failing process status

The following program counts CSV records whose rows contain a service value. Save it as count_records.py and run python3 count_records.py input.csv. It uses the standard library, writes the count to standard output and operational messages to standard error, so the caller can read the result without also parsing the log.

import argparse
import csv
import logging

log = logging.getLogger("record_counter")

def count_records(path):
    with open(path, newline="", encoding="utf-8") as stream:
        rows = csv.DictReader(stream)
        if not rows.fieldnames or "service" not in rows.fieldnames:
            raise ValueError("CSV must contain a service column")
        count = 0
        for row in rows:
            if not (row.get("service") or "").strip():
                raise ValueError("CSV contains an empty service value")
            count += 1
        return count

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("path")
    args = parser.parse_args()
    logging.basicConfig(
        level=logging.INFO,
        format="%(asctime)s %(levelname)s %(name)s %(message)s",
    )
    try:
        count = count_records(args.path)
    except (OSError, UnicodeError, ValueError, csv.Error):
        log.exception("record_count_failed")
        return 1
    print(count)
    log.info("record_count_completed records=%d", count)
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Give it an invalid file and follow the path. The counting function checks for a service header, then rejects blank service values as it reads. Instead of returning a plausible partial count, it raises an exception. Main catches the listed failures, records the traceback with log.exception and returns one.

That is the place where this program decides the operation failed, so it records the error there once. Logging in the counting function and again in its caller would make one failure look like two separate error records. The lower-level function supplies the exception; the operation-level handler supplies the final account.

An empty file and a header-only file reveal why this matters. The first has not supplied the required structure and is rejected. The second has a service column and zero records, which is a valid count. Returning zero records in both cases would hide the difference between a completed count and an invalid input.

Finally, SystemExit passes main's return value out as the process status. The scheduler receives failure status one, while the responder receives the reason. Both can now describe the same failed count.

Logs and process status serve different readers

Read diagram description

The CSV example logs a terminal exception once at the operation boundary and exits with status 1. A successful run prints the count and exits with status 0. Confirm that the scheduler preserves the process status. Diagram labels: Read + validate CSV: Required service field; invalid input raises failure; Success: Count to stdout; INFO to stderr; Failure: One exception log to stderr; no result; Exit 0: Caller sees success; Exit 1: Caller sees failure.

Some jobs deliberately finish useful partial work after a recoverable error. Define what their output and status mean instead of treating every logged error alike. The caller must understand that contract if it is to choose the right next action.

From a local traceback to the collected record

As the program grows, stable event names and limited fields, such as component, deployment version and outcome, help someone find the relevant records. Correlation identifiers can connect events without copying the whole request. Keep credentials, raw support text and customer payloads out by default. Tracebacks can also expose paths or exception details, so their access and retention need care.

Then follow the log beyond the local terminal. A collector needs to preserve timestamps and fields and make delivery failures visible. A buffer absorbs a burst; backpressure slows intake when forwarding falls behind. Decide what happens when the buffer fills, because a missing record could be the next investigation's only clue. The log strategy guide follows that larger route.

Run valid and invalid inputs through the actual scheduler once it is connected. Does the invalid count remain a failed scheduled job, and can the responder still retrieve its explanation? When both answers are yes, the two readers from the opening finally receive the same story.