On this page2 sections
A scheduled job tries to count records in a CSV file. In this hypothetical run, the input is missing its service column. The program prints an ERROR message and exits with status zero. Anyone reading the log can see the count failed. The scheduler, looking at the exit status, has every reason to mark the run successful.
Give the two readers a result they can use. The scheduler needs the process exit status, a small number the operating system reports when the program ends. The responder needs the reason the job stopped. Python lets the program send those answers separately without mixing either one into the count.
Follow the count from the file to the scheduler
You’ll need Python 3 and a terminal. Save the program as count_records.py, put a CSV file beside it and run python3 count_records.py input.csv. The CSV needs a service column with a nonblank value on every record. A file with just the header is a useful first input. The program should print 0 to standard output, the stream where callers can read the count, and send its completion log separately to standard error.
import argparse
import csv
import logging
log = logging.getLogger("record_counter")
def count_records(path):
with open(path, newline="", encoding="utf-8") as stream:
rows = csv.DictReader(stream, strict=True)
if not rows.fieldnames or "service" not in rows.fieldnames:
raise ValueError("CSV must contain a service column")
count = 0
for row in rows:
if not (row.get("service") or "").strip():
raise ValueError("CSV contains an empty service value")
count += 1
return count
def main():
parser = argparse.ArgumentParser()
parser.add_argument("path")
args = parser.parse_args()
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s %(message)s",
)
try:
count = count_records(args.path)
except (OSError, UnicodeError, ValueError, csv.Error):
log.exception("record_count_failed")
return 1
print(count)
log.info("record_count_completed records=%d", count)
return 0
if __name__ == "__main__":
raise SystemExit(main())
Now try a truly empty file. Before the program can count a row, it needs to find the required column. The empty file never supplies that structure, so count_records raises a ValueError. A row with a blank service value can fail later, after some rows have already passed. The caller gets no count in either case, because a partial tally would look too much like a valid answer.
A file can also stop halfway through a quoted field. Without strict=True, the reader can quietly treat the remaining lines as one field and count it. The strict setting makes that unfinished field raise csv.Error instead; a properly closed field can still span several lines. This example checks for those parsing errors and a missing or blank service field. It is not a complete check of every rule your CSV feed may need.
Notice where the message appears. The counting function raises the exception with the problem it found. The caller, main, decides that the whole count has failed. That is where log.exception writes one error record with the traceback, which shows the calls that led to the failure. Logging the same exception in both places would give an investigator two records for one failed count.
main then returns one, and SystemExit passes that value out as the exit status in this script. The header-only file still has a column and yields a legitimate zero count with a success status. The empty file yields no count and a failure status. That distinction survives even if the scheduler never reads a line of Python logging.
Read diagram description
The CSV example logs a terminal exception once at the operation boundary and exits with status 1. A successful run prints the count and exits with status 0. Confirm that the scheduler preserves the process status. Diagram labels: Read + validate CSV: Required service field; strict CSV parsing; Success: Count to stdout; INFO to stderr; Failure: One exception log to stderr; no result; Exit 0: Caller sees success; Exit 1: Caller sees failure.
Another job might deliver a partial result after a recoverable error. In that case, define what partial output means and which status tells its caller to use it. Here, a complete count is the result, so an unreadable file or a missing service field sends the caller down the failure path.
Can the person on call still find the exception?
The default logging setup in this standalone example sends messages to standard error. Once a scheduler runs it, check where those messages go and who can retrieve them. A stable event name such as record_count_failed gives the responder something to search for. Component, version and a run identifier can connect the event to other evidence without copying credentials, support text or customer data into the log. A traceback can still contain paths or sensitive exception details, so give it suitable access and retention.
Follow the message past the scheduler too. A collector should preserve the timestamps and fields the search needs. It may use a buffer to hold a burst while forwarding catches up. If the receiver is slower than intake, the collector may slow the source, a behavior called backpressure, or eventually drop records according to its settings. The log strategy guide follows what those losses mean for an investigation.
Try a valid file and one without the service column through the actual scheduler once it is connected. The run without the column should show failure there and have one exception record the responder can retrieve. Now the scheduler can choose its failure path and the person on call can find out why the count never arrived.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- Python CSV strict parsingdocs.python.org
- Python logging.Logger.exceptiondocs.python.org
- Python sys.exitdocs.python.org
- Python StreamHandlerdocs.python.org
