Aggregate operational counts with Python

One service reports two errors in 100 requests; another reports nine in 900. Average their 2% and 1% rates and you get 1.5%. But together they handled eleven failures in a thousand requests: 1.1%. The difference is that the smaller service got an equal vote in the average. Our Python calculation will count requests instead.

The program reads a local JSON file using Python's standard library and returns combined requests, errors and error fraction. Invalid input fails the command. It does not fetch API data or prove every region replied; the surrounding collector must establish that coverage before calling the result a fleet total.

Why the combined error rate is 1.1%

Use Python 3 and a folder for the program and input. Each row must cover a distinct service over the same window, with integer request and error counts. Save this illustrative data as input.json:

[
  {"requests": 100, "errors": 2},
  {"requests": 900, "errors": 9}
]

Save the program alongside it as aggregate.py. Python's JSON module handles reading and writing, with no additional package needed.

import json
import sys

def aggregate(rows):
    if not isinstance(rows, list) or not rows:
        raise ValueError("Expected a nonempty array of count records")
    requests = errors = 0
    for row in rows:
        if not isinstance(row, dict):
            raise ValueError("Each record must be an object")
        n, e = row.get("requests"), row.get("errors")
        if type(n) is not int or type(e) is not int:
            raise ValueError("Counts must be integers")
        if n < 0 or e < 0 or e > n:
            raise ValueError("Require 0 <= errors <= requests")
        requests += n
        errors += e
    return {"requests": requests, "errors": errors,
            "error_fraction": errors / requests if requests else None}

def main():
    if len(sys.argv) != 2:
        print("Usage: python3 aggregate.py input.json", file=sys.stderr)
        return 2
    try:
        with open(sys.argv[1], encoding="utf-8") as stream:
            result = aggregate(json.load(stream))
    except (OSError, UnicodeError, ValueError) as exc:
        print(f"Aggregation failed: {exc}", file=sys.stderr)
        return 1
    print(json.dumps(result, allow_nan=False))
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

From that directory, run:

python3 aggregate.py input.json

A successful run exits with zero and prints:

{"requests": 1000, "errors": 11, "error_fraction": 0.011}

The larger service supplied nine tenths of the requests, so its 1% rate should pull the combined result closer to 1% than to 2%. Adding errors and requests separately preserves that weighting: eleven divided by a thousand is 0.011, or 1.1%.

Before adding a row, the function checks for a nonempty list of objects, integer counts, no negative values and no more errors than requests. Exact type(value) is int checks also reject JSON true and false. Python treats booleans as an integer subtype, but an on/off value is not a request count. Only validated counts reach the division.

Zero requests with zero errors is a valid row but supplies no population for a rate. The result contains error_fraction: null, because Python's None becomes JSON null. A hundred requests with no errors instead yields 0.0, an observed zero-error fraction. Keep that difference visible.

A missing file, malformed JSON or invalid row sends an error to standard error and returns status one without printing a partial result. Omitting the filename gives the usage message and status two. The logging and exit-status guide explains how those results reach a scheduler.

Valid counts do not prove complete collection

Imagine a fleet collector failing to reach one region and replacing the error with an empty list. Every received row might be valid while that region disappears. If it had more failures, the apparent fleet rate would improve even though no customer's request improved.

The arithmetic cannot discover an omitted source. A collection manifest must record expected and received regions, the shared window and failed pages. If east and west replied but central did not, say “east and west only; central unavailable.” That is useful scoped evidence. The caller can hold the fleet report or publish the narrower result; the received rows alone cannot justify calling it complete.

Separate an unreachable region, a successful source with no observations and a partial response. This function rejects an empty list, so represent a legitimate no-observations result elsewhere in the collector. Having nothing to count and failing to get the counts need different records.

Aggregate counts before dividing

Read diagram description

Disjoint populations over the same window: 2 + 9 = 11 errors and 100 + 900 = 1,000 requests, so the combined error fraction is 1.1%. Averaging the two percentages would incorrectly produce 1.5%. Diagram labels: Service A: 2 errors / 100 requests = 2%; Service B: 9 errors / 900 requests = 1%; Sum counts: 11 errors / 1,000 requests; Combined error fraction: 11 ÷ 1,000 = 0.011 = 1.1%.

An API client brings another set of failure states

An API reader adds authentication, connect and read timeouts, size limits, pagination and schema checks. Make unexpected status codes and malformed responses visible. Bounded retries with backoff let temporary trouble clear without allowing a scheduled collection to run forever.

Carry coverage into the final report. If a region's third page fails, keep which pages arrived, which did not, their observation window and freshness beside the number. Otherwise the arithmetic can erase a failure the collector already knew about.

The example also assumes disjoint populations and matching windows. It does not deduplicate, align time zones or reconcile different objectives. Settle those data rules before applying it across a fleet. The metrics and denominators guide explains why they alter the meaning of the fraction.

Try the whole route with complete data, a failed source and a valid source with no observations. One check verifies eleven out of a thousand becomes 1.1%. The other verifies that the report cannot disguise an unreachable region as a measured result. A useful total needs both.