Free analyzer · No account required
AI evaluation scorecard
Compare a binary AI decision with reviewed outcomes and see which mistakes an accuracy score can hide.
Your inputs stay in this browser tab. Saving to this device is optional.
Draft name and device saving
Current draft matches the last save or initial state.
Complete your inputs
Calculated result
Awaiting complete inputsEnter your values to see the result, or choose Try an example to explore a completed worksheet.
How this result is calculated
The confusion matrix compares the AI decision with the reviewed outcome. Precision asks how many flagged cases were truly positive: TP ÷ (TP + FP). Recall asks how many true positive cases were caught: TP ÷ (TP + FN). Accuracy counts all correct decisions, while F1 = 2TP ÷ (2TP + FP + FN), equivalent to the harmonic mean of precision and recall when both are defined. A zero denominator produces an undefined result, not a perfect score.
Keep your work
Resume later
Save an editable JSON draft. Use Open draft to bring it back into this tool.
Share a report
Reports include your inputs, assumptions and any unresolved checks.
Read the complete report
# AI evaluation scorecard Tool defaults; replace with your settings before operational use. ## Review issues - Correctly flagged positive cases: enter a whole number from 0 to 1000000000000. - Incorrectly flagged negative cases: enter a whole number from 0 to 1000000000000. - Missed positive cases: enter a whole number from 0 to 1000000000000. - Correctly rejected negative cases: enter a whole number from 0 to 1000000000000. - Define what a positive decision means. ## Inputs Task, model and evaluation set: Not entered What counts as positive?: Not entered Correctly flagged positive cases: Not entered Incorrectly flagged negative cases: Not entered Missed positive cases: Not entered Correctly rejected negative cases: Not entered ## Method The confusion matrix compares the AI decision with the reviewed outcome. Precision asks how many flagged cases were truly positive: TP ÷ (TP + FP). Recall asks how many true positive cases were caught: TP ÷ (TP + FN). Accuracy counts all correct decisions, while F1 = 2TP ÷ (2TP + FP + FN), equivalent to the harmonic mean of precision and recall when both are defined. A zero denominator produces an undefined result, not a perfect score. This supports binary decisions with mutually exclusive, reviewed labels. It does not evaluate open-ended answer quality or prove an agent is safe to deploy. Dataset selection, class balance, annotation quality and threshold changes affect the result. Keep unseen test cases separate from examples used to tune the system. https://aiopssre.com/ai-evaluation-scorecard/
How it works
The confusion matrix compares the AI decision with the reviewed outcome. Precision asks how many flagged cases were truly positive: TP ÷ (TP + FP). Recall asks how many true positive cases were caught: TP ÷ (TP + FN). Accuracy counts all correct decisions, while F1 = 2TP ÷ (2TP + FP + FN), equivalent to the harmonic mean of precision and recall when both are defined. A zero denominator produces an undefined result, not a perfect score.
Follow the example
The example gets 90 of 100 decisions right, so accuracy is 90%. It catches 18 of the 20 positive cases, giving 90% recall. But 8 of the 26 flagged cases are false alarms, so precision is 69.23%. Those figures describe different mistakes and should remain visible together.
What to account for
This supports binary decisions with mutually exclusive, reviewed labels. It does not evaluate open-ended answer quality or prove an agent is safe to deploy. Dataset selection, class balance, annotation quality and threshold changes affect the result. Keep unseen test cases separate from examples used to tune the system.