Learning path · 5 steps
Build a small, repeatable ticket-routing evaluation
Finish with queue definitions, a labeled test set, saved model responses and a local evaluator. The six supplied tickets make the mechanics manageable, while separate error counts show why a wrong destination and unnecessary triage should not be collapsed into one impressive-looking score.
In this guide 6 sections
These walkthroughs use the downloadable synthetic support-service project. Work in a disposable copy so you can compare the proposed changes with the original defects. Each tool needs separate authorized access; the starter includes no production credentials or incident data.
What you will build
A routing trial you can rerun after a prompt change
Keep the request definition, raw responses and predictions.json together. The local script checks structure and separates exact matches, wrong specialists, unnecessary triage and missed triage; it does not send tickets anywhere.
Included example files
A small support-service project
Synthetic code, logs, a deliberately stale runbook, and six support tickets. The JavaScript uses Node.js 22+ with no packages or network calls. Product accounts and model access are separate.
- routing/tickets.json
- routing/request.json
- routing/predictions.example.json
- routing/evaluate.mjs
Set the queues before seeing predictions
Read the four criteria in request.json and the authored labels in tickets.json. Access, application and network name the evidence that belongs with each specialist; triage gives incomplete or unmatched reports a destination. Set those meanings before inspecting predictions, so the answer is judged against the task rather than the task being rewritten to fit the answer.
Check before continuingTicket A is labeled access and D triage before a model is called. The labels are not edited to match a later answer.
Verify the local checker with known data
Run the supplied manual example. It deliberately sends A to triage and D to access: one ticket gets unnecessary triage, while another misses triage it needed. Seeing both counts change explains more than a single accuracy score. This file demonstrates the checker; it is not model output.
Terminal commandnode routing/evaluate.mjs routing/predictions.example.jsonCheck before continuingThe result is total 6, exact 4, unnecessaryTriage 1, missedTriage 1 and wrongSpecialist 0.
Collect actual Choice responses
Use the Jev walkthrough’s copyable request body or matching Playground fields. Supply each ticket as state. Save the raw responses, then record the returned queue and actual model ID in predictions.json using the example file’s structure.
Check before continuingEach A–F ticket has one recorded response; manual-demo is not left as the model name for an actual run.
Compare decisions, then refine the definitions
Run the checker against your predictions and read each disagreement beside its ticket and queue definitions. Was information missing, did two criteria overlap, or did the model choose poorly despite a clear report? If you revise the request, keep the first run and save the second separately. Changing expected labels merely to improve the count would erase the problem the exercise is meant to reveal.
Terminal commandnode routing/evaluate.mjs routing/predictions.jsonCheck before continuingEach changed prediction can be traced to a saved request, response and model version.
Design the next evaluation before routing real work
Write evaluation-notes.md describing observed disagreements, remaining ambiguities and the kinds of unseen reports to add next. Compare the effort with a simple rules-based baseline. Leave real ticket execution disconnected until a larger, representative evaluation supports your policy.
Check before continuingThe folder contains request.json, labels, raw responses, predictions.json and evaluation-notes.md. No deployment-accuracy claim is drawn from six examples.
Check the finished project
node routing/evaluate.mjs routing/predictions.jsonThe command should accept one valid result per ticket. Review disagreements individually and preserve the returned model IDs and raw probability distributions. The six-ticket set demonstrates a process, not production accuracy or a safe automation threshold.
Tool guides for this path
An authored walkthrough using the synthetic starter project and linked official tool documentation. Baseline fixture behavior and the local evaluator were checked. The assistants, review service and model calls were not run for this path.
Browse all learning paths