On this page12 sections
A spam filter makes its decision before you notice the message. A transcription app works through a recording. In a chat window, you watch an answer arrive piece by piece. All three use trained models, but they ask you to wait for different kinds of work.
Using a trained model to produce an output is inference. A message becomes a spam label, audio becomes a transcript, a photograph yields an identified object and a prompt produces an answer. Start with the result you need before choosing a model or buying a machine to run it.
That distinction becomes practical when the quick demo slows as colleagues join, or a downloaded model refuses to fit on your laptop. Compare complete setups by whether they finish the actual job within your data, hardware, cost and time constraints.
Follow one request: what should return, which program and machine will produce it, and whether the answer earns the wait. The local exercise later makes that trip visible without asking you to build an entire AI platform first.
Use what the model learned, or change what it learned?
Training changes a model’s learned numerical parameters, usually called weights, using examples and an objective. Inference applies those values to predict or generate. One changes the model; the other uses it. The Google machine-learning glossary describes this distinction.
For a language model, the immediate prediction usually concerns the next token, a piece of text. The model assigns probabilities to possible continuations, and a decoding procedure selects one. That new text joins the context for the next prediction. Repeating the process builds an answer. Optimized serving can accelerate the basic dependency without turning it into retrieval of a prewritten paragraph.
Paste a runbook into the prompt and the model has material for this request; its weights have not thereby changed. Fine-tuning is a separate training operation. Whether a provider retains requests or later trains on them is another policy question. An answer using the runbook alone establishes neither of those other events.
This gives a practical starting point: supply a document, code snippet or current sanitized procedure and ask for an explanation before beginning a training project. Then remove the procedure and see what the model invents without it. The contrast can reveal more about the setup than another smooth demonstration with everything supplied.
Give generation the work that needs interpretation
Language models are useful around messy language: an unfamiliar function to explain, an email to draft from notes, documents to summarize or a support request to categorize. Several phrasings may serve the job well. The model can make a useful first pass without needing one exact sentence to be the answer.
A column of numbers has a different requirement. A calculator or spreadsheet formula can add it directly. A parser can extract a known field from well-formed JSON. A model may help write those tools, but asking it to generate an answer on every execution adds cost and uncertainty where a reliable calculation already exists.
Decide whether the application needs a label or a paragraph
Show three systems the same photograph. One says “cat,” another marks the cat’s outline and a third describes the scene. All performed inference. Only one may have returned the shape your application can use.
Classification and computer vision can produce categories, scores, detected objects or image regions, for example, sorting photos or locating a defective part. A chat model’s description differs from a detector’s coordinates. If the next program needs a box around each object, a graceful paragraph is an awkward substitute.
Speech recognition turns audio into text. A meeting transcript may then go to a language model for a summary. Keep those stages separate while troubleshooting: the summary can faithfully repeat a name the transcription stage heard incorrectly. The speech-recognition guide explains audio-to-text, and the pipeline guide shows the different input and output shapes across tasks.
For document search, an embedding model returns a vector, a list of numbers, rather than a finished answer. Compare query and document vectors to find related material, then let a generative model explain selected passages. That combination is often called RAG, retrieval-augmented generation. It supplies context without retraining, but does not guarantee the answer follows the retrieved text. The vector itself is not a compressed paragraph waiting to be read back. Ollama provides an embedding endpoint separately from text generation.
Image generation creates new visual content. Diffusion models commonly do this by repeatedly removing noise, rather than following the next-text-token loop discussed below. The Diffusers pipeline explanation shows that progression. Image size, generation steps and pipeline affect the work; a text model’s tokens-per-second rate says little about the wait for an image.
Models accepting several input types, such as images and text, are called multimodal. That does not mean every endpoint accepts every medium or returns every output. Choose whether the task is transcription, classification, retrieval, description or generation first. The rest of this guide focuses on generative text: its prefill/decode and token-rate details do not describe every inference system, though the questions about useful results, data and cost travel more broadly.
Why serving the answer became a substantial job
Inference existed before chatbots. What expanded was the variety of requests one language model could handle and the computation needed to deliver its replies.
The 2017 Attention Is All You Need paper introduced the Transformer architecture, initially demonstrated on machine translation. Its attention-based approach made training more parallelizable than the recurrent approaches it compared with. That training result does not mean a modern chat model generates its whole answer simultaneously; later output still depends on what has already been generated.
In 2020, Language Models are Few-Shot Learners showed GPT-3 handling several tasks from instructions and examples in its input, without task-specific weight updates during those evaluations. The prompt became an important interface: applications could describe different jobs to an existing model instead of training a separate one for each.
The 2023 PagedAttention paper, which introduced vLLM, addressed the growing memory used during generation so a server could handle requests more efficiently. Once several people ask at the same time, memory allocation and sharing matter alongside the model’s ability to answer.
That leads to a familiar operating choice: call a hosted model through an API or run a downloadable one yourself. In either arrangement, the calculations still happen somewhere. The API transfers much of the machinery’s operation to someone else; it does not remove the machinery.
A downloaded model still needs software and a machine
The model supplies learned parameters and architecture. A runtime loads them and performs the calculations. Hardware, a CPU, GPU or other accelerator, does that work. Your application prepares requests and uses the output above these layers. An API is an interface between software components, not a separate kind of model.
“Local” and “API” are therefore not opposites. A laptop model can expose an HTTP API. Ollama provides a local endpoint and hosted models, and its API documentation describes access to cloud models through a local server. Seeing localhost in an address alone does not establish where the request data ultimately goes.
A dropdown can hide these distinctions until a failure makes them matter. The model artifact may have an unsupported format, the runtime may not understand it or the machine may lack working memory. Separating the three gives troubleshooting a place to begin.
If a hosted model explains a question well but its downloaded counterpart will not load, the prompt has not necessarily stopped working. Check format and memory. Conversely, a model can load and stream beautifully while confusing readiness with liveness. More GPU memory will not repair that definition.
A model request does not itself open your files or execute commands. An agent application describes tools, receives a requested call, performs it under its permissions and returns the result. The tool-calling documentation shows that exchange. Running the model locally does not automatically sandbox the application using it.
What happens during the pause before the first words?
The runtime first turns the request into tokens. A token may be a word, fragment, punctuation or another piece; token counts are not interchangeable with word or character counts. Different tokenizers split the same text differently, as the Hugging Face tokenization guide demonstrates.
The input may include instructions, conversation history and retrieved documents as well as the latest question. A short question beneath a large incident bundle is still a large request for the runtime to process.
A typical autoregressive transformer first processes that input during prefill, establishing information needed to begin. During decode, it generates further tokens using the input and output so far. A key-value cache, or KV cache, retains intermediate attention information to reuse earlier work. It saves recomputation; it does not store fact-checked answers. The cache documentation explains this mechanism.
Behind the waiting cursor, the request may queue, weights may load and input must be processed. Then the answer arrives in pieces. A long initial pause followed by quick text points to a different part of that trip from text that begins promptly but arrives very slowly.
Keep three measurements separate when describing that experience:
- When did the answer start? Time to first token, or TTFT, depends on the specified measurement point. The first readable answer text at your client can arrive later than a server's first generated token, especially if the system emits metadata or separate reasoning output first.
- When was the answer usable? An engineer can read an explanation as it streams, but an application may need the entire response before parsing and checking it. Completion latency matters even when the opening sentence appears quickly.
- How much work did the service handle? Throughput might count generated tokens across all requests per second. It describes a different property from the wait experienced by one caller.
The vLLM metrics reference distinguishes those scopes. A server may increase aggregate throughput while making individual callers wait longer, so a busy GPU or healthy token-rate graph cannot settle the user-experience question. The streaming-latency guide measures the client side without treating every network event as a model token.
Ask the same question twice, once for a sentence and once for a long explanation. Input work is similar, but the longer answer needs more generation. Now keep the answer short and add a long document: input work has changed instead. The first word’s arrival cannot explain both parts of the wait.
Choose a destination the data is allowed to reach
If private documents, source code or incident material cannot leave an approved environment, a convenient endpoint outside it is not a candidate. A suitable model inside the boundary may help, provided the application, logs and tools respect that boundary too. Local inference does not automatically make the whole application local.
Where an approved provider may receive the data, hosted inference is often the simpler trial. You can judge answers before choosing accelerators, fitting weights in memory or maintaining serving software. You still own limits, timeouts, output checks and the consequences of provider unavailability. Retention and training terms must fit the use; the word API supplies no privacy policy.
Local execution is especially useful for offline work, control of the exact model artifact or a workload confined to a machine. It also makes loading and memory behavior visible, which is why the exercise uses it. That is a reason to learn locally, not a claim every production assistant belongs on a self-hosted model.
Weights occupy memory, but requests need working space and often a KV cache too. Longer inputs and simultaneous requests can exhaust a configuration that handles one short question. Quantization represents some model values using fewer bits, reducing storage or memory requirements. Quality and speed effects depend on model, format, runtime and hardware; the Hugging Face optimization guide covers these tradeoffs.
The parameter count on a model card is therefore only part of the load. The runtime must support the artifact, and memory must accommodate intended request lengths and users. Try a representative conversation before buying around a number that merely fits the weights.
Local is not automatically faster. Avoiding a remote network trip may still leave a laptop spending much longer generating text. A hosted system may generate quickly after a long queue. Compare the complete request under expected demand, not either setup’s isolated best case.
Rented hardware or a dedicated endpoint offers a middle ground between a laptop and a provider’s shared API. It can supply more control and capacity, while leaving someone responsible for deployment and the bill. “Cloud” names an arrangement and location, not one fixed level of privacy or responsibility.
Pay for an answer you can use, not just one that arrived
Hosted APIs often price input and output tokens separately, with further rules for caches, tools and other services. Self-hosting packages cost as hardware or rental, power, maintenance and capacity paid for even while nobody asks anything.
At invented rates of $1 per million input tokens and $4 per million output tokens, 2,000 input tokens and 500 output tokens cost $0.004: $0.002 each way. Three attempts cost $0.012 before other charges. This is teaching arithmetic, not a vendor quote. Whether the attempts produced useful work is the more demanding calculation.
Imagine a readiness-probe helper that replies quickly but says a failed readiness probe restarts the container. Someone must catch and correct that error, then judge the next reply more cautiously. An excellent token rate can still leave more work than it removed. A slower correct answer may finish the task sooner.
Compare local and hosted setups with the same acceptance test, including retries, rejected requests, unfinished answers and review effort. For self-hosting, divide costs across realistic accepted-task volume rather than assuming the machine stays busy all month. The token-cost guide develops that accounting. The exercise below starts with one answer whose central claim you can independently check.
Four controls, four visible effects
These settings are easier to grasp through what goes wrong: the end of a source never arrives, an answer stops halfway, repeated replies differ or partial text arrives before a complete response. API names vary, but the effects are recognizable.
Context length is how much tokenized material the model can consider within its configured window. Instructions, history, retrieved text and generation need space. An advertised maximum does not establish the runtime’s current setting, available memory or handling of excess input. Before blaming a summary for missing the document’s ending, inspect whether that ending was sent.
Output limits cap generation, controlling work and cost. A cap is not a target; reaching it can leave an answer unfinished. Asking for three sentences is a model instruction, unlike a runtime-enforced token limit. Inspect the finish reason rather than treating every returned response as complete.
Temperature and other sampling settings influence which possible next token is chosen. Lower temperature generally favors higher-probability choices. It does not create a fact-checker or guarantee identical results across backends. Record and hold settings steady in comparisons; Ollama’s parameter reference includes temperature, context and output-token controls.
Streaming returns partial output as it becomes available instead of waiting for completion. It helps a person begin reading, but an application expecting structured data must still assemble and validate the full result. Where supported, a JSON schema constrains shape; see the structured-output documentation. The correctly shaped object can still contain a wrong answer.
These controls shape the request, not its missing evidence. Lower temperature cannot restore an omitted source, and a longer answer can simply prolong a confused definition. Judge the setup by what its answer actually does.
Make one request whose answer you already know how to check
Example status: the original checks covered documentation, shell syntax, argument preservation and JSON. No local or hosted model was run for this guide. No captured answer or measured performance is claimed.
Use Ollama’s local runtime to see the request and returned timing fields. Synthetic input avoids private material. The Kubernetes question has a clear reference; the same request shape can serve a writing or coding question you can judge independently. Prepare the following:
- Ollama installed from its official quickstart, with its local server running.
- A downloaded local text model supported by your machine. Review its license, download size and memory guidance before choosing it. The current quickstart uses
gemma4:e2b; this exercise works with the suitable local model you actually installed. - A POSIX-compatible shell, such as a macOS or Linux terminal, and curl 7.76.0 or newer for
--fail-with-body. The curl manual documents the flags. PowerShell has different quoting rules, so use a compatible shell rather than pasting these commands unchanged.
Open Ollama, choose and download a local model, and retain the default loopback listener. On Linux, the quickstart says to run ollama serve if needed. This local request needs neither cloud sign-in nor an API key. For the documented quickstart example, ollama pull gemma4:e2b downloads the model. The cited documentation lists about 7.2 GB and recommends 8 GB available VRAM, or unified memory on a Mac; longer contexts require more room. Choose a smaller supported local model if necessary. Finishing the download does not prove the model will run comfortably.
Ask the server which models it has before choosing the request name:
curl --fail-with-body --silent --show-error \
http://localhost:11434/api/tags
The model-list response supplies each entry’s name and digest. Replace YOUR_LOCAL_MODEL with the exact name you downloaded. Keep its digest for later comparison: a friendly tag can move to a different artifact.
If Kubernetes is unfamiliar, substitute this prompt: “A fictional library is open Monday through Friday, 09:00 to 17:00, and closed on Saturday. Can I visit at 10:00 on Saturday? Explain using only this note.” The answer is no. Saturday’s closure applies even though 10:00 fits weekday hours. That checks use of a supplied exception; the Kubernetes prompt below checks a technical distinction. Choose the question you can explain yourself.
curl --fail-with-body --silent --show-error \
--max-time 120 \
http://localhost:11434/api/generate \
-H 'Content-Type: application/json' \
-d '{
"model": "YOUR_LOCAL_MODEL",
"prompt": "Explain what a Kubernetes readiness probe checks in three sentences.",
"stream": false
}'
The request uses the documented generate endpoint. stream: false asks for one response, making its fields easier to inspect. The 120-second client deadline bounds the exercise; it promises nothing about generation speed.
Find nonempty text in response. done: true means generation ended, not that the answer is complete or correct. The documentation’s ordinary-stop example uses done_reason: "stop". Investigate another finish reason or a visibly cut-off answer before declaring success.
Read the answer against your chosen question. The fictional library is closed Saturday despite its weekday hours. For Kubernetes, readiness determines whether a container can accept traffic; a failed liveness probe can lead to restart after the failure threshold. Wording may vary, but the consequence must not. The Kubernetes probe documentation gives the technical reference.
A refused connection calls for checking the local server. A missing model calls for comparing its name with the retrieved list. For unexpected slowness, ollama ps shows CPU, GPU or mixed loading, as the Ollama FAQ describes. A timeout leaves the result unresolved; inspect what happened before retrying. Keep this unauthenticated learning exercise on loopback, not exposed to the network.
Move the idea to a hosted endpoint, not just the hostname
A hosted request still contains a model, input and generation settings. Destination, authentication and service contract change. Ollama’s API introduction, for example, documents https://ollama.com/api/chat with bearer authentication, unlike the local no-key request. Chat uses a messages array while the earlier generate example uses prompt. Changing the hostname alone is not a general migration method.
Before a hosted call, choose a model from current provider documentation, confirm price or plan limits and keep its key in the caller’s secret configuration, not source control or browser code. Start with synthetic input. Inspect answer and usage, then test relevant failures such as rate limits and timeouts.
If the local and hosted choices use different models, describe the comparison as two complete setups. That may be exactly the decision you need. It does not isolate the model, quantization, runtime, hardware or network as the reason for a difference.
Read a timing field without making it a benchmark
Ollama returns usage and timing fields. load_duration measures model loading; eval_count counts generated output tokens; eval_duration records generation time in nanoseconds. With positive duration, calculate eval_count / (eval_duration / 1,000,000,000). This is generation rate, not the full wait for a useful result.
A second request may avoid loading, and repeated input may benefit from caching. That can explain a faster repeat without proving a general speedup. When comparing runtimes, keep model, prompt and settings steady where possible and recheck the answer. The inference-compatibility guide explores protocol and output differences once software depends on them.
After one checked request, you know which model answered, where it ran, what the wait contained and whether its main claim held up. Try the same question on an approved hosted setup and the comparison has a purpose: how did cost, delay and the work left to you change?
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- Google machine-learning glossarydevelopers.google.com
- speech-recognition guidehuggingface.co
- pipeline guidehuggingface.co
- RAGhuggingface.co
- embedding endpointdocs.ollama.com
- Diffusers pipeline explanationhuggingface.co
- Attention Is All You Needarxiv.org
- Language Models are Few-Shot Learnersarxiv.org
- PagedAttention paperarxiv.org
- API documentationdocs.ollama.com
- tool-calling documentationdocs.ollama.com
- Hugging Face tokenization guidehuggingface.co
- cache documentationhuggingface.co
- vLLM metrics referencedocs.vllm.ai
- Hugging Face optimization guidehuggingface.co
- parameter referencedocs.ollama.com
- structured-output documentationdocs.ollama.com
- official quickstartdocs.ollama.com
- curl manualcurl.se
- model-list responsedocs.ollama.com
- generate endpointdocs.ollama.com
- Kubernetes probe documentationkubernetes.io
- Ollama FAQdocs.ollama.com
- usage and timing fieldsdocs.ollama.com
