Private AI teams still talk about model choice like it settles the product.

It does not.

Once a model moves into a real route, users stop interacting with raw weights. They interact with a serving stack. That stack decides how messages are formatted, which endpoints exist, which request fields are honored, whether tools work, how structured outputs are constrained, and what happens when the model is cold or unloaded.

That is why private AI needs a runtime contract.

A runtime contract defines the exact serving path for a route and the behaviors that come with it. Without one, teams think they are testing “the model” when they are really testing a temporary combination of template, parser, runtime version, backend flags, and warm-state policy.

Why this matters now

The open-weight stack is getting easier to deploy and more fragmented at the same time.

Today the same enterprise team might evaluate one model through Hugging Face Transformers for local prototyping, vLLM for high-throughput serving, TGI for OpenAI-style integration, and Ollama for local developer workflows. All of those routes can be useful. None of them are identical products just because the model family name matches.

Current primary-source docs already show the pattern.

The docs already show that runtime choice is product behavior

1. Message arrays do not mean prompt equivalence

Hugging Face’s current Transformers chat templating docs make the core point clearly: chat models still continue sequences of tokens, and different models may use different formats or control tokens, even if they were fine-tuned from the same base model.

That matters more than many teams admit.

If the same role-and-content messages are wrapped into different control-token layouts, the route is not equivalent at prompt time. The docs even show two models fine-tuned from the same Mistral-7B base model using different chat formats. So the contract cannot stop at “we send OpenAI-style messages.” It also has to specify which template turns those messages into model input.

2. OpenAI-compatible endpoints still carry runtime-specific rules

vLLM’s current online serving docs say its HTTP server supports a set of OpenAI APIs, including completions and chat completions. The same page also says chat completions are only applicable to text-generation models with a chat template. It notes that the user parameter is ignored, and that parallel_tool_calls changes whether the server returns zero-or-one tool call or allows more than one, while also warning that multi-tool behavior is model-dependent.

That is not just plumbing. That is route behavior.

A team can point two systems at “the same model” and still get different runtime semantics depending on which fields matter, which are ignored, and which template is active for chat turns.

3. Tool calling can require parser and template choices, not just a capable model

vLLM’s current tool-calling docs make this even more explicit. The quickstart for Llama 3.1 tool calling uses --enable-auto-tool-choice, a model-specific --tool-call-parser, and a specific tool-calling --chat-template. The same page says named function calling should work with most structured-output backends and that you are guaranteed a validly parsable function call, not a high-quality one.

That line is useful because it separates two things enterprises often blur together.

The runtime can make output parseable. That does not guarantee the underlying tool choice, argument quality, or business correctness are good enough for release. A runtime contract needs to pin both the parser path and the validation path after the parser succeeds.

Abstract evidence pipeline showing private AI route identity across templates, parsers, schema handling, and runtime policy
Private AI route identity lives in more than the model file. Templates, parsers, structured-output backends, and warm-state policy all shape what the route can actually do.

4. Feature support arrives as runtime release policy, not as a timeless model property

Hugging Face’s current TGI docs say the Messages API is fully compatible with the OpenAI Chat Completion API starting from version 1.4.0. A separate TGI guidance page says JSON and regex grammars plus tools and functions are available starting from version 1.4.3, and that these features are compatible with OpenAI client libraries.

That means capability is tied to the serving release, not only to the model family.

If one environment is on a newer TGI version and another is not, “same model” can still mean different tool, grammar, and integration behavior. The runtime version belongs in the release identity.

5. Local developer routes expose different controls than production serving routes

Ollama’s current OpenAI compatibility docs list supported features for /v1/chat/completions including chat completions, streaming, JSON mode, reproducible outputs, vision, tools, reasoning controls for thinking models, and specific request fields. Its current chat API docs also expose keep_alive for model keep-alive duration, with examples like 5m or 0 to unload immediately, and return metrics such as load_duration.

So even a local route that looks OpenAI-shaped can carry runtime behaviors around feature scope, residency, and load time that need to be written down.

If one route keeps a model warm and another unloads it aggressively, the same model name can produce a different first-request experience, a different concurrency profile, and a different operator story.

What breaks when runtime is left implicit

1. Teams test a demo path and buy a different production path

A model that looks clean in local prototyping may fail once the production runtime changes the template, parser, schema backend, or supported request fields.

2. Procurement discussions over-focus on weights and under-specify execution

“Which model are we using?” sounds precise but often is not. The more useful question is “which model, on which runtime, with which template, parser, and release settings?”

3. Cross-environment drift becomes invisible until workflows break

Dev may run Ollama, staging may run TGI, and production may run vLLM. If those differences are undocumented, failures show up as mysterious tool bugs or schema regressions instead of as contract drift.

4. Valid JSON gets confused with reliable workflow behavior

A runtime can coerce structure without guaranteeing good choices. Parseability is necessary. It is not enough.

5. Performance discussions lose the route context

Warm-state policy, load duration, and tool parser overhead all change the real user experience even when benchmarked model quality stays fixed.

What a runtime contract should include

1. Exact runtime identity

Name the serving stack, runtime version, exposed endpoint family, and any model alias the route uses.

2. Prompt construction rules

Document the chat template, tokenizer assumptions, and whether the route depends on model-specific control tokens or tool templates.

3. Tool-calling path

State whether tool use is named, required, automatic, or disabled; which parser is active; and what validation happens after the tool call is parsed.

4. Structured-output behavior

Define which schema or grammar features are supported, which backend is active, and how the route fails when constrained output cannot be satisfied.

5. Request-field semantics

List the fields the route honors, ignores, or only partially supports. “Accepted by the client” is not the same thing as “implemented by the server.”

6. Warm-state policy

Record keep-alive, sleep, unload, and load-duration expectations so first-request behavior is part of the product promise.

7. Validation pack

Test the exact runtime with a short real workflow: multi-turn chat, one structured tool call, one constrained-output request, and one request after idle time.

Questions buyers and operators should ask

  1. What exact runtime and version are in each environment?
  2. What chat template turns message arrays into actual model input?
  3. Which request fields are ignored, downgraded, or runtime-specific?
  4. Does tool calling require a parser, template, or model-specific flag?
  5. Which structured-output backend or grammar path is active on this route?
  6. Can the team prove the same workflow on the exact runtime that will ship?
  7. What happens to behavior after a runtime upgrade, not just after a model upgrade?
  8. How does the route behave on the first request after unload or idle time?

The commercial takeaway

Private AI is entering the phase where runtime choice is no longer an implementation detail for specialists. It is part of the commercial product.

Current docs already show why. Message arrays depend on templates. OpenAI-compatible endpoints still carry runtime-specific semantics. Tool calling can require parser and template decisions. Grammar and tool support can arrive in runtime releases. Local serving paths expose their own feature and warm-state controls.

So the real release question is not just “Which model did we pick?”

It is:

Which exact runtime contract are we promising users, operators, and downstream systems?

The teams that answer that question clearly will spend less time debugging “mysterious model drift” and more time shipping private AI routes that behave the same way in production as they did in evaluation.

Related reads

Sources consulted