Private AI teams often document a route with the model name, endpoint shape, and hardware tier.
That still leaves out one of the most important operating decisions: where repeated requests land.
If a route depends on warm prompt prefixes, short-lived cache reuse, or queue-local state, then two requests that look identical at the API edge can behave differently just because they hit different replicas.
That is why private AI needs a session-affinity contract.
A session-affinity contract defines whether a route is supposed to stay sticky, what kind of stickiness it needs, how long that stickiness should last, what boundaries separate tenants or workflow classes, and what should happen when the system cannot preserve locality.
This is not the same thing as a KV cache contract. A cache contract defines what a replica is allowed to keep hot. A session-affinity contract defines whether repeat work is expected to reach that same hot replica in the first place.
Why this matters now
Prompt and prefix caching are no longer niche tricks.
Current OpenAI docs describe routing requests to a machine based on a hash of the prompt prefix and allow a prompt_cache_key to influence routing and improve hit rates. Anthropic's current prompt caching docs define explicit 5-minute and 1-hour TTL options around cache_control. vLLM documents automatic prefix caching that reuses prior KV work for shared prefixes.
At the same time, self-hosted AI is increasingly deployed behind Kubernetes Services, reverse proxies, and multi-instance serving layers. That means the routing policy at the infrastructure edge can now decide whether the route behaves like a warm path or a cold path.
If that policy is undocumented, teams start treating replica placement as random infrastructure noise when it is actually a material part of user experience, cost behavior, and reliability.
The docs already show why session affinity belongs in the route contract
1. OpenAI treats cache routing as an explicit part of prompt caching behavior
OpenAI's current prompt caching guide says caching is enabled automatically for prompts that are 1024 tokens or longer.
The same guide says requests are routed to a machine based on a hash of the initial prefix and that prompt_cache_key is combined with that prefix hash to influence routing and improve cache hit rates when many requests share long prefixes.
That is a direct signal that request placement matters. A route with long reusable prefixes is not just a model-plus-endpoint problem. It is also a routing problem.
Even if you self-host, buyers and operators are now being trained by platform docs to expect cache-aware placement, not generic round-robin behavior.
2. Anthropic makes cache lifetime and prefix scope explicit
Anthropic's current prompt caching docs describe 5-minute and optional 1-hour TTLs and frame prompt caching around explicit cache_control breakpoints.
The docs also make clear that prompt structure matters, including system content, tools, and messages up to the cache boundary.
That means warm-path behavior has a time window and a scope boundary.
If an enterprise route expects repeated use of the same long instructions, tools, or evidence frame within that window, then the organization should document whether those repeated requests need sticky placement, cache-key grouping, or deliberate acceptance of cache misses.
3. vLLM exposes both prefix reuse and multi-instance serving choices
vLLM's current Automatic Prefix Caching example says the engine can reuse cached KV pairs from previous prompts if a new query shares the same prefix, reducing redundant computation and improving inference speed.
Its current engine-arguments docs separately expose --enable-prefix-caching and multiple prefix-caching hash algorithms, making locality-sensitive behavior a first-class serving choice.
The same engine-arguments page also says --data-parallel-rank enables external load balancer mode for MoE data-parallel deployments.
That matters because it shows self-hosted AI routes are now explicitly designed to sit behind external routing layers. Once that happens, session policy is no longer implied by the runtime. It has to be chosen by the operator.
4. Kubernetes defaults prove generic traffic distribution is not the same thing as AI-aware locality
Kubernetes' current Service docs say the default .spec.sessionAffinity is None.
The same docs say teams can set .spec.sessionAffinity to ClientIP and configure .spec.sessionAffinityConfig.clientIP.timeoutSeconds, with a documented default of 10800 seconds, or 3 hours, when that stickiness mode is used.
That is useful infrastructure behavior, but it is not a full AI routing policy by itself.
Some AI routes may need tenant-level or workflow-level grouping instead of raw client IP stickiness. Others may need no affinity at all because they are designed for cold fairness, not warm reuse. The point is not that every route should become sticky. The point is that the route should say which behavior it expects.
What breaks when session affinity stays implicit
1. Warm-path benchmarks stop matching production
A route may look fast in demos because repeated requests hit the same hot worker. Generic production balancing can erase those gains without changing the model or prompt at all.
2. Queue classes interfere with each other
If interactive chat, batch review, and long-running agent traffic all land on the same warmed replicas without policy, the very locality that helps one workload can become hidden contention for another.
3. Teams blame the wrong layer
Operators often blame the model when latency or cache-hit behavior swings. In reality, the change may come from request placement, replica churn, or a service-level routing policy that quietly spread related work across workers.
4. Tenant boundaries get hand-waved
If the route relies on sticky reuse but does not define whether grouping is per tenant, per workflow, or per operator session, then cache locality and security expectations can drift apart.
5. Rollbacks become incomplete
Rolling back the checkpoint is not enough if the route also changed from sticky placement to generic spread, or if stickiness timeout changed while the endpoint name stayed the same.
What a session-affinity contract should include
A useful session-affinity contract is not complicated. It makes route placement inspectable.
1. Affinity mode
State whether the route is affinity-sensitive, affinity-optional, or deliberately affinity-free. Not every route should be sticky, but every route should have an explicit expectation.
2. Grouping key
Document what the locality is tied to: tenant, workflow, prompt prefix, operator session, request class, or a provider-specific cache key such as prompt_cache_key.
3. Stickiness window
Record how long locality should last. That window should line up with real cache TTLs, workload rhythm, and user experience, not just generic infrastructure defaults.
4. Isolation boundary
Say which traffic classes may share warm replicas and which must stay isolated. The route should define whether chat, batch, and agent runs can benefit from the same locality pool.
5. Cold-path fallback
Define what happens when stickiness cannot be preserved. Does the route accept a cold miss, downgrade concurrency, reroute to a slower lane, or warn the operator that warm-path assumptions no longer hold?
6. Validation evidence
Keep performance and correctness evidence for both sticky and non-sticky conditions. A route qualified only under perfect locality is not fully qualified.
7. Release linkage
Bind affinity policy to the release record alongside cache policy, scheduler policy, and topology assumptions so placement changes are reviewed like any other route change.
Questions buyers and operators should ask
- Does this route expect repeated requests to land on the same worker, or is it designed to behave the same across replicas?
- What exact key or boundary controls locality: client IP, tenant, workflow, prompt prefix, or something else?
- How long is warm-path behavior expected to persist before a request should be treated as cold?
- What happens to latency, cache-hit rate, and output stability when related requests spread across workers?
- Are interactive, batch, and agent workloads sharing the same warmed replica pool?
- Does the release record show routing policy changes alongside model and cache changes?
- What evidence proves the route is still acceptable when affinity is lost during failover, autoscaling, or deploy churn?
If those answers are fuzzy, the system may still be useful for experiments. It is not yet a well-specified private AI route.
The commercial takeaway
Private AI is moving closer to the application edge.
That does not just mean model choice and GPU sizing. It also means the infrastructure layer is now visibly shaping the route that users actually experience.
Current primary-source docs already show that cache reuse depends on placement, TTL, and routing choices. The teams that get dependable value from local inference will not treat session affinity as an accidental byproduct of the load balancer. They will define it as part of the route contract.
That is how a private AI endpoint stops being a generic pool of replicas and starts behaving like a reliable product.
Related reads
- Private AI Needs a KV Cache Contract: Why Prefix Reuse, KV Dtype, and Eviction Change the Route
- Prompt Caching Is a Workflow Policy: The Enterprise AI Guide to Cache Boundaries
- Private AI Needs a Scheduler Contract: Why Batching Policy Is Part of the Product
- AI Workflow Services