Private AI teams often say a route is local because the model runs on their own GPU.

That is only part of the story.

The route is also shaped by where weights are cached, whether those files survive pod turnover, whether startup depends on a mounted disk, whether the node root disk is carrying the load, and whether offline recovery assumes artifacts that are not actually durable.

That is why private AI needs a storage contract.

A storage contract is the release record that binds cache location, mounted volume policy, persistence class, capacity budget, rebuild assumptions, and rollback-ready artifact availability into one visible operating boundary.

This is not the same thing as a model-pull contract. A model-pull contract answers how artifacts are acquired. A storage contract answers where they live once acquired, what survives a restart, and what the serving route depends on under real load.

Why this matters now

The current primary-source docs already make storage a first-class concern.

Hugging Face's current environment-variable docs say HF_HOME controls where huggingface_hub stores local data, including the token and cache, while HF_HUB_CACHE controls where repositories from the Hub are cached locally. Transformers' current installation docs say from_pretrained() downloads and locally caches a model, checks whether the cached model is up to date, and in offline or firewalled environments requires downloaded and cached files ahead of time.

TGI's current launcher docs expose both --huggingface-hub-cache and --weights-cache-override, explicitly saying the cache location can be overridden if an operator wants to provide a mounted disk. vLLM's current engine-arguments docs expose --download-dir as the directory used to download and load weights, defaulting to the usual Hugging Face cache path.

Ollama's current FAQ documents default model directories on macOS, Linux, and Windows, then says teams can move that location with OLLAMA_MODELS. Kubernetes' current storage docs say emptyDir is empty at Pod startup and uses local root-disk or RAM-backed storage, while PersistentVolumes have a lifecycle independent of any individual Pod. Its resource-management docs also treat ephemeral-storage as a built-in resource type.

Put together, those docs describe a route dependency graph. Storage is not just a box underneath the model. It is part of what makes the route reproducible, warm, and recoverable.

The docs already show why storage is part of route identity

1. Cache location is an operating choice, not a convenience setting

Hugging Face's current docs make the local cache path explicit. HF_HOME controls where local data lives. HF_HUB_CACHE controls where Hub repositories are cached. Transformers' current docs then explain that cached models are reused when up to date and must already be present for offline or firewalled use.

That means the route depends on a storage decision before the first production request arrives. If two environments use different cache roots, different cleanup policies, or different shared-disk assumptions, they are not operationally identical just because they point at the same model name.

2. Mounted-disk policy is already exposed at the serving layer

TGI's current launcher docs do not hide storage behind infrastructure abstraction. They expose cache override flags and explicitly say operators can point the route at a mounted disk. vLLM likewise exposes --download-dir as a serving-time decision, not a hidden implementation detail.

That is a clear signal from the tools themselves. Storage layout is part of the served product.

If staging uses a fast persistent volume while production uses a thin root disk or a different shared mount, the route may keep the same model alias while behaving differently under startup, reload, or rollback pressure.

Abstract evidence pipeline showing private AI model caches, mounted volumes, persistence classes, and rollback evidence
Reliable private AI depends on more than weights and GPUs. It depends on a visible record of cache location, mounted storage, persistence behavior, and what survives a restart or rollback.

3. Local defaults do not remove governance

Ollama's current FAQ is useful because it makes the hidden dependency concrete. It documents default model directories, then says teams can move them with OLLAMA_MODELS.

That means “it runs locally” still leaves important questions unanswered. Which directory? On which disk? Under which user? With what backup, cleanup, or permission policy? A local default path is still a production dependency if the route needs it to start.

4. Kubernetes distinguishes storage that dies with the Pod from storage that does not

Kubernetes' current ephemeral-volume docs say emptyDir is empty at Pod startup, uses local storage from the kubelet base directory or RAM, and follows the Pod's lifetime. Its PersistentVolume docs say PVs have a lifecycle independent of any individual Pod that uses them.

That is not infrastructure trivia. It is a route-level difference.

A model cache on emptyDir is a different recovery posture from a cache on a PersistentVolume. One may disappear during normal pod turnover. The other may survive long enough to make warm restarts and rollback drills realistic.

5. Storage headroom belongs in release planning

Kubernetes' resource-management docs treat ephemeral-storage as a built-in resource type. That matters because it puts local disk headroom into the same class of scheduling and limit-setting concerns that operators already manage for CPU and memory.

If a route depends on large local caches, temporary extraction files, or node-local media preprocessing, storage cannot stay an implicit assumption. It needs a named budget.

6. Filesystem characteristics can change load behavior

vLLM's current engine-arguments docs do more than expose --download-dir. They also document safetensors load strategies and note filesystem-sensitive behavior, including NFS detection.

That is a reminder that storage is not only about persistence. It can also shape startup behavior and how quickly a route becomes useful after the process comes back.

What breaks when the storage contract is missing

1. The same endpoint has different warm-start behavior across environments

One environment starts fast because the artifacts sit on a persistent mounted disk. Another starts cold because the cache vanished with the pod or landed on a different node. The endpoint name matches. The route behavior does not.

2. Rollback depends on cache luck

The team knows which model revision and runtime image were good last week, but it cannot prove those artifacts still exist on the storage path the route actually uses today. Recovery becomes re-download plus hope instead of rollback plus evidence.

3. Offline claims quietly depend on online rescue

Transformers' docs are direct about offline use requiring cached files ahead of time. If the organization has not preserved the right cache on the right disk, “offline-capable” may really mean “online until the next restart.”

4. Local demos hide shared-system risk

A developer laptop may carry a warm cache in one known directory. Production may rely on a different user account, a different mount, a different cleanup job, or a shared root disk. The demo proved the model. It did not prove the storage contract.

5. Capacity surprises show up as route instability

Big checkpoints, multimodal artifacts, temporary files, and duplicated caches consume disk faster than many teams expect. Without a storage budget, the route looks flaky even though the real failure is basic capacity planning.

6. Incident review cannot explain what survived

After a restart or node replacement, operators need to know whether the route reused a verified cache, redownloaded artifacts, loaded from a mounted volume, or came back missing critical files. Without a storage contract, that timeline stays fuzzy.

What a storage contract should include

1. Artifact locations

Record the exact cache roots, model directories, mounted paths, and any runtime-specific overrides such as HF_HOME, HF_HUB_CACHE, OLLAMA_MODELS, --huggingface-hub-cache, --weights-cache-override, or --download-dir.

2. Persistence class

State whether the route depends on root-disk cache, emptyDir, network-attached storage, node-local persistent storage, or a PersistentVolume with a clear lifecycle. Name what survives a pod restart, node replacement, and rollback.

3. Warm-start expectation

Define how the route is expected to behave when the cache is already present versus absent. Do not let “cold but healthy” and “broken” collapse into the same operational state.

4. Capacity budget

Keep a visible disk budget for base weights, tokenizer files, adapter layers, media artifacts, temporary files, and any duplicate cache surfaces. Include the point at which the route should stop accepting new work or escalate.

5. Offline and rebuild evidence

Prove that the route can start from the intended storage path without fresh internet access when that matters. If the organization claims local recovery readiness, the storage contract should show the artifacts that make that claim true.

6. Cleanup and retention policy

Define who may prune caches, what can be evicted, how long prior-good artifacts are kept, and which paths are protected because rollback depends on them.

7. Permission and ownership rules

Local paths still need governance. Record which service account owns the artifacts, which processes can write them, and how mounted storage is permissioned across environments.

8. Route validation on the real storage path

Run validation on the same storage topology production will use. A route tested against one mounted disk and deployed on another is not fully validated.

Questions buyers and operators should ask

  1. Where do model artifacts and caches live for this route right now?
  2. Does that storage survive pod restarts, node replacement, and rollback?
  3. What exact flags or environment variables override the default cache location?
  4. Can the team prove offline or firewalled startup from the intended storage path?
  5. What disk budget exists for weights, adapters, temporary files, and multimodal artifacts?
  6. Which cleanup process is allowed to remove old artifacts, and what rollback evidence protects prior-good versions?

If those answers are vague, the organization may have a model strategy, but it does not yet have a dependable route.

The commercial takeaway

Private AI maturity is moving beyond model selection and API compatibility.

The next reliability step is admitting that storage policy is part of the production contract. Cache roots, mounted disks, persistence class, and disk budgets all shape whether a local inference route is truly warm, recoverable, and repeatable.

That is how a private AI deployment stops being a machine that once worked and becomes a route the organization can trust to come back the same way tomorrow.

If the organization cannot name where the route's artifacts live, what survives a restart, and which storage path rollback depends on, it does not fully control the route.

Related reads

Sources consulted