Enterprise AI teams now use the phrase private AI for at least four different products.
Sometimes they mean a hosted API that does not train on customer data by default. Sometimes they mean a zero-retention exception. Sometimes they mean traffic stays on private networking into a managed provider service. Sometimes they mean the model actually runs on infrastructure they own.
Those are not interchangeable controls.
That is why private AI needs a residency contract.
A residency contract defines where prompts, files, outputs, logs, caches, model weights, and tool traffic are allowed to live. It also defines which parts of the workflow remain on provider infrastructure, which stay on customer-controlled systems, and what evidence proves the boundary really exists.
Why this matters now
The market language has gotten looser just as procurement questions have gotten sharper.
Buyers ask for private AI and often receive a slide deck that mixes together data-retention policy, private-network transport, provider-managed model hosting, and self-hosted local inference. Each control matters. None of them answers the full question by itself.
If a team wants to keep prompts off the public internet, that is a network-path requirement. If it wants the model provider to avoid training on customer content, that is a data-use requirement. If it wants logs to disappear quickly, that is a retention requirement. If it wants inference to happen on hardware it controls, that is an execution-location requirement.
Once those requirements get collapsed into one adjective, architecture and security reviews start talking past each other.
The docs already separate different kinds of “private”
1. Data-use protection is not the same thing as self-hosting
OpenAI’s current Your data guide says that, as of March 1, 2023, data sent to the OpenAI API is not used to train or improve OpenAI models unless the customer explicitly opts in.
That is a real control. It matters.
But the same guide also says abuse-monitoring logs are generated by default and retained for up to 30 days, unless a customer is approved for Zero Data Retention or Modified Abuse Monitoring.
So a hosted API can be privacy-conscious without being the same thing as locally hosted inference. The model may not train on your data by default, yet prompts and responses can still sit inside a provider-controlled monitoring path unless you negotiate a different mode.
2. Private networking is a different control again
OpenAI’s current Private Link guide says Azure workloads can reach regional OpenAI API endpoints through Azure Private Link instead of connecting directly to public API endpoints. The guide says to use it when an organization has strict requirements to keep traffic on Azure private networking.
That is a transport-path control, not proof that the model now runs on your hardware.
A route can stay on private networking and still terminate inside a provider-managed service boundary. That may be exactly what the organization wants. It is still different from self-hosting, and it should be described differently in security reviews and buying decisions.
3. Managed foundation-model services define another boundary
AWS’s current Amazon Bedrock data-protection page says model providers do not have access to Amazon Bedrock logs or to customer prompts and completions. The same page points customers to Amazon VPC and AWS PrivateLink.
That is another legitimate pattern: managed inference inside a hyperscaler control plane with documented separation from model-provider access.
Again, useful. Again, not the same thing as a model running on the customer’s own box.
Residency questions should distinguish at least three layers here: the model provider, the cloud provider running the service boundary, and the customer environment connecting into it.
4. Local execution is its own operating model
Hugging Face’s current Transformers pipeline tutorial says Pipeline runs on a CPU by default, that device=0 runs on the first GPU, and that device_map="auto" lets Accelerate choose devices automatically.
That is a materially different residency posture from provider-managed inference.
When the model runs on infrastructure you control, the prompt path, output storage, model weights, and cache behavior can all stay inside your own environment if you design the rest of the workflow to match. That does not happen automatically just because a vendor says the experience is private. It happens because execution location is explicit.
5. Endpoint-local traces can still exist even when retention is negotiated
Anthropic’s current Claude Code data-usage page is a good reminder that residency does not stop at the model endpoint.
That page says commercial users have a standard 30-day retention period, that zero data retention is available to qualified accounts for Claude Code on Claude for Enterprise, and that Claude Code clients store session transcripts locally in plaintext under ~/.claude/projects/ for 30 days by default to enable session resumption.
The lesson is broader than one product.
Even if a vendor gives you a stronger cloud-retention option, you still need to account for local caches, operator workstations, browser sessions, artifact downloads, and other endpoint traces. A residency contract that ignores those paths is incomplete.
What breaks when residency stays vague
1. Procurement approves the wrong control
A buyer may request “private AI” expecting customer-controlled execution, then receive a provider-hosted API with favorable training terms and assume the requirement is satisfied.
2. Security review checks the wrong boundary
Network teams may validate private transport while application teams assume logs and outputs never touch provider systems. Those are different reviews.
3. Tool egress reopens the boundary
Even if inference stays inside an approved environment, attached tools, browser actions, file fetches, and web retrieval can push workflow data somewhere else. Residency needs to cover the whole route, not just the model call.
4. Endpoint traces get forgotten
Local transcripts, downloaded outputs, operator screenshots, and cached attachments often sit outside the cloud-control story. That can matter just as much as provider retention.
5. Audit evidence turns into marketing language
If nobody wrote down where each class of data is allowed to live, teams end up with screenshots of vendor settings instead of a durable operating record.
What a residency contract should include
A useful residency contract is not abstract. It names exact locations and exact exceptions.
1. Execution location
State where inference runs: provider API, provider private-network endpoint, managed cloud service, customer VPC, on-prem cluster, workstation GPU, or another approved environment.
2. Network path
Document whether traffic goes over public internet, private connectivity such as Azure Private Link or AWS PrivateLink, VPN, or a dedicated internal network.
3. Retention policy
Record what the provider retains by default, what is excluded, which exception tiers are enabled, and how long any monitoring or application-state data persists.
4. Local artifact policy
Define what is allowed to land on operator laptops, developer workstations, shared browsers, notebooks, temp directories, and exported deliverables.
5. Tool and retrieval egress
List which tools may leave the core boundary, what they can transmit, which domains or systems are allowed, and which actions require approval.
6. Logging and telemetry destination
Specify where prompts, outputs, traces, costs, and error records are stored, who can read them, and how long they persist.
7. Model and weight location
Separate the question of where the request runs from where the weights live. Managed services, cached model artifacts, and self-hosted checkpoints are different answers.
8. Evidence of control
Keep proof that the stated mode is active: contract tier, endpoint configuration, approved projects, network design, retention settings, and a current test that shows the route behaves the way the contract says it should.
Questions buyers and operators should ask
- Does “private AI” here mean no training on our data, shorter retention, private networking, customer-controlled execution, or some combination?
- Where do prompts, attachments, outputs, and logs physically and logically reside during the workflow?
- What is retained by default, for how long, and what must be approved to change that default?
- Which parts of the route stay on provider infrastructure even when traffic uses private networking?
- Do operator devices keep local transcripts, cached files, or downloaded artifacts by default?
- Which tools, connectors, or retrieval steps can send data outside the core model boundary?
- What evidence can the vendor or internal platform team provide that the active route matches the promised residency mode?
If those answers are fuzzy, the organization does not yet have a private AI posture. It has a marketing phrase.
The commercial takeaway
Private AI is becoming a real buying category. That is good news. It means more teams are taking data control and deployment shape seriously.
The problem is that the category is already being stretched across multiple, very different operating models.
Current primary-source docs show the difference clearly. A hosted API with default 30-day abuse logs is one product. A qualified zero-retention mode is another. A provider endpoint reached through private networking is another. A model running on infrastructure you control is another again. Endpoint-local transcripts create yet another layer that must be governed.
The teams that buy and deploy well will stop asking, “Is it private?” and start asking, “What exact residency contract does this route satisfy?”
That is the question that keeps architecture, security, procurement, and operations aligned before the first production workload ships.
Related reads
- Private AI Needs a GPU Topology Contract: Why the Same Model Changes Across Nodes
- Private AI Needs a KV Cache Contract: Why Prefix Reuse, KV Dtype, and Eviction Change the Route
- Security Boundaries
- AI Workflow Services