Key takeaways
- Private AI means the model, and the data it sees, stay inside an environment you control: your cloud tenancy, your data centre, or an air-gapped site.
- It pays off when data cannot leave, when load is steady enough to beat per-call pricing, or when a model version must stay fixed for years.
- It costs you capacity planning, model operations and evaluation work that a hosted service does for you.
- Most organisations end up hybrid: private models for sensitive or high-volume work, hosted models for the rest, behind one routing layer.
- Decide per workload, with an evaluation set, not per company with a policy slide.
What “private” actually means
The word covers three quite different set-ups, and the trade-offs change with each:
- In your cloud tenancy: models served on compute in your own cloud account, so data stays inside your tenancy and your network controls.
- On your own hardware: GPUs in your data centre, under your physical and operational control.
- Air-gapped: no external network at all, with models and updates brought in by controlled transfer.
Four reasons it pays off
- Data that cannot leave. Patient records, regulated financial data, classified or export-controlled material, or contracts that forbid processing by a third party.
- Steady, high volume. When the same workload runs all day, owned capacity can cost less than paying per call — but only while it stays busy.
- A version that must not move. A model that changes underneath a validated process can break it. Owning the weights lets you pin a version until you decide to move.
- Latency and locality. Inference next to the data, or at a site with poor connectivity.
What it costs you
- Capacity planning: hardware sized for the peak, or a queue your users will notice.
- Model operations: serving, scaling, monitoring, patching and upgrading, on your schedule and your staff.
- Evaluation: proving that a smaller open-weight model is good enough for the task, and noticing when it stops being so.
- Securing the stack itself: the model server becomes one more system to harden, patch and audit.
A simple test, per workload
| Question | Points toward private | Points toward hosted |
|---|---|---|
| Can this data leave our boundary? | No, or only with heavy contractual work | Yes, under a data processing agreement |
| How steady is the load? | All day and predictable | Spiky or occasional |
| How capable must the model be? | A smaller model meets the bar on our evaluation set | Only the largest models do |
| How long must behaviour stay fixed? | Years, under validation | Weeks or months is fine |
Hybrid is the usual answer
In practice a routing layer sends each request to the right place: sensitive or high-volume tasks to a private model, rare or hardest tasks to a hosted one under a provider policy. The routing rules are where governance lives — which classes of data may go where, which providers are allowed — and every routing decision should be logged.
Start with an evaluation set
Collect real examples
A few hundred real inputs for the task, each with what a good answer looks like.
Run the candidates
Hosted and private models against the same set, scored the same way.
Compare at your volume
Quality, latency and cost per thousand requests at the load you actually expect.
Decide, then re-run
Choose per workload, and re-run the set whenever a model changes.