Local LLM Security: The Complete Guide for Self-Hosted AI
Running your own AI model is the single biggest step you can take to keep your prompts and documents in the building. But self-hosting is not security by itself — it changes where the data lives, then hands you the controls. This guide walks through every layer that actually makes a local LLM safe: the threat model in plain English, network isolation, access control, encryption, and logging. It is the same control set we configure on a real TIS install, with links to the deeper page for each.
What "local LLM security" actually means
Local LLM security is the set of controls that keep a model you run yourself — the weights, the prompts, the documents it reads, and the logs it writes — on hardware you own and reachable only by people you choose. A cloud API can't give you that: the moment a prompt leaves your network, where it goes and how long it's kept is the vendor's decision, not yours. Self-hosting puts that decision back in your hands. What it does not do is configure itself. A private box with no network isolation, no access control, and no encryption is still a box on your network that anyone can reach. Security is the work you do after you bring the model home, across five layers: network, identity, encryption, logging, and updates.
Where TIS stops — the honesty boundary
TIS builds and secures private AI infrastructure that supports your compliance — on-premise data, encryption, access control, and audit logging. We are not a law firm or auditor and do not certify, audit, or provide legal advice. For sign-off, consult your own compliance advisor, QSA, CPA, C3PAO, or counsel.
The threat model in plain English
The model itself is an attack surface. The current standard for naming those risks is the OWASP Top 10 for LLM Applications (2025). Here is each risk and what it means on a box you host yourself — including the honest note that going local helps with some risks and not others.
| Code | Risk | What it means on a self-hosted box |
|---|---|---|
| LLM01 | Prompt Injection | Crafted input makes the model follow hidden instructions. Local does NOT fix this — it is a design-level issue. |
| LLM02 | Sensitive Information Disclosure | The model leaks data it shouldn't. On-prem genuinely helps: there is no vendor holding your inputs. |
| LLM03 | Supply Chain | A compromised model, library, or dependency. You vet and pin what you install on an isolated box. |
| LLM04 | Data & Model Poisoning | Tainted training or fine-tuning data skews the model. You control the data that touches it. |
| LLM05 | Improper Output Handling | Treating model output as trusted code or commands. Strict output handling matters whether local or cloud. |
| LLM06 | Excessive Agency | An agent has more permissions than the task needs. Least privilege limits the damage if it is misused. |
| LLM07 | System Prompt Leakage | The system prompt is exposed, revealing secrets baked into it. Keep secrets out of the prompt entirely. |
| LLM08 | Vector & Embedding Weaknesses | The RAG vector store leaks or is tampered with. It holds your data and must be secured and encrypted too. |
| LLM09 | Misinformation | Confident, wrong output. Human review on consequential answers; local does not change accuracy. |
| LLM10 | Unbounded Consumption | Runaway resource use or denial of service. Rate limits and quotas on your own endpoints. |
List per the OWASP Top 10 for LLM Applications, 2025 edition (LLM01–LLM10); System Prompt Leakage and Vector & Embedding Weaknesses are the 2025 additions. Re-verify against the official OWASP publication before relying on it.
Layer 1 — Network isolation
The first job is to make sure the model is reachable only by your people, and that it cannot quietly call out. Isolation ranges from LAN-only inference up to a full air-gap. Run this checklist on any self-hosted build.
LAN-only inference
The model answers only devices on your local network — no public exposure, no outbound API calls leaving the building.
VLAN segmentation
Put the AI server in its own network zone so it can reach only what it needs, not your whole network.
No outbound egress
Block outbound routes by default so the box cannot phone home, leak telemetry, or exfiltrate data.
Firewall / DMZ placement
Place the server behind a firewall with explicit allow rules; never on a flat network with everything else.
Reverse-proxy hardening
If you front the endpoint with a proxy, harden it — TLS, auth, rate limits — and expose nothing else.
Air-gap for the highest tier
For the most sensitive data, remove the internet path entirely. See the air-gapped server guide for the trade-offs.
Deciding how far to isolate is its own decision — our air-gapped AI server guide covers who actually needs no internet and what it costs you in convenience.
Layer 2 — Identity & access
The most common self-hosted mistake is "everyone is admin." Role-based access control fixes that by granting permissions by job, backed by single sign-on and multi-factor authentication on the inference endpoint. Four roles cover most private AI deployments.
| Role | Can do | Cannot do |
|---|---|---|
| Inference consumer | Send prompts and read responses through the sanctioned interface. | Change models, settings, or see other users' data. |
| Prompt engineer | Build and tune prompts, templates, and RAG sources. | Alter system config, access control, or server settings. |
| Model admin | Deploy and update models, manage settings and integrations. | Operate outside an audited, logged change process. |
| Auditor (read-only) | Review access logs and configuration for oversight. | Change anything — strictly read-only by design. |
Role design, SSO, and MFA done properly is a deep topic — see access control for private AI for how we configure it.
Layer 3 — Encryption
Encryption answers three questions: is the data protected on disk, is it protected on the wire, and where do the keys live? At rest, AES-256 (full-disk or per-volume) means a stolen drive is unreadable. In transit, TLS 1.3 protects traffic even inside your own LAN — "internal" is not the same as "safe." The part people skip is key management: encryption is only as strong as the control over where keys are stored, how they rotate, and who holds them. And the easy-to-miss spots — vector stores, caches, swap, and backups — hold your data too and need the same treatment.
We treat AES-256 and TLS 1.3 as the default best practice on a TIS build, not as a claim that any single framework mandates them. Go deeper on at-rest, in-transit, and key custody in our encryption for private AI guide.
Layer 4 — Logging & auditability
If you can't show who accessed what, you can't prove anything to yourself or an auditor. The point of logging on a private AI server is evidence you control: which identity sent which request, when, and what the model did with it. Immutable, tamper-evident logs mean those records can't be silently altered after the fact. You set the retention — some frameworks expect long horizons (HIPAA audit records are commonly kept six years) — and the logs live on hardware you own, not in a vendor's console you can't query.
The honest framing throughout: this is evidence that supports a compliance program. It does not, by itself, certify you against any standard.
Layer 5 — Updates & supply chain on an isolated box
An isolated box still has to stay current, and that is where supply chain risk (LLM03) shows up. The answer is not to open an outbound path "just for updates." It is signed updates applied on your schedule: model and OS packages are verified against a known signature, carried in via controlled media for an air-gapped site, and applied through a logged change process. You vet and pin what you install rather than pulling whatever a public registry serves that day. Nothing auto-phones-home.
This is the discipline that keeps the strongest isolation tier usable in practice — see the air-gapped AI server guide for the full offline-update procedure.
Self-hosted vs. cloud: where each control is stronger
Self-hosting is not a magic shield. It is materially stronger on data control and materially equal on the design-level LLM risks. The honest breakdown:
| Control | Self-hosted (local) | Cloud API |
|---|---|---|
| Data residency | On a box in your building; you can point at it | A vendor region you don't choose or see |
| Vendor access to prompts | None — no vendor API in the path | Vendor staff and systems can reach it |
| Input retention / training | You set it; nothing kept "to improve a service" | Vendor policy, not yours |
| Audit logs | Yours, on hardware you control | Partial, in the vendor's console |
| Offline operation | Runs LAN-only or fully air-gapped | Dies with the connection |
| Prompt injection (LLM01) | Still your problem — design-level | Still a problem — design-level |
| Patching / scale convenience | You own updates and capacity planning | Vendor handles it; easier to scale |
Cloud genuinely wins on convenience and scale; local wins on data control, residency, and audit. The design-level LLM risks belong to both.
How TIS implements this on a real install
We don't hand you a checklist and walk away — we configure every layer above on a machine we hand-build and install on-site. The hardware is a local LLM server specced to your workload. On top of it we set the network isolation, the RBAC roles with SSO and MFA, AES-256 and TLS 1.3 by default, the logging you can hand an auditor, and the signed-update procedure that keeps an isolated box current.
Each layer has its own deep page: secure local AI for the LAN-only and air-gap options, private AI infrastructure for the end-to-end build, and the focused guides on access control, encryption, and air-gapping. The infrastructure work ties back to our main private AI infrastructure page.
We secure local AI on-site across Texas
You don't have to wire up the isolation, roles, encryption, and logging yourself. We spec the controls to your real workload, build the server, and install it on-site across Houston, Katy, Fulshear, Sugar Land and the Fort Bend area — then stay on call. See our Texas service areas.
Local LLM security questions
Is a self-hosted AI server automatically secure?+
No. Owning the box removes the vendor-exposure risk, but you still need network isolation, access control, encryption, and logging. Self-hosting changes where your data lives; it does not configure the controls for you.
Does running an LLM locally stop prompt injection?+
No. Prompt injection is a design-level LLM risk (OWASP LLM01) because models process instructions and data in the same channel. Local deployment helps with data exposure, not injection. We reduce the blast radius with strict output handling and least agency.
What is the difference between air-gapped and LAN-only?+
Air-gapped means the server has no physical or network path to the public internet at all. LAN-only allows access from devices on your local network but blocks outbound calls. Most businesses are well served by LAN-only with strong access control; air-gap is for the highest-sensitivity cases.
Is my data encrypted on a private AI server?+
On a TIS build, yes — AES-256 at rest and TLS 1.3 in transit by default, including the spots people forget like vector stores, caches, and backups. Encryption is only as good as key management, so we control where keys live and how they rotate.
How do you update an isolated or air-gapped AI server?+
Signed updates applied on your schedule via controlled media transfer — never an automatic phone-home. On an air-gapped box, update packages are verified before they are carried in, so the model stays current without opening an outbound path.
Does private AI make my business HIPAA or PCI compliant?+
No. Private AI infrastructure supports a compliance program by keeping data on-prem with encryption, access control, and audit logs. Certification stays with your auditor, assessor, or counsel. TIS builds and secures the infrastructure; we do not certify, audit, or provide legal advice.
Go deeper on the OWASP LLM Top 10, or back to Private AI Security.
Lock down your self-hosted AI the right way
Tell us the models you want to run and the data they'll touch — we'll wire up isolation, access control, encryption, and logging on a server you own outright, installed on-site across Houston and Fort Bend County.