AgentofHeart
MyBoerse.bz Pro Member
Self-Hosting LLMs Securely: Hardening Ollama and vLLM
MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHz, 2 Ch
Level: Intermediate | Genre: eLearning | Language: English | Duration: 109 Lectures ( 10h 33m ) | Size: 2.2 GB
Running an open-weight model on your own hardware keeps prompts and data in house, but it also makes you the operator of a new kind of server. This course treats a model server as a server: you measure what the default install exposes, harden it layer by layer, and prove every change with a check that runs.
You start by looking, not fixing. Ollama and a CPU build of vLLM are installed the documented way, their API surfaces are mapped endpoint by endpoint, and a verification script records every check that fails. That baseline scoreboard is what the rest of the course turns green.
Ollama is hardened on the host first: bind address, CORS, a dedicated service user, systemd sandboxing and resource limits, with the documentation's own statement that it does not require authentication taken at face value. Then the same service goes into a container that runs as non-root, read-only, without capabilities and published on loopback only, with the image pinned by digest and scanned.
vLLM gets its own section. You read its security guidance line by line, set the bind address explicitly, and measure exactly which endpoints -api-key protects and which still answer without it. What the flags cannot close, the gateway does: nginx with TLS from a private CA, one key per client, an endpoint allowlist, rate and size limits, and an LLM proxy layer for per-client budgets.
The host is then isolated with a default-deny firewall and no route out, and models reach it only through a staging step. Model files are treated as code-adjacent: you compare pickle, safetensors and GGUF, scan files, pin Hugging Face downloads to a full commit, verify digests, sign and verify a model, and serve only what a model manifest lists.
The last sections are about running it: secrets out of compose files, what the servers log about prompts, structured gateway logs, private metrics, alert rules, Falco on the model container, advisory tracking and controlled upgrades with rollback. Two incident drills and a full rebuild from one repository close the course, with a final evidence pack mapped to NIST AI RMF, ISO/IEC 42001, MITRE ATLAS and the EU AI Act milestones.
This is a defensive course. Everything runs on your own lab host; nothing is pointed at systems you do not own and there is no internet scanning. GPU hosts and multi-node vLLM are covered conceptually and marked as such, because the recorded bench is CPU only. Twelve downloadable documents come with it, including a threat-model worksheet, hardening checklists for Ollama and vLLM, a gateway guide, a model intake policy, monitoring and incident runbooks and an auditor evidence index.
RapidGator
DDownload
NitroFlare
