What Is Ollama? Running an LLM on Your Own Server

What Ollama Is

Ollama packages the awkward parts of running a language model — obtaining the weights, loading them, managing memory, exposing an interface — behind a handful of simple commands. "ollama run llama3" downloads the model if needed and drops you into a conversation.

It also runs a local HTTP server, which is the part that matters for building things. Your application talks to a local endpoint in much the same shape as a commercial API, so switching an existing integration from a paid service to a self-hosted model is often a change of base URL rather than a rewrite.

It supports the common open-weight families — Llama, Mistral, Gemma, Qwen, Phi and others — in a range of sizes, and handles the quantised versions that make running them on ordinary hardware possible at all.

Why People Want to Run Their Own

• Data never leaves the machine. For legal documents, medical records, customer data or anything under a confidentiality obligation, this is often the whole reason and no amount of cost saving would change the decision.
• Fixed cost instead of per-token billing. Heavy, repetitive workloads — classifying thousands of records, summarising every incoming message — become predictable rather than proportional to volume.
• No rate limits. Your throughput is bounded by your hardware, not by someone else's quota.
• It works offline, and it keeps working when a provider changes pricing, deprecates a model, or has an outage.
• Experimentation is free. Trying twenty prompt variations costs nothing but time, which changes how willing you are to iterate.
• Model stability. A commercial model can change under you without notice; a downloaded one behaves the same way next year.

The Truth About CPU vs GPU

This is where expectations need setting honestly. Ollama runs on CPU perfectly well in the sense that it works — but "works" and "feels good" are different things.

RAM Is the Real Bottleneck

People shop for CPU cores. The number that actually decides whether a model runs at all is RAM, because the entire model has to be held in memory while it works. Not enough memory does not mean slow — it means it will not start.

When a GPU-Free VPS Is Worth It

Being specific about this saves people from an expensive disappointment in either direction.

Installing It on a VPS, in Outline

• Start with a VPS whose RAM matches the model you intend to run, using the table above. This is the decision that determines whether anything else works.
• Install Ollama with the official install script. It is a single command on a standard Linux distribution.
• Pull a small model first — a 3-billion-parameter one — and measure the actual speed on your own hardware before committing to something larger.
• Keep the API bound to localhost by default. Ollama has no authentication of its own, and exposing it directly to the internet means anyone who finds it can use your server.
• If you need remote access, put it behind a reverse proxy with authentication and HTTPS. This is not optional if the machine has a public IP.
• Watch memory during the first real workload. Swapping to disk turns "slow" into "unusable", and it is the most common cause of a setup that worked in testing and collapsed in use.

Security Worth Taking Seriously

Ollama ships without authentication. That is a reasonable default for something intended to run on your laptop, and a serious problem the moment it runs on a server with a public IP address.

An exposed Ollama endpoint is an open invitation: anyone who finds it can run inference on your hardware at your expense, and depending on your setup may be able to see what else is on the machine. Automated scanners find open ports within hours of them opening, not days.

The minimum safe configuration is to keep it listening on localhost and reach it through an SSH tunnel, or to place a reverse proxy in front of it that enforces authentication and TLS. Both are straightforward, and either is far less work than dealing with the consequences of skipping them.

Also give some thought to what you feed it. Self-hosting means the data stays on your server rather than going to a third party — but only if the server itself is secured, which is your responsibility now rather than a vendor's.