Run Ollama locally: a no-nonsense setup guide

By Ricardo Ribeiro · · 3 min read · Ollama, LLM, Local AI, Setup

Why run a model on your own machine

Most people meet large language models through a browser tab. You type, a server somewhere answers, and a bill shows up at the end of the month. Running Ollama locally flips that. The model lives on your machine, your data stays on your machine, and the only cost is the electricity and the RAM it takes to run.

A local model is usually slower than a hosted one, so this is not about speed. It is about control. You can use it offline, you can feed it private documents without sending them anywhere, and you can tinker with it until it does what you want.

What you need

Ollama runs on Windows, macOS, and Linux. The hardware requirements are modest for small models and steep for big ones. A rule of thumb: the model size in gigabytes roughly matches the RAM you want. A 7B model is comfortable on 8 GB of RAM. A 70B model wants 64 GB and a decent GPU.

You do not need a GPU to start. The CPU will handle a small model, just slowly. If you have an NVIDIA card, Ollama uses it automatically.

Install

Head to ollama.com and download the installer for your system. On macOS and Windows it is a one-click install. On Linux it is a single command:

curl -fsSL https://ollama.com/install.sh | sh

After install, check that it works:

ollama --version

Pull a model

Ollama does not ship with a model. You pull one from its library. The most popular starting point is Llama 3.2, a small model that runs on modest hardware:

ollama pull llama3.2

That downloads a few gigabytes, so grab a coffee. When it finishes, run it:

ollama run llama3.2

You are now talking to a language model that lives entirely on your machine. Type a question, get an answer, type /bye to leave.

Use it from code

The command line is fine for a quick chat, but the real value is calling it from your own programs. Ollama exposes a REST API on port 11434. A request looks like this:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?"
}'

There are official clients for Python, JavaScript, and Go, so you can drop a local model into a script without much ceremony.

Where it gets interesting

Once the basics work, the fun starts. You can run a model as a drop-in replacement for an API you already use, keep a private assistant that never phones home, or experiment with fine-tuning on your own data. The setup is the boring part. What you do with a model that answers only to you is the interesting part.

Common gotchas

  • If the model answers slowly, it is probably running on CPU. Check with ollama ps to see what is loaded.
  • If you run out of memory, the model gets swapped to disk and everything grinds to a halt. Pull a smaller model.
  • Ollama keeps the last model loaded in memory. If you switch models a lot, it can feel sluggish. That is normal.

← All posts

Privacy · Cookies

Need a hand with a WordPress site that keeps breaking? See what I do and how I work.