Your first local model, step by step
From nothing to a model answering on your own computer: which size to pick, four commands, how to see what it's using, and how to remove it again.
So far this course has been about how things work. This lesson is different: at the end of it, a language model runs on your own computer, with no account and no subscription. You don’t need a special graphics card to try it. A small model runs on an ordinary laptop, just more slowly.
I use a tool called Ollama here, because it handles the setup details and runs on Windows, macOS and Linux. My own engine, imp, isn’t the right start: it’s built for one specific card. If you’d rather click than type, LM Studio does the same job with a window and buttons.
Step 1: pick a size that fits#
The one question that matters is how much memory you have. On a computer with a suitable graphics card, that’s the card’s VRAM. Without one, it’s your normal RAM, and the model runs on the processor. How a model fits on a card explains why size is the deciding factor.
| Your memory | try | download |
|---|---|---|
| 4 GB | qwen3:1.7b |
1.4 GB |
| 8 GB | qwen3:4b |
2.5 GB |
| 16 GB | qwen3:8b |
5.2 GB |
| 24 GB | qwen3:14b |
9.3 GB |
| 32 GB or more | qwen3:30b |
18.6 GB |
These are cautious starting points, and the sizes are from Ollama’s library on 7 November 2026. Qwen3 is only an example, and the library changes all the time. With what you know from reading model specs, you can judge any other entry the same way. If in doubt, start one row lower. And if the model is meant for work, check its license first: what “open” means for a model explains what to look for.
Step 2: install it#
On Windows and macOS, download the installer from ollama.com/download and run it. On Linux there’s a script:
curl -fsSL https://ollama.com/install.sh | sh
That runs a script from the internet with your rights, so read it first if that bothers you. If you already use Docker, you can skip the install:
docker run -d --name ollama -p 127.0.0.1:11434:11434 ollama/ollama
In that case, put docker exec -it ollama in front of the ollama commands below.
Step 3: start a conversation#
Open a terminal and type this, with the name you picked in step 1:
ollama run qwen3:8b
The first time, it downloads the file, so this step takes as long as your connection
needs. After that you get a prompt. Type something and press Enter. Qwen3 is a model that
thinks before it answers, so you may see it
reasoning first. Type /bye to leave.
The model now runs from a file on your own disk and keeps working without a network connection.
Step 4: look at what it’s using#
While a model is loaded, open a second terminal and ask what’s running:
ollama ps
This is what I got for the smallest Qwen3 on a machine with no GPU in use, shortened a little:
NAME SIZE PROCESSOR CONTEXT RUNNER
qwen3:0.6b 1.0 GB 100% CPU 4096 llamacpp
Three things are worth reading here:
- SIZE is memory in use, and it’s bigger than the download. That file was 522 MB. The rest is working memory, much of it for the context window.
- PROCESSOR says where it runs.
100% GPUis what you want on a graphics card. A split like40%/60% CPU/GPUmeans the model didn’t fit, and it will be much slower. Pick a smaller one. - CONTEXT is how much text it can hold at once, here 4096 tokens. That’s less than the model can do. It’s a default that saves memory.
Step 5: use it from other programs#
Ollama also answers on a local address, in the same format as the big cloud APIs. Many tools can talk to it if you give them this address, and you can try it by hand:
curl http://localhost:11434/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model": "qwen3:8b", "messages": [{"role": "user", "content": "Reply with the word ok."}]}'
One thing to watch here. With "max_tokens": 40 the answer came back empty: the
model spent all 40 tokens on thinking and never got to the reply. If you set a limit,
leave room.
What to expect#
- It’s less capable than the big services. A model that fits at home is much smaller, and it makes things up more often. Try it on your own tasks before you trust it.
- It knows nothing new. It has no web access here, and its reading stopped at a cutoff.
- Your text stays with you, as long as you run a normal local model. Ollama’s library also lists cloud models, which run on someone else’s server, so look at what you pick.
- It can be slow. Without a graphics card, a few words per second is normal for a mid-sized model. Why a GPU helps is its own lesson.
Cleaning up#
Models are big files, and removing them is easy:
ollama list
ollama rm qwen3:8b
I ran these commands with Ollama 0.40.15 in a Linux container on the processor, using
qwen3:0.6b. The larger models in the table are sizes from the library, not something I
tested for this lesson.
You now have a working local model. From here, why you’d run a model yourself is worth a read, now that you’ve got one to compare against.