5.0 KiB
5.0 KiB
Ollama Model Selection Guide
Model Categories & Recommendations
Coding Models (Shell commands, code generation, /code tasks)
| Model | Size | Context | Best For | RAM (CPU) |
|---|---|---|---|---|
deepseek-r1:7b |
4.7 GB | 32K | Best overall — strong coding, good instruction following, multi-language | ~6 GB |
qwen2.5-coder:14b |
9.0 GB | 32K | Better quality, slower | ~12 GB |
deepseek-coder-v2:16b |
9.4 GB | 128K | Long context coding tasks | ~12 GB |
codellama:13b |
7.4 GB | 16K | Good alternative, older | ~10 GB |
starcoder2:7b |
4.4 GB | 16K | Fast, decent quality | ~6 GB |
Recommendation: deepseek-r1:7b — best balance of quality/size/speed for Hermes /shell and /code.
Chat / General Reasoning Models
| Model | Size | Context | Best For | RAM (CPU) |
|---|---|---|---|---|
llama3.1:8b |
4.9 GB | 128K | Best general — strong reasoning, multilingual, tool use | ~6 GB |
llama3.1:70b |
40 GB | 128K | Best quality, needs GPU | ~48 GB |
nemotron3:8b |
4.7 GB | 4K | NVIDIA optimized, fast | ~6 GB |
gemma2:9b |
5.4 GB | 8K | Google, strong multilingual | ~7 GB |
phi3:14b |
8.2 GB | 128K | Microsoft, good reasoning | ~11 GB |
command-r:35b |
20 GB | 128K | RAG-optimized, tool use | ~24 GB |
Recommendation: llama3.1:8b — best balance for chat fallback.
Embedding Models (RAG, memory, semantic search)
| Model | Size | Dim | Best For | RAM |
|---|---|---|---|---|
nomic-embed-text |
274 MB | 768 | Best general — fast, good quality | ~1 GB |
nomic-embed-text:v1.5 |
274 MB | 768 | Improved version | ~1 GB |
mxbai-embed-large |
668 MB | 1024 | Higher quality, larger | ~2 GB |
bge-m3 |
1.2 GB | 1024 | Multi-lingual, dense+sparse | ~3 GB |
all-minilm:l6-v2 |
90 MB | 384 | Fast, small, lower quality | ~500 MB |
Recommendation: nomic-embed-text — best speed/quality for Hermes memory/RAG.
Fast / Small Models (Quick responses, low RAM)
| Model | Size | Context | Best For | RAM |
|---|---|---|---|---|
gemma3:4b |
~3 GB | 32K | Fast chat, multilingual | ~4 GB |
phi3:3.8b |
2.3 GB | 128K | Very fast, decent quality | ~3 GB |
qwen2.5:3b |
2.0 GB | 32K | Fast, multilingual | ~3 GB |
llama3.2:3b |
2.0 GB | 128K | Tiny but capable | ~3 GB |
Quantization Guide
Ollama uses GGUF quantization. Suffix indicates quantization level:
| Suffix | Bits | Quality | Speed | Size Reduction |
|---|---|---|---|---|
:latest / :fp16 |
16 | Best | Slowest | Baseline |
:q8_0 |
8 | Near-lossless | Fast | ~50% |
:q6_k |
6 | Very good | Fast | ~60% |
:q5_k_m |
5 | Good | Faster | ~65% |
:q4_k_m |
4 | Recommended | Fast | ~75% |
:q4_0 |
4 | Good | Fastest | ~75% |
:q3_k_m |
3 | Acceptable | Very fast | ~80% |
:q2_k |
2 | Low | Fastest | ~85% |
Default (no suffix) = q4_k_m usually — best balance.
For Hermes: Use defaults (:latest → q4_k_m). Explicitly pull :q4_k_m if you want certainty.
Multi-Model Setup for Hermes
# Primary coding model (used by fallback)
ollama pull qwen2.5-coder:7b
# General chat fallback (optional - can use same)
ollama pull llama3.1:8b
# Embeddings for memory/RAG
ollama pull nomic-embed-text
# Optional: fast model for quick responses
ollama pull gemma3:4b
Configuring Different Models for Different Tasks
In ~/.hermes/config.yaml, you can set up multiple fallback providers:
providers:
ollama:
base_url: http://localhost:11434/v1
api_key: ollama
fallback_providers:
# Primary fallback for coding
- provider: ollama
model: qwen2.5-coder:7b
# Secondary fallback for chat (if first fails)
- provider: ollama
model: llama3.1:8b
Performance Tuning (CPU Inference)
Environment Variables
# Threads (default: all cores)
export OLLAMA_NUM_PARALLEL=2 # concurrent requests
export OLLAMA_NUM_THREAD=8 # threads per request
# Context window (default: model default, e.g., 4096 or 8192)
# Increase in request: "num_ctx": 16384
# GPU offloading (if GPU available)
# export OLLAMA_GPU_LAYERS=999 # offload all layers
For Headless Server (This Session's Setup)
# 8 cores, 16GB RAM - good config:
OLLAMA_NUM_PARALLEL=2
OLLAMA_NUM_THREAD=8
# Add to ollama service:
# systemctl edit ollama
# [Service]
# Environment="OLLAMA_NUM_PARALLEL=2"
# Environment="OLLAMA_NUM_THREAD=8"
Model Storage
- Default:
/usr/share/ollama/.ollama/models - Override:
OLLAMA_MODELS=/path/to/models - Each model: ~1-40 GB depending on size/quantization
# Check usage
du -sh /usr/share/ollama/.ollama/models
# Move to larger disk
systemctl stop ollama
mv /usr/share/ollama /mnt/storage/
ln -s /mnt/storage/ollama /usr/share/ollama
systemctl start ollama
Updating Models
# Pull latest version
ollama pull qwen2.5-coder:7b
# Remove old
ollama rm qwen2.5-coder:7b:old
# List with sizes
ollama list