173 lines
5.0 KiB
Markdown
173 lines
5.0 KiB
Markdown
# Ollama Model Selection Guide
|
|
|
|
## Model Categories & Recommendations
|
|
|
|
### Coding Models (Shell commands, code generation, /code tasks)
|
|
|
|
| Model | Size | Context | Best For | RAM (CPU) |
|
|
|-------|------|---------|----------|-----------|
|
|
| `deepseek-r1:7b` | 4.7 GB | 32K | **Best overall** — strong coding, good instruction following, multi-language | ~6 GB |
|
|
| `qwen2.5-coder:14b` | 9.0 GB | 32K | Better quality, slower | ~12 GB |
|
|
| `deepseek-coder-v2:16b` | 9.4 GB | 128K | Long context coding tasks | ~12 GB |
|
|
| `codellama:13b` | 7.4 GB | 16K | Good alternative, older | ~10 GB |
|
|
| `starcoder2:7b` | 4.4 GB | 16K | Fast, decent quality | ~6 GB |
|
|
|
|
**Recommendation:** `deepseek-r1:7b` — best balance of quality/size/speed for Hermes `/shell` and `/code`.
|
|
|
|
### Chat / General Reasoning Models
|
|
|
|
| Model | Size | Context | Best For | RAM (CPU) |
|
|
|-------|------|---------|----------|-----------|
|
|
| `llama3.1:8b` | 4.9 GB | 128K | **Best general** — strong reasoning, multilingual, tool use | ~6 GB |
|
|
| `llama3.1:70b` | 40 GB | 128K | Best quality, needs GPU | ~48 GB |
|
|
| `nemotron3:8b` | 4.7 GB | 4K | NVIDIA optimized, fast | ~6 GB |
|
|
| `gemma2:9b` | 5.4 GB | 8K | Google, strong multilingual | ~7 GB |
|
|
| `phi3:14b` | 8.2 GB | 128K | Microsoft, good reasoning | ~11 GB |
|
|
| `command-r:35b` | 20 GB | 128K | RAG-optimized, tool use | ~24 GB |
|
|
|
|
**Recommendation:** `llama3.1:8b` — best balance for chat fallback.
|
|
|
|
### Embedding Models (RAG, memory, semantic search)
|
|
|
|
| Model | Size | Dim | Best For | RAM |
|
|
|-------|------|-----|----------|-----|
|
|
| `nomic-embed-text` | 274 MB | 768 | **Best general** — fast, good quality | ~1 GB |
|
|
| `nomic-embed-text:v1.5` | 274 MB | 768 | Improved version | ~1 GB |
|
|
| `mxbai-embed-large` | 668 MB | 1024 | Higher quality, larger | ~2 GB |
|
|
| `bge-m3` | 1.2 GB | 1024 | Multi-lingual, dense+sparse | ~3 GB |
|
|
| `all-minilm:l6-v2` | 90 MB | 384 | Fast, small, lower quality | ~500 MB |
|
|
|
|
**Recommendation:** `nomic-embed-text` — best speed/quality for Hermes memory/RAG.
|
|
|
|
### Fast / Small Models (Quick responses, low RAM)
|
|
|
|
| Model | Size | Context | Best For | RAM |
|
|
|-------|------|---------|----------|-----|
|
|
| `gemma3:4b` | ~3 GB | 32K | Fast chat, multilingual | ~4 GB |
|
|
| `phi3:3.8b` | 2.3 GB | 128K | Very fast, decent quality | ~3 GB |
|
|
| `qwen2.5:3b` | 2.0 GB | 32K | Fast, multilingual | ~3 GB |
|
|
| `llama3.2:3b` | 2.0 GB | 128K | Tiny but capable | ~3 GB |
|
|
|
|
---
|
|
|
|
## Quantization Guide
|
|
|
|
Ollama uses GGUF quantization. Suffix indicates quantization level:
|
|
|
|
| Suffix | Bits | Quality | Speed | Size Reduction |
|
|
|--------|------|---------|-------|----------------|
|
|
| `:latest` / `:fp16` | 16 | Best | Slowest | Baseline |
|
|
| `:q8_0` | 8 | Near-lossless | Fast | ~50% |
|
|
| `:q6_k` | 6 | Very good | Fast | ~60% |
|
|
| `:q5_k_m` | 5 | Good | Faster | ~65% |
|
|
| `:q4_k_m` | 4 | **Recommended** | Fast | ~75% |
|
|
| `:q4_0` | 4 | Good | Fastest | ~75% |
|
|
| `:q3_k_m` | 3 | Acceptable | Very fast | ~80% |
|
|
| `:q2_k` | 2 | Low | Fastest | ~85% |
|
|
|
|
**Default (no suffix)** = `q4_k_m` usually — best balance.
|
|
|
|
**For Hermes:** Use defaults (`:latest` → `q4_k_m`). Explicitly pull `:q4_k_m` if you want certainty.
|
|
|
|
---
|
|
|
|
## Multi-Model Setup for Hermes
|
|
|
|
```bash
|
|
# Primary coding model (used by fallback)
|
|
ollama pull qwen2.5-coder:7b
|
|
|
|
# General chat fallback (optional - can use same)
|
|
ollama pull llama3.1:8b
|
|
|
|
# Embeddings for memory/RAG
|
|
ollama pull nomic-embed-text
|
|
|
|
# Optional: fast model for quick responses
|
|
ollama pull gemma3:4b
|
|
```
|
|
|
|
### Configuring Different Models for Different Tasks
|
|
|
|
In `~/.hermes/config.yaml`, you can set up multiple fallback providers:
|
|
|
|
```yaml
|
|
providers:
|
|
ollama:
|
|
base_url: http://localhost:11434/v1
|
|
api_key: ollama
|
|
|
|
fallback_providers:
|
|
# Primary fallback for coding
|
|
- provider: ollama
|
|
model: qwen2.5-coder:7b
|
|
# Secondary fallback for chat (if first fails)
|
|
- provider: ollama
|
|
model: llama3.1:8b
|
|
```
|
|
|
|
---
|
|
|
|
## Performance Tuning (CPU Inference)
|
|
|
|
### Environment Variables
|
|
|
|
```bash
|
|
# Threads (default: all cores)
|
|
export OLLAMA_NUM_PARALLEL=2 # concurrent requests
|
|
export OLLAMA_NUM_THREAD=8 # threads per request
|
|
|
|
# Context window (default: model default, e.g., 4096 or 8192)
|
|
# Increase in request: "num_ctx": 16384
|
|
|
|
# GPU offloading (if GPU available)
|
|
# export OLLAMA_GPU_LAYERS=999 # offload all layers
|
|
```
|
|
|
|
### For Headless Server (This Session's Setup)
|
|
|
|
```bash
|
|
# 8 cores, 16GB RAM - good config:
|
|
OLLAMA_NUM_PARALLEL=2
|
|
OLLAMA_NUM_THREAD=8
|
|
|
|
# Add to ollama service:
|
|
# systemctl edit ollama
|
|
# [Service]
|
|
# Environment="OLLAMA_NUM_PARALLEL=2"
|
|
# Environment="OLLAMA_NUM_THREAD=8"
|
|
```
|
|
|
|
---
|
|
|
|
## Model Storage
|
|
|
|
- Default: `/usr/share/ollama/.ollama/models`
|
|
- Override: `OLLAMA_MODELS=/path/to/models`
|
|
- Each model: ~1-40 GB depending on size/quantization
|
|
|
|
```bash
|
|
# Check usage
|
|
du -sh /usr/share/ollama/.ollama/models
|
|
|
|
# Move to larger disk
|
|
systemctl stop ollama
|
|
mv /usr/share/ollama /mnt/storage/
|
|
ln -s /mnt/storage/ollama /usr/share/ollama
|
|
systemctl start ollama
|
|
```
|
|
|
|
---
|
|
|
|
## Updating Models
|
|
|
|
```bash
|
|
# Pull latest version
|
|
ollama pull qwen2.5-coder:7b
|
|
|
|
# Remove old
|
|
ollama rm qwen2.5-coder:7b:old
|
|
|
|
# List with sizes
|
|
ollama list
|
|
``` |