# Ollama Model Selection Guide

## Model Categories & Recommendations

### Coding Models (Shell commands, code generation, /code tasks)

| Model | Size | Context | Best For | RAM (CPU) |
|-------|------|---------|----------|-----------|
| `deepseek-r1:7b` | 4.7 GB | 32K | **Best overall** — strong coding, good instruction following, multi-language | ~6 GB |
| `qwen2.5-coder:14b` | 9.0 GB | 32K | Better quality, slower | ~12 GB |
| `deepseek-coder-v2:16b` | 9.4 GB | 128K | Long context coding tasks | ~12 GB |
| `codellama:13b` | 7.4 GB | 16K | Good alternative, older | ~10 GB |
| `starcoder2:7b` | 4.4 GB | 16K | Fast, decent quality | ~6 GB |

**Recommendation:** `deepseek-r1:7b` — best balance of quality/size/speed for Hermes `/shell` and `/code`.

### Chat / General Reasoning Models

| Model | Size | Context | Best For | RAM (CPU) |
|-------|------|---------|----------|-----------|
| `llama3.1:8b` | 4.9 GB | 128K | **Best general** — strong reasoning, multilingual, tool use | ~6 GB |
| `llama3.1:70b` | 40 GB | 128K | Best quality, needs GPU | ~48 GB |
| `nemotron3:8b` | 4.7 GB | 4K | NVIDIA optimized, fast | ~6 GB |
| `gemma2:9b` | 5.4 GB | 8K | Google, strong multilingual | ~7 GB |
| `phi3:14b` | 8.2 GB | 128K | Microsoft, good reasoning | ~11 GB |
| `command-r:35b` | 20 GB | 128K | RAG-optimized, tool use | ~24 GB |

**Recommendation:** `llama3.1:8b` — best balance for chat fallback.

### Embedding Models (RAG, memory, semantic search)

| Model | Size | Dim | Best For | RAM |
|-------|------|-----|----------|-----|
| `nomic-embed-text` | 274 MB | 768 | **Best general** — fast, good quality | ~1 GB |
| `nomic-embed-text:v1.5` | 274 MB | 768 | Improved version | ~1 GB |
| `mxbai-embed-large` | 668 MB | 1024 | Higher quality, larger | ~2 GB |
| `bge-m3` | 1.2 GB | 1024 | Multi-lingual, dense+sparse | ~3 GB |
| `all-minilm:l6-v2` | 90 MB | 384 | Fast, small, lower quality | ~500 MB |

**Recommendation:** `nomic-embed-text` — best speed/quality for Hermes memory/RAG.

### Fast / Small Models (Quick responses, low RAM)

| Model | Size | Context | Best For | RAM |
|-------|------|---------|----------|-----|
| `gemma3:4b` | ~3 GB | 32K | Fast chat, multilingual | ~4 GB |
| `phi3:3.8b` | 2.3 GB | 128K | Very fast, decent quality | ~3 GB |
| `qwen2.5:3b` | 2.0 GB | 32K | Fast, multilingual | ~3 GB |
| `llama3.2:3b` | 2.0 GB | 128K | Tiny but capable | ~3 GB |

---

## Quantization Guide

Ollama uses GGUF quantization. Suffix indicates quantization level:

| Suffix | Bits | Quality | Speed | Size Reduction |
|--------|------|---------|-------|----------------|
| `:latest` / `:fp16` | 16 | Best | Slowest | Baseline |
| `:q8_0` | 8 | Near-lossless | Fast | ~50% |
| `:q6_k` | 6 | Very good | Fast | ~60% |
| `:q5_k_m` | 5 | Good | Faster | ~65% |
| `:q4_k_m` | 4 | **Recommended** | Fast | ~75% |
| `:q4_0` | 4 | Good | Fastest | ~75% |
| `:q3_k_m` | 3 | Acceptable | Very fast | ~80% |
| `:q2_k` | 2 | Low | Fastest | ~85% |

**Default (no suffix)** = `q4_k_m` usually — best balance.

**For Hermes:** Use defaults (`:latest` → `q4_k_m`). Explicitly pull `:q4_k_m` if you want certainty.

---

## Multi-Model Setup for Hermes

```bash
# Primary coding model (used by fallback)
ollama pull qwen2.5-coder:7b

# General chat fallback (optional - can use same)
ollama pull llama3.1:8b

# Embeddings for memory/RAG
ollama pull nomic-embed-text

# Optional: fast model for quick responses
ollama pull gemma3:4b
```

### Configuring Different Models for Different Tasks

In `~/.hermes/config.yaml`, you can set up multiple fallback providers:

```yaml
providers:
  ollama:
    base_url: http://localhost:11434/v1
    api_key: ollama

fallback_providers:
  # Primary fallback for coding
  - provider: ollama
    model: qwen2.5-coder:7b
  # Secondary fallback for chat (if first fails)
  - provider: ollama
    model: llama3.1:8b
```

---

## Performance Tuning (CPU Inference)

### Environment Variables

```bash
# Threads (default: all cores)
export OLLAMA_NUM_PARALLEL=2        # concurrent requests
export OLLAMA_NUM_THREAD=8          # threads per request

# Context window (default: model default, e.g., 4096 or 8192)
# Increase in request: "num_ctx": 16384

# GPU offloading (if GPU available)
# export OLLAMA_GPU_LAYERS=999      # offload all layers
```

### For Headless Server (This Session's Setup)

```bash
# 8 cores, 16GB RAM - good config:
OLLAMA_NUM_PARALLEL=2
OLLAMA_NUM_THREAD=8

# Add to ollama service:
# systemctl edit ollama
# [Service]
# Environment="OLLAMA_NUM_PARALLEL=2"
# Environment="OLLAMA_NUM_THREAD=8"
```

---

## Model Storage

- Default: `/usr/share/ollama/.ollama/models`
- Override: `OLLAMA_MODELS=/path/to/models`
- Each model: ~1-40 GB depending on size/quantization

```bash
# Check usage
du -sh /usr/share/ollama/.ollama/models

# Move to larger disk
systemctl stop ollama
mv /usr/share/ollama /mnt/storage/
ln -s /mnt/storage/ollama /usr/share/ollama
systemctl start ollama
```

---

## Updating Models

```bash
# Pull latest version
ollama pull qwen2.5-coder:7b

# Remove old
ollama rm qwen2.5-coder:7b:old

# List with sizes
ollama list
```