Initial: Mesh-Configs vom 03.09.2026 01:23

2026-09-03 01:23:55 +02:00
commit 4b65134f8c
797 changed files with 146197 additions and 0 deletions
@@ -0,0 +1,173 @@
+# Ollama Model Selection Guide
+
+## Model Categories & Recommendations
+
+### Coding Models (Shell commands, code generation, /code tasks)
+
+| Model | Size | Context | Best For | RAM (CPU) |
+|-------|------|---------|----------|-----------|
+| `deepseek-r1:7b` | 4.7 GB | 32K | **Best overall** — strong coding, good instruction following, multi-language | ~6 GB |
+| `qwen2.5-coder:14b` | 9.0 GB | 32K | Better quality, slower | ~12 GB |
+| `deepseek-coder-v2:16b` | 9.4 GB | 128K | Long context coding tasks | ~12 GB |
+| `codellama:13b` | 7.4 GB | 16K | Good alternative, older | ~10 GB |
+| `starcoder2:7b` | 4.4 GB | 16K | Fast, decent quality | ~6 GB |
+
+**Recommendation:** `deepseek-r1:7b` — best balance of quality/size/speed for Hermes `/shell` and `/code`.
+
+### Chat / General Reasoning Models
+
+| Model | Size | Context | Best For | RAM (CPU) |
+|-------|------|---------|----------|-----------|
+| `llama3.1:8b` | 4.9 GB | 128K | **Best general** — strong reasoning, multilingual, tool use | ~6 GB |
+| `llama3.1:70b` | 40 GB | 128K | Best quality, needs GPU | ~48 GB |
+| `nemotron3:8b` | 4.7 GB | 4K | NVIDIA optimized, fast | ~6 GB |
+| `gemma2:9b` | 5.4 GB | 8K | Google, strong multilingual | ~7 GB |
+| `phi3:14b` | 8.2 GB | 128K | Microsoft, good reasoning | ~11 GB |
+| `command-r:35b` | 20 GB | 128K | RAG-optimized, tool use | ~24 GB |
+
+**Recommendation:** `llama3.1:8b` — best balance for chat fallback.
+
+### Embedding Models (RAG, memory, semantic search)
+
+| Model | Size | Dim | Best For | RAM |
+|-------|------|-----|----------|-----|
+| `nomic-embed-text` | 274 MB | 768 | **Best general** — fast, good quality | ~1 GB |
+| `nomic-embed-text:v1.5` | 274 MB | 768 | Improved version | ~1 GB |
+| `mxbai-embed-large` | 668 MB | 1024 | Higher quality, larger | ~2 GB |
+| `bge-m3` | 1.2 GB | 1024 | Multi-lingual, dense+sparse | ~3 GB |
+| `all-minilm:l6-v2` | 90 MB | 384 | Fast, small, lower quality | ~500 MB |
+
+**Recommendation:** `nomic-embed-text` — best speed/quality for Hermes memory/RAG.
+
+### Fast / Small Models (Quick responses, low RAM)
+
+| Model | Size | Context | Best For | RAM |
+|-------|------|---------|----------|-----|
+| `gemma3:4b` | ~3 GB | 32K | Fast chat, multilingual | ~4 GB |
+| `phi3:3.8b` | 2.3 GB | 128K | Very fast, decent quality | ~3 GB |
+| `qwen2.5:3b` | 2.0 GB | 32K | Fast, multilingual | ~3 GB |
+| `llama3.2:3b` | 2.0 GB | 128K | Tiny but capable | ~3 GB |
+
+---
+
+## Quantization Guide
+
+Ollama uses GGUF quantization. Suffix indicates quantization level:
+
+| Suffix | Bits | Quality | Speed | Size Reduction |
+|--------|------|---------|-------|----------------|
+| `:latest` / `:fp16` | 16 | Best | Slowest | Baseline |
+| `:q8_0` | 8 | Near-lossless | Fast | ~50% |
+| `:q6_k` | 6 | Very good | Fast | ~60% |
+| `:q5_k_m` | 5 | Good | Faster | ~65% |
+| `:q4_k_m` | 4 | **Recommended** | Fast | ~75% |
+| `:q4_0` | 4 | Good | Fastest | ~75% |
+| `:q3_k_m` | 3 | Acceptable | Very fast | ~80% |
+| `:q2_k` | 2 | Low | Fastest | ~85% |
+
+**Default (no suffix)** = `q4_k_m` usually — best balance.
+
+**For Hermes:** Use defaults (`:latest` → `q4_k_m`). Explicitly pull `:q4_k_m` if you want certainty.
+
+---
+
+## Multi-Model Setup for Hermes
+
+```bash
+# Primary coding model (used by fallback)
+ollama pull qwen2.5-coder:7b
+
+# General chat fallback (optional - can use same)
+ollama pull llama3.1:8b
+
+# Embeddings for memory/RAG
+ollama pull nomic-embed-text
+
+# Optional: fast model for quick responses
+ollama pull gemma3:4b
+```
+
+### Configuring Different Models for Different Tasks
+
+In `~/.hermes/config.yaml`, you can set up multiple fallback providers:
+
+```yaml
+providers:
+  ollama:
+    base_url: http://localhost:11434/v1
+    api_key: ollama
+
+fallback_providers:
+  # Primary fallback for coding
+  - provider: ollama
+    model: qwen2.5-coder:7b
+  # Secondary fallback for chat (if first fails)
+  - provider: ollama
+    model: llama3.1:8b
+```
+
+---
+
+## Performance Tuning (CPU Inference)
+
+### Environment Variables
+
+```bash
+# Threads (default: all cores)
+export OLLAMA_NUM_PARALLEL=2        # concurrent requests
+export OLLAMA_NUM_THREAD=8          # threads per request
+
+# Context window (default: model default, e.g., 4096 or 8192)
+# Increase in request: "num_ctx": 16384
+
+# GPU offloading (if GPU available)
+# export OLLAMA_GPU_LAYERS=999      # offload all layers
+```
+
+### For Headless Server (This Session's Setup)
+
+```bash
+# 8 cores, 16GB RAM - good config:
+OLLAMA_NUM_PARALLEL=2
+OLLAMA_NUM_THREAD=8
+
+# Add to ollama service:
+# systemctl edit ollama
+# [Service]
+# Environment="OLLAMA_NUM_PARALLEL=2"
+# Environment="OLLAMA_NUM_THREAD=8"
+```
+
+---
+
+## Model Storage
+
+- Default: `/usr/share/ollama/.ollama/models`
+- Override: `OLLAMA_MODELS=/path/to/models`
+- Each model: ~1-40 GB depending on size/quantization
+
+```bash
+# Check usage
+du -sh /usr/share/ollama/.ollama/models
+
+# Move to larger disk
+systemctl stop ollama
+mv /usr/share/ollama /mnt/storage/
+ln -s /mnt/storage/ollama /usr/share/ollama
+systemctl start ollama
+```
+
+---
+
+## Updating Models
+
+```bash
+# Pull latest version
+ollama pull qwen2.5-coder:7b
+
+# Remove old
+ollama rm qwen2.5-coder:7b:old
+
+# List with sizes
+ollama list
+```