mesh-configs/skills/mlops-inference/ollama-local-llm/SKILL.md

---
name: ollama-local-llm
description: "Set up and run local LLMs with Ollama — install, model selection, OpenAI-compatible API, Hermes integration, and voice pipeline (STT/TTS)."
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux]
metadata:
  hermes:
    tags: [ollama, local-llm, inference, voice, stt, tts, hermes-integration]
    category: mlops/inference
    related_skills: [llama-cpp, serving-llms-vllm, hermes-agent, hermes-gateway-systemd]
---

# Ollama Local LLM Setup & Hermes Integration

Complete guide to running local LLMs via Ollama, integrating with Hermes Agent as fallback provider, and wiring up local voice (STT/TTS) for Telegram voice messages.

## When to Use

- User wants **local, free, offline LLM** (no API costs)
- User wants **Hermes fallback** when cloud provider fails/times out
- User needs **Telegram voice support** (STT → LLM → TTS) on headless server
- Setting up **coding + chat + embedding models** for different tasks

## Prerequisites

- Linux server (x86_64 or ARM64), 8GB+ RAM recommended
- No GPU required (CPU inference works, slower)
- `curl`, `systemd`, `docker` (optional, for container execution)

---

## 1. Install Ollama

```bash
# Official install script (installs to /usr/local/bin, creates ollama user, systemd service)
curl -fsSL https://ollama.com/install.sh | sh

# Verify
ollama --version
systemctl status ollama
```

**Note:** Runs as `ollama` user by default. API at `http://localhost:11434`.

---

## 2. Pull Models (Recommended Set)

```bash
# Coding model (used for /shell, /code, code generation
ollama pull deepseek-r1:7b      # 4.7 GB — best coding model for size

# Chat / general reasoning
ollama pull llama3.1:8b           # 4.9 GB — strong general chat

# Embeddings (for RAG, memory, semantic search)
ollama pull nomic-embed-text      # 274 MB — fast, good quality

# Optional: smaller fast model for quick responses
# ollama pull gemma3:4b           # ~3 GB — fast fallback

# List installed
ollama list
```

**Model selection guide:**
| Use Case | Model | Size | RAM (CPU) |
|----------|-------|------|-----------|
| Code generation, shell commands | `deepseek-r1:7b` | 4.7 GB | ~6 GB |
| General chat, reasoning | `llama3.1:8b` | 4.9 GB | ~6 GB |
| Embeddings / RAG | `nomic-embed-text` | 274 MB | ~1 GB |
| Fast responses | `gemma3:4b` | ~3 GB | ~4 GB |

---

## 3. Verify OpenAI-Compatible API

Ollama exposes `/v1/chat/completions` (OpenAI-compatible):

```bash
# Test chat endpoint
curl -s -X POST http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer *** \
  -d '{"model": "deepseek-r1:7b", "messages": [{"role": "user", "content": "test"}], "stream": false}' | jq .

# List models via API
curl -s http://localhost:11434/v1/models | jq .
```

---

## 4. Configure Hermes to Use Ollama as Fallback

Edit `~/.hermes/config.yaml`:

```yaml
model:
  default: nvidia/nemotron-3-ultra-550b-a55b:free  # or your primary cloud model
  provider: openrouter
  base_url: https://openrouter.ai/api/v1
  api_mode: chat_completions

providers:
  ollama:
    base_url: http://localhost:11434/v1
    api_key: ollama  # dummy key, required by OpenAI SDK

fallback_providers:
  - provider: ollama
    model: deepseek-r1:7b
```

**How it works:**
- Primary: OpenRouter free model (cloud)
- On rate limit (429), overload (529), timeout, or connection error → automatic failover to Ollama
- Uses `deepseek-r1:7b` for coding tasks, falls back to same for chat

---

## 5. Local Voice Pipeline (STT + TTS) for Telegram

### Install Dependencies (in Hermes venv)

```bash
# Use Hermes' virtual environment
/usr/local/lib/hermes-agent/venv/bin/python -m pip install faster-whisper piper-tts
```

### Configure STT (faster-whisper) in `~/.hermes/config.yaml`

```yaml
stt:
  enabled: true
  provider: local  # uses faster-whisper locally
  local:
    model: base    # tiny, base, small, medium, large-v3
    language: ''   # auto-detect, or 'de', 'en', etc.
  use_gateway: true
```

### Configure TTS (Piper) in `~/.hermes/config.yaml`

```yaml
tts:
  provider: piper  # local piper-tts
  piper:
    voice: en_US-lessac-medium  # or de_DE-thorsten-medium for German
  use_gateway: true
```

### Download Piper Voice (first run auto-downloads, or pre-fetch)

```bash
# Test TTS (downloads voice model on first use)
/usr/local/lib/hermes-agent/venv/bin/python -c "
from piper.voice import PiperVoice
voice = PiperVoice.load('en_US-lessac-medium')  # triggers download
print('Voice ready')
"
```

**Available voices:** https://github.com/rhasspy/piper/blob/master/VOICES.md
- English: `en_US-lessac-medium`, `en_US-amy-medium`, `en_GB-northern_english_male-medium`
- German: `de_DE-thorsten-medium`, `de_DE-karlsson-medium`

### Telegram Voice Flow

1. User sends **voice message** to Telegram bot
2. Gateway downloads `.ogg` → converts → **faster-whisper (STT)** → text
3. Text → **LLM** (OpenRouter or Ollama fallback) → response text
4. Response text → **Piper (TTS)** → `.mp3` → sent back as **voice message**

**All local, no cloud API costs.**

---

## 6. Docker Integration (Optional)

For container-based execution in Hermes:

```bash
# Ensure docker group access for hermes user
usermod -aG docker $USER  # or root if running as root
newgrp docker

# Verify
docker run --rm hello-world
```

In `~/.hermes/config.yaml`:
```yaml
terminal:
  backend: docker  # or local, singularity, modal
  docker_image: nikolaik/python-nodejs:python3.11-nodejs20
```

---

## 7. Troubleshooting

### Ollama Not Starting / Port Conflict

```bash
# Check port
ss -tlnp | grep 11434

# Check logs
journalctl -u ollama -n 50 --no-pager

# Restart
systemctl restart ollama
```

### Model Pull Stuck / Slow

```bash
# Check disk space
df -h /usr/share/ollama  # default model storage

# Cancel and retry
pkill -f "ollama pull"
ollama pull <model>
```

### Hermes Fallback Not Triggering

```bash
# Verify config
hermes status  # shows configured fallback

# Test Ollama endpoint directly
curl -s http://localhost:11434/v1/models | jq .

# Check Hermes logs for fallback attempts
journalctl -u hermes-gateway -n 100 | grep -i fallback
```

### Voice Not Working

```bash
# Test STT
/usr/local/lib/hermes-agent/venv/bin/python -c "
from faster_whisper import WhisperModel
model = WhisperModel('base', device='cpu', compute_type='int8')
segments, _ = model.transcribe('/path/to/test.ogg', language='de')
for s in segments: print(s.text)
"

# Test TTS
hermes tts "Test message"  # or use text_to_speech tool

# Check gateway logs for voice processing
journalctl -u hermes-gateway -n 50 | grep -i -e tts -e stt -e voice
```

### Gateway Crash Loop (systemd)

See `hermes-gateway-systemd` skill for full diagnosis. Quick fix:

```bash
# If config in /root/.hermes but service runs as user:
# Edit /etc/systemd/system/hermes-gateway.service (or user service)
User=root
WorkingDirectory=/root/.hermes
Environment="HOME=/root"
Environment="HERMES_HOME=/root/.hermes"

systemctl daemon-reload
pkill -9 -f 'hermes_cli.main gateway run'
systemctl start hermes-gateway
```

---

## References

- `references/ollama-model-guide.md` — detailed model selection, quantization, performance
- `references/voice-pipeline.md` — STT/TTS configuration, voice selection, latency tuning
- `references/hermes-fallback-config.md` — advanced fallback chains, multiple local models

---

## Related Skills

- `llama-cpp` — GGUF inference alternative (better for quantized models)
- `serving-llms-vllm` — high-throughput serving (multi-user, OpenAI API)
- `hermes-agent` — main Hermes config, gateway, platforms
- `hermes-gateway-systemd` — service troubleshooting, crash loops