mesh-configs/skills/mlops-inference/ollama-local-llm/SKILL.md

name, description, version, author, license, platforms, metadata

name	description	version	author	license	platforms	metadata
ollama-local-llm	Set up and run local LLMs with Ollama — install, model selection, OpenAI-compatible API, Hermes integration, and voice pipeline (STT/TTS).	1.0.0	Hermes Agent	MIT	linux	hermes tags category related_skills ollama local-llm inference voice stt tts hermes-integration mlops/inference llama-cpp serving-llms-vllm hermes-agent hermes-gateway-systemd

Ollama Local LLM Setup & Hermes Integration

Complete guide to running local LLMs via Ollama, integrating with Hermes Agent as fallback provider, and wiring up local voice (STT/TTS) for Telegram voice messages.

When to Use

• User wants local, free, offline LLM (no API costs)
• User wants Hermes fallback when cloud provider fails/times out
• User needs Telegram voice support (STT → LLM → TTS) on headless server
• Setting up coding + chat + embedding models for different tasks

Prerequisites

• Linux server (x86_64 or ARM64), 8GB+ RAM recommended
• No GPU required (CPU inference works, slower)
• curl, systemd, docker (optional, for container execution)

---

1. Install Ollama

# Official install script (installs to /usr/local/bin, creates ollama user, systemd service)
curl -fsSL https://ollama.com/install.sh | sh

# Verify
ollama --version
systemctl status ollama

Note: Runs as ollama user by default. API at http://localhost:11434.

---

2. Pull Models (Recommended Set)

# Coding model (used for /shell, /code, code generation
ollama pull deepseek-r1:7b      # 4.7 GB — best coding model for size

# Chat / general reasoning
ollama pull llama3.1:8b           # 4.9 GB — strong general chat

# Embeddings (for RAG, memory, semantic search)
ollama pull nomic-embed-text      # 274 MB — fast, good quality

# Optional: smaller fast model for quick responses
# ollama pull gemma3:4b           # ~3 GB — fast fallback

# List installed
ollama list

Model selection guide:

Use Case	Model	Size	RAM (CPU)
Code generation, shell commands	deepseek-r1:7b	4.7 GB	~6 GB
General chat, reasoning	llama3.1:8b	4.9 GB	~6 GB
Embeddings / RAG	nomic-embed-text	274 MB	~1 GB
Fast responses	gemma3:4b	~3 GB	~4 GB

---

3. Verify OpenAI-Compatible API

Ollama exposes /v1/chat/completions (OpenAI-compatible):

# Test chat endpoint
curl -s -X POST http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer *** \
  -d '{"model": "deepseek-r1:7b", "messages": [{"role": "user", "content": "test"}], "stream": false}' | jq .

# List models via API
curl -s http://localhost:11434/v1/models | jq .

---

4. Configure Hermes to Use Ollama as Fallback

Edit ~/.hermes/config.yaml:

model:
  default: nvidia/nemotron-3-ultra-550b-a55b:free  # or your primary cloud model
  provider: openrouter
  base_url: https://openrouter.ai/api/v1
  api_mode: chat_completions

providers:
  ollama:
    base_url: http://localhost:11434/v1
    api_key: ollama  # dummy key, required by OpenAI SDK

fallback_providers:
  - provider: ollama
    model: deepseek-r1:7b

How it works:

• Primary: OpenRouter free model (cloud)
• On rate limit (429), overload (529), timeout, or connection error → automatic failover to Ollama
• Uses deepseek-r1:7b for coding tasks, falls back to same for chat

---

5. Local Voice Pipeline (STT + TTS) for Telegram

Install Dependencies (in Hermes venv)

# Use Hermes' virtual environment
/usr/local/lib/hermes-agent/venv/bin/python -m pip install faster-whisper piper-tts

Configure STT (faster-whisper) in ~/.hermes/config.yaml

stt:
  enabled: true
  provider: local  # uses faster-whisper locally
  local:
    model: base    # tiny, base, small, medium, large-v3
    language: ''   # auto-detect, or 'de', 'en', etc.
  use_gateway: true

Configure TTS (Piper) in ~/.hermes/config.yaml

tts:
  provider: piper  # local piper-tts
  piper:
    voice: en_US-lessac-medium  # or de_DE-thorsten-medium for German
  use_gateway: true

Download Piper Voice (first run auto-downloads, or pre-fetch)

# Test TTS (downloads voice model on first use)
/usr/local/lib/hermes-agent/venv/bin/python -c "
from piper.voice import PiperVoice
voice = PiperVoice.load('en_US-lessac-medium')  # triggers download
print('Voice ready')
"

Available voices: https://github.com/rhasspy/piper/blob/master/VOICES.md

• English: en_US-lessac-medium, en_US-amy-medium, en_GB-northern_english_male-medium
• German: de_DE-thorsten-medium, de_DE-karlsson-medium

Telegram Voice Flow

1. User sends voice message to Telegram bot
2. Gateway downloads .ogg → converts → faster-whisper (STT) → text
3. Text → LLM (OpenRouter or Ollama fallback) → response text
4. Response text → Piper (TTS) → .mp3 → sent back as voice message

All local, no cloud API costs.

---

6. Docker Integration (Optional)

For container-based execution in Hermes:

# Ensure docker group access for hermes user
usermod -aG docker $USER  # or root if running as root
newgrp docker

# Verify
docker run --rm hello-world

In ~/.hermes/config.yaml:

terminal:
  backend: docker  # or local, singularity, modal
  docker_image: nikolaik/python-nodejs:python3.11-nodejs20

---

7. Troubleshooting

Ollama Not Starting / Port Conflict

# Check port
ss -tlnp | grep 11434

# Check logs
journalctl -u ollama -n 50 --no-pager

# Restart
systemctl restart ollama

Model Pull Stuck / Slow

# Check disk space
df -h /usr/share/ollama  # default model storage

# Cancel and retry
pkill -f "ollama pull"
ollama pull <model>

Hermes Fallback Not Triggering

# Verify config
hermes status  # shows configured fallback

# Test Ollama endpoint directly
curl -s http://localhost:11434/v1/models | jq .

# Check Hermes logs for fallback attempts
journalctl -u hermes-gateway -n 100 | grep -i fallback

Voice Not Working

# Test STT
/usr/local/lib/hermes-agent/venv/bin/python -c "
from faster_whisper import WhisperModel
model = WhisperModel('base', device='cpu', compute_type='int8')
segments, _ = model.transcribe('/path/to/test.ogg', language='de')
for s in segments: print(s.text)
"

# Test TTS
hermes tts "Test message"  # or use text_to_speech tool

# Check gateway logs for voice processing
journalctl -u hermes-gateway -n 50 | grep -i -e tts -e stt -e voice

Gateway Crash Loop (systemd)

See hermes-gateway-systemd skill for full diagnosis. Quick fix:

# If config in /root/.hermes but service runs as user:
# Edit /etc/systemd/system/hermes-gateway.service (or user service)
User=root
WorkingDirectory=/root/.hermes
Environment="HOME=/root"
Environment="HERMES_HOME=/root/.hermes"

systemctl daemon-reload
pkill -9 -f 'hermes_cli.main gateway run'
systemctl start hermes-gateway

---

References

• references/ollama-model-guide.md — detailed model selection, quantization, performance
• references/voice-pipeline.md — STT/TTS configuration, voice selection, latency tuning
• references/hermes-fallback-config.md — advanced fallback chains, multiple local models

---

Related Skills

• llama-cpp — GGUF inference alternative (better for quantized models)
• serving-llms-vllm — high-throughput serving (multi-user, OpenAI API)
• hermes-agent — main Hermes config, gateway, platforms
• hermes-gateway-systemd — service troubleshooting, crash loops