Initial: Mesh-Skills, Configs, Scripts
This commit is contained in:
root
@@ -0,0 +1,291 @@
|
||||
---
|
||||
name: ollama-local-llm
|
||||
description: "Set up and run local LLMs with Ollama — install, model selection, OpenAI-compatible API, Hermes integration, and voice pipeline (STT/TTS)."
|
||||
version: 1.0.0
|
||||
author: Hermes Agent
|
||||
license: MIT
|
||||
platforms: [linux]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [ollama, local-llm, inference, voice, stt, tts, hermes-integration]
|
||||
category: mlops/inference
|
||||
related_skills: [llama-cpp, serving-llms-vllm, hermes-agent, hermes-gateway-systemd]
|
||||
---
|
||||
|
||||
# Ollama Local LLM Setup & Hermes Integration
|
||||
|
||||
Complete guide to running local LLMs via Ollama, integrating with Hermes Agent as fallback provider, and wiring up local voice (STT/TTS) for Telegram voice messages.
|
||||
|
||||
## When to Use
|
||||
|
||||
- User wants **local, free, offline LLM** (no API costs)
|
||||
- User wants **Hermes fallback** when cloud provider fails/times out
|
||||
- User needs **Telegram voice support** (STT → LLM → TTS) on headless server
|
||||
- Setting up **coding + chat + embedding models** for different tasks
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Linux server (x86_64 or ARM64), 8GB+ RAM recommended
|
||||
- No GPU required (CPU inference works, slower)
|
||||
- `curl`, `systemd`, `docker` (optional, for container execution)
|
||||
|
||||
---
|
||||
|
||||
## 1. Install Ollama
|
||||
|
||||
```bash
|
||||
# Official install script (installs to /usr/local/bin, creates ollama user, systemd service)
|
||||
curl -fsSL https://ollama.com/install.sh | sh
|
||||
|
||||
# Verify
|
||||
ollama --version
|
||||
systemctl status ollama
|
||||
```
|
||||
|
||||
**Note:** Runs as `ollama` user by default. API at `http://localhost:11434`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Pull Models (Recommended Set)
|
||||
|
||||
```bash
|
||||
# Coding model (used for /shell, /code, code generation
|
||||
ollama pull deepseek-r1:7b # 4.7 GB — best coding model for size
|
||||
|
||||
# Chat / general reasoning
|
||||
ollama pull llama3.1:8b # 4.9 GB — strong general chat
|
||||
|
||||
# Embeddings (for RAG, memory, semantic search)
|
||||
ollama pull nomic-embed-text # 274 MB — fast, good quality
|
||||
|
||||
# Optional: smaller fast model for quick responses
|
||||
# ollama pull gemma3:4b # ~3 GB — fast fallback
|
||||
|
||||
# List installed
|
||||
ollama list
|
||||
```
|
||||
|
||||
**Model selection guide:**
|
||||
| Use Case | Model | Size | RAM (CPU) |
|
||||
|----------|-------|------|-----------|
|
||||
| Code generation, shell commands | `deepseek-r1:7b` | 4.7 GB | ~6 GB |
|
||||
| General chat, reasoning | `llama3.1:8b` | 4.9 GB | ~6 GB |
|
||||
| Embeddings / RAG | `nomic-embed-text` | 274 MB | ~1 GB |
|
||||
| Fast responses | `gemma3:4b` | ~3 GB | ~4 GB |
|
||||
|
||||
---
|
||||
|
||||
## 3. Verify OpenAI-Compatible API
|
||||
|
||||
Ollama exposes `/v1/chat/completions` (OpenAI-compatible):
|
||||
|
||||
```bash
|
||||
# Test chat endpoint
|
||||
curl -s -X POST http://localhost:11434/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer *** \
|
||||
-d '{"model": "deepseek-r1:7b", "messages": [{"role": "user", "content": "test"}], "stream": false}' | jq .
|
||||
|
||||
# List models via API
|
||||
curl -s http://localhost:11434/v1/models | jq .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Configure Hermes to Use Ollama as Fallback
|
||||
|
||||
Edit `~/.hermes/config.yaml`:
|
||||
|
||||
```yaml
|
||||
model:
|
||||
default: nvidia/nemotron-3-ultra-550b-a55b:free # or your primary cloud model
|
||||
provider: openrouter
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
api_mode: chat_completions
|
||||
|
||||
providers:
|
||||
ollama:
|
||||
base_url: http://localhost:11434/v1
|
||||
api_key: ollama # dummy key, required by OpenAI SDK
|
||||
|
||||
fallback_providers:
|
||||
- provider: ollama
|
||||
model: deepseek-r1:7b
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
- Primary: OpenRouter free model (cloud)
|
||||
- On rate limit (429), overload (529), timeout, or connection error → automatic failover to Ollama
|
||||
- Uses `deepseek-r1:7b` for coding tasks, falls back to same for chat
|
||||
|
||||
---
|
||||
|
||||
## 5. Local Voice Pipeline (STT + TTS) for Telegram
|
||||
|
||||
### Install Dependencies (in Hermes venv)
|
||||
|
||||
```bash
|
||||
# Use Hermes' virtual environment
|
||||
/usr/local/lib/hermes-agent/venv/bin/python -m pip install faster-whisper piper-tts
|
||||
```
|
||||
|
||||
### Configure STT (faster-whisper) in `~/.hermes/config.yaml`
|
||||
|
||||
```yaml
|
||||
stt:
|
||||
enabled: true
|
||||
provider: local # uses faster-whisper locally
|
||||
local:
|
||||
model: base # tiny, base, small, medium, large-v3
|
||||
language: '' # auto-detect, or 'de', 'en', etc.
|
||||
use_gateway: true
|
||||
```
|
||||
|
||||
### Configure TTS (Piper) in `~/.hermes/config.yaml`
|
||||
|
||||
```yaml
|
||||
tts:
|
||||
provider: piper # local piper-tts
|
||||
piper:
|
||||
voice: en_US-lessac-medium # or de_DE-thorsten-medium for German
|
||||
use_gateway: true
|
||||
```
|
||||
|
||||
### Download Piper Voice (first run auto-downloads, or pre-fetch)
|
||||
|
||||
```bash
|
||||
# Test TTS (downloads voice model on first use)
|
||||
/usr/local/lib/hermes-agent/venv/bin/python -c "
|
||||
from piper.voice import PiperVoice
|
||||
voice = PiperVoice.load('en_US-lessac-medium') # triggers download
|
||||
print('Voice ready')
|
||||
"
|
||||
```
|
||||
|
||||
**Available voices:** https://github.com/rhasspy/piper/blob/master/VOICES.md
|
||||
- English: `en_US-lessac-medium`, `en_US-amy-medium`, `en_GB-northern_english_male-medium`
|
||||
- German: `de_DE-thorsten-medium`, `de_DE-karlsson-medium`
|
||||
|
||||
### Telegram Voice Flow
|
||||
|
||||
1. User sends **voice message** to Telegram bot
|
||||
2. Gateway downloads `.ogg` → converts → **faster-whisper (STT)** → text
|
||||
3. Text → **LLM** (OpenRouter or Ollama fallback) → response text
|
||||
4. Response text → **Piper (TTS)** → `.mp3` → sent back as **voice message**
|
||||
|
||||
**All local, no cloud API costs.**
|
||||
|
||||
---
|
||||
|
||||
## 6. Docker Integration (Optional)
|
||||
|
||||
For container-based execution in Hermes:
|
||||
|
||||
```bash
|
||||
# Ensure docker group access for hermes user
|
||||
usermod -aG docker $USER # or root if running as root
|
||||
newgrp docker
|
||||
|
||||
# Verify
|
||||
docker run --rm hello-world
|
||||
```
|
||||
|
||||
In `~/.hermes/config.yaml`:
|
||||
```yaml
|
||||
terminal:
|
||||
backend: docker # or local, singularity, modal
|
||||
docker_image: nikolaik/python-nodejs:python3.11-nodejs20
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Troubleshooting
|
||||
|
||||
### Ollama Not Starting / Port Conflict
|
||||
|
||||
```bash
|
||||
# Check port
|
||||
ss -tlnp | grep 11434
|
||||
|
||||
# Check logs
|
||||
journalctl -u ollama -n 50 --no-pager
|
||||
|
||||
# Restart
|
||||
systemctl restart ollama
|
||||
```
|
||||
|
||||
### Model Pull Stuck / Slow
|
||||
|
||||
```bash
|
||||
# Check disk space
|
||||
df -h /usr/share/ollama # default model storage
|
||||
|
||||
# Cancel and retry
|
||||
pkill -f "ollama pull"
|
||||
ollama pull <model>
|
||||
```
|
||||
|
||||
### Hermes Fallback Not Triggering
|
||||
|
||||
```bash
|
||||
# Verify config
|
||||
hermes status # shows configured fallback
|
||||
|
||||
# Test Ollama endpoint directly
|
||||
curl -s http://localhost:11434/v1/models | jq .
|
||||
|
||||
# Check Hermes logs for fallback attempts
|
||||
journalctl -u hermes-gateway -n 100 | grep -i fallback
|
||||
```
|
||||
|
||||
### Voice Not Working
|
||||
|
||||
```bash
|
||||
# Test STT
|
||||
/usr/local/lib/hermes-agent/venv/bin/python -c "
|
||||
from faster_whisper import WhisperModel
|
||||
model = WhisperModel('base', device='cpu', compute_type='int8')
|
||||
segments, _ = model.transcribe('/path/to/test.ogg', language='de')
|
||||
for s in segments: print(s.text)
|
||||
"
|
||||
|
||||
# Test TTS
|
||||
hermes tts "Test message" # or use text_to_speech tool
|
||||
|
||||
# Check gateway logs for voice processing
|
||||
journalctl -u hermes-gateway -n 50 | grep -i -e tts -e stt -e voice
|
||||
```
|
||||
|
||||
### Gateway Crash Loop (systemd)
|
||||
|
||||
See `hermes-gateway-systemd` skill for full diagnosis. Quick fix:
|
||||
|
||||
```bash
|
||||
# If config in /root/.hermes but service runs as user:
|
||||
# Edit /etc/systemd/system/hermes-gateway.service (or user service)
|
||||
User=root
|
||||
WorkingDirectory=/root/.hermes
|
||||
Environment="HOME=/root"
|
||||
Environment="HERMES_HOME=/root/.hermes"
|
||||
|
||||
systemctl daemon-reload
|
||||
pkill -9 -f 'hermes_cli.main gateway run'
|
||||
systemctl start hermes-gateway
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- `references/ollama-model-guide.md` — detailed model selection, quantization, performance
|
||||
- `references/voice-pipeline.md` — STT/TTS configuration, voice selection, latency tuning
|
||||
- `references/hermes-fallback-config.md` — advanced fallback chains, multiple local models
|
||||
|
||||
---
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `llama-cpp` — GGUF inference alternative (better for quantized models)
|
||||
- `serving-llms-vllm` — high-throughput serving (multi-user, OpenAI API)
|
||||
- `hermes-agent` — main Hermes config, gateway, platforms
|
||||
- `hermes-gateway-systemd` — service troubleshooting, crash loops
|
||||
Reference in New Issue
Block a user