Initial: Mesh-Skills, Configs, Scripts
This commit is contained in:
root
@@ -0,0 +1,296 @@
|
||||
# Local Voice Pipeline (STT + TTS) for Hermes Gateway
|
||||
|
||||
## Overview
|
||||
|
||||
This reference covers setting up **fully local** Speech-to-Text (STT) and Text-to-Speech (TTS) for Telegram voice messages on a headless server — no cloud APIs, no costs.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Telegram Voice (.ogg) → Gateway → ffmpeg → faster-whisper (STT) → Text
|
||||
↓
|
||||
LLM (OpenRouter/Ollama) → Response Text → Piper (TTS) → MP3 → Telegram Voice
|
||||
```
|
||||
|
||||
All components run locally on the server.
|
||||
|
||||
---
|
||||
|
||||
## STT: faster-whisper
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# In Hermes venv
|
||||
/usr/local/lib/hermes-agent/venv/bin/python -m pip install faster-whisper
|
||||
```
|
||||
|
||||
### Model Options
|
||||
|
||||
| Model | Size | Speed | Quality | Languages | RAM |
|
||||
|-------|------|-------|---------|-----------|-----|
|
||||
| `tiny` | 39 MB | Fastest | Low | 99 | ~1 GB |
|
||||
| `base` | 74 MB | Fast | Good | 99 | ~1 GB |
|
||||
| `small` | 244 MB | Medium | Better | 99 | ~2 GB |
|
||||
| `medium` | 769 MB | Slow | High | 99 | ~5 GB |
|
||||
| `large-v3` | 1.5 GB | Slowest | Best | 99 | ~10 GB |
|
||||
|
||||
**Recommendation for server:** `base` or `small` — good balance.
|
||||
|
||||
### Configuration (`~/.hermes/config.yaml`)
|
||||
|
||||
```yaml
|
||||
stt:
|
||||
enabled: true
|
||||
provider: local
|
||||
local:
|
||||
model: base # tiny, base, small, medium, large-v3
|
||||
language: '' # auto-detect, or 'de', 'en', 'fr', etc.
|
||||
use_gateway: true
|
||||
```
|
||||
|
||||
### Compute Types (for CPU)
|
||||
|
||||
| Type | Speed | Quality | RAM |
|
||||
|------|-------|---------|-----|
|
||||
| `int8` | Fastest | Good | Lowest |
|
||||
| `int8_float16` | Fast | Better | Medium |
|
||||
| `float16` | Medium | Best | Higher |
|
||||
| `float32` | Slow | Best | Highest |
|
||||
|
||||
**Default:** `int8` — best for CPU-only servers.
|
||||
|
||||
### Testing STT
|
||||
|
||||
```bash
|
||||
/usr/local/lib/hermes-agent/venv/bin/python -c "
|
||||
from faster_whisper import WhisperModel
|
||||
model = WhisperModel('base', device='cpu', compute_type='int8')
|
||||
segments, info = model.transcribe('test.ogg', language='de')
|
||||
for s in segments:
|
||||
print(f'[{s.start:.1f}s -> {s.end:.1f}s] {s.text}')
|
||||
print(f'Language: {info.language} (p={info.language_probability:.2f})')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## TTS: Piper
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# In Hermes venv
|
||||
/usr/local/lib/hermes-agent/venv/bin/python -m pip install piper-tts
|
||||
```
|
||||
|
||||
### Voice Models
|
||||
|
||||
Voices auto-download on first use from: https://github.com/rhasspy/piper/blob/master/VOICES.md
|
||||
|
||||
**Popular voices:**
|
||||
|
||||
| Language | Voice | Quality | Size |
|
||||
|----------|-------|---------|------|
|
||||
| English (US) | `en_US-lessac-medium` | Natural, clear | ~50 MB |
|
||||
| English (US) | `en_US-amy-medium` | Female, warm | ~50 MB |
|
||||
| English (UK) | `en_GB-northern_english_male-medium` | British accent | ~50 MB |
|
||||
| **German** | `de_DE-thorsten-medium` | **Best German male** | ~50 MB |
|
||||
| **German** | `de_DE-karlsson-medium` | German male, different tone | ~50 MB |
|
||||
| **German** | `de_DE-eva_k-x_low` | Female, fast (low quality) | ~10 MB |
|
||||
| French | `fr_FR-siwis-medium` | Good French | ~50 MB |
|
||||
| Spanish | `es_ES-sharvard-medium` | Good Spanish | ~50 MB |
|
||||
|
||||
### Configuration (`~/.hermes/config.yaml`)
|
||||
|
||||
```yaml
|
||||
tts:
|
||||
provider: piper
|
||||
piper:
|
||||
voice: en_US-lessac-medium # or de_DE-thorsten-medium for German
|
||||
use_gateway: true
|
||||
```
|
||||
|
||||
### Pre-download Voice (avoids first-request latency)
|
||||
|
||||
```bash
|
||||
# Test download
|
||||
/usr/local/lib/hermes-agent/venv/bin/python -c "
|
||||
from piper.voice import PiperVoice
|
||||
voice = PiperVoice.load('de_DE-thorsten-medium') # triggers download
|
||||
print(f'Voice loaded: {voice.config.num_symbols} symbols, {voice.config.sample_rate} Hz')
|
||||
"
|
||||
|
||||
# Check downloaded files
|
||||
ls -la ~/.cache/piper/
|
||||
```
|
||||
|
||||
### Testing TTS
|
||||
|
||||
```bash
|
||||
# Via Hermes tool
|
||||
text_to_speech text="Hallo, das ist ein Test auf Deutsch."
|
||||
|
||||
# Or direct Python
|
||||
/usr/local/lib/hermes-agent/venv/bin/python -c "
|
||||
from piper.voice import PiperVoice
|
||||
voice = PiperVoice.load('de_DE-thorsten-medium')
|
||||
import soundfile as sf
|
||||
audio = voice.synthesize('Hallo, das ist ein Test.', speaker_id=0)
|
||||
sf.write('/tmp/test.wav', audio, voice.config.sample_rate)
|
||||
print('Generated /tmp/test.wav')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Telegram Voice Integration
|
||||
|
||||
### Requirements
|
||||
|
||||
1. **ffmpeg** for audio conversion (`.ogg` → `.wav` for STT)
|
||||
2. **Gateway** configured with Telegram platform
|
||||
3. **STT + TTS** configured as above
|
||||
|
||||
```bash
|
||||
apt-get install -y ffmpeg
|
||||
```
|
||||
|
||||
### Flow Details
|
||||
|
||||
1. **User sends voice** → Telegram delivers `.ogg` (Opus codec) to gateway
|
||||
2. **Gateway downloads** → saves to temp file
|
||||
3. **ffmpeg converts** → `.wav` (16kHz mono) for STT
|
||||
4. **faster-whisper transcribes** → German/English text
|
||||
5. **LLM processes** → generates response
|
||||
6. **Piper synthesizes** → `.wav` → converted to `.mp3`
|
||||
7. **Gateway sends** → Telegram voice message
|
||||
|
||||
### Latency Estimates (CPU-only, 8 cores)
|
||||
|
||||
| Step | Time (approx) |
|
||||
|------|---------------|
|
||||
| Download + ffmpeg | 0.5-1s |
|
||||
| STT (base model) | 1-3s |
|
||||
| LLM (Ollama deepseek-r1:7b) | 3-10s |
|
||||
| TTS (Piper) | 0.5-2s |
|
||||
| **Total** | **5-16s** |
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
```yaml
|
||||
# In config.yaml - reduce STT model for speed
|
||||
stt:
|
||||
local:
|
||||
model: tiny # fastest, lower accuracy
|
||||
# OR
|
||||
model: base # recommended balance
|
||||
|
||||
# For TTS - use lower quality voice for speed
|
||||
tts:
|
||||
piper:
|
||||
voice: de_DE-eva_k-x_low # fast, smaller, lower quality
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### STT Not Working
|
||||
|
||||
```bash
|
||||
# Check model downloads
|
||||
ls -la ~/.cache/huggingface/hub/models--Systran--faster-whisper-*/
|
||||
|
||||
# Test manually
|
||||
/usr/local/lib/hermes-agent/venv/bin/python -c "
|
||||
from faster_whisper import WhisperModel
|
||||
model = WhisperModel('base', device='cpu', compute_type='int8')
|
||||
print('Model loaded OK')
|
||||
"
|
||||
```
|
||||
|
||||
### TTS Not Working / No Audio
|
||||
|
||||
```bash
|
||||
# Check voice cache
|
||||
ls -la ~/.cache/piper/
|
||||
|
||||
# Test direct synthesis
|
||||
/usr/local/lib/hermes-agent/venv/bin/python -c "
|
||||
from piper.voice import PiperVoice
|
||||
voice = PiperVoice.load('en_US-lessac-medium')
|
||||
audio = voice.synthesize('Test', speaker_id=0)
|
||||
print(f'Generated {len(audio)} samples at {voice.config.sample_rate} Hz')
|
||||
"
|
||||
```
|
||||
|
||||
### Telegram Voice Not Sent
|
||||
|
||||
```bash
|
||||
# Check gateway logs
|
||||
journalctl -u hermes-gateway -n 50 | grep -i -e voice -e tts -e stt -e ogg
|
||||
|
||||
# Verify ffmpeg
|
||||
ffmpeg -version
|
||||
|
||||
# Check Telegram bot has voice permissions
|
||||
# BotFather → Bot Settings → Voice Messages: ENABLED
|
||||
```
|
||||
|
||||
### Gateway Crash on Voice
|
||||
|
||||
```bash
|
||||
# Increase memory limits
|
||||
# In service file:
|
||||
MemoryMax=4G
|
||||
MemoryHigh=3G
|
||||
|
||||
# Or reduce model sizes (tiny STT, low-quality TTS)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Language-Specific Notes
|
||||
|
||||
### German STT
|
||||
|
||||
- `base` model: Good for standard German
|
||||
- `small` model: Better for dialects, technical terms
|
||||
- Set `language: 'de'` in config for faster processing (skips detection)
|
||||
|
||||
### German TTS
|
||||
|
||||
- `de_DE-thorsten-medium`: **Best quality**, natural male voice
|
||||
- `de_DE-karlsson-medium`: Alternative male voice
|
||||
- `de_DE-eva_k-x_low`: Fast, robotic female — only for speed-critical
|
||||
|
||||
### Mixed Language
|
||||
|
||||
- STT auto-detects well with `language: ''`
|
||||
- TTS: Use appropriate voice per language (config is global, so pick primary)
|
||||
|
||||
---
|
||||
|
||||
## Advanced: Custom Voice Training (Piper)
|
||||
|
||||
For custom voices: https://github.com/rhasspy/piper/blob/master/TRAINING.md
|
||||
|
||||
Requires: ~20min clean audio + transcripts, GPU for training (faster).
|
||||
|
||||
Not covered here — use pre-trained voices.
|
||||
|
||||
---
|
||||
|
||||
## Cost Summary
|
||||
|
||||
| Component | Cost |
|
||||
|-----------|------|
|
||||
| Ollama (LLM) | **Free** (local) |
|
||||
| faster-whisper (STT) | **Free** (local) |
|
||||
| Piper (TTS) | **Free** (local) |
|
||||
| ffmpeg | **Free** |
|
||||
| Telegram Bot API | **Free** |
|
||||
| **Total** | **$0/month** |
|
||||
|
||||
No API keys, no cloud dependencies, works offline.
|
||||
Reference in New Issue
Block a user