296 lines
7.3 KiB
Markdown
296 lines
7.3 KiB
Markdown
# Local Voice Pipeline (STT + TTS) for Hermes Gateway
|
|
|
|
## Overview
|
|
|
|
This reference covers setting up **fully local** Speech-to-Text (STT) and Text-to-Speech (TTS) for Telegram voice messages on a headless server — no cloud APIs, no costs.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Telegram Voice (.ogg) → Gateway → ffmpeg → faster-whisper (STT) → Text
|
|
↓
|
|
LLM (OpenRouter/Ollama) → Response Text → Piper (TTS) → MP3 → Telegram Voice
|
|
```
|
|
|
|
All components run locally on the server.
|
|
|
|
---
|
|
|
|
## STT: faster-whisper
|
|
|
|
### Installation
|
|
|
|
```bash
|
|
# In Hermes venv
|
|
/usr/local/lib/hermes-agent/venv/bin/python -m pip install faster-whisper
|
|
```
|
|
|
|
### Model Options
|
|
|
|
| Model | Size | Speed | Quality | Languages | RAM |
|
|
|-------|------|-------|---------|-----------|-----|
|
|
| `tiny` | 39 MB | Fastest | Low | 99 | ~1 GB |
|
|
| `base` | 74 MB | Fast | Good | 99 | ~1 GB |
|
|
| `small` | 244 MB | Medium | Better | 99 | ~2 GB |
|
|
| `medium` | 769 MB | Slow | High | 99 | ~5 GB |
|
|
| `large-v3` | 1.5 GB | Slowest | Best | 99 | ~10 GB |
|
|
|
|
**Recommendation for server:** `base` or `small` — good balance.
|
|
|
|
### Configuration (`~/.hermes/config.yaml`)
|
|
|
|
```yaml
|
|
stt:
|
|
enabled: true
|
|
provider: local
|
|
local:
|
|
model: base # tiny, base, small, medium, large-v3
|
|
language: '' # auto-detect, or 'de', 'en', 'fr', etc.
|
|
use_gateway: true
|
|
```
|
|
|
|
### Compute Types (for CPU)
|
|
|
|
| Type | Speed | Quality | RAM |
|
|
|------|-------|---------|-----|
|
|
| `int8` | Fastest | Good | Lowest |
|
|
| `int8_float16` | Fast | Better | Medium |
|
|
| `float16` | Medium | Best | Higher |
|
|
| `float32` | Slow | Best | Highest |
|
|
|
|
**Default:** `int8` — best for CPU-only servers.
|
|
|
|
### Testing STT
|
|
|
|
```bash
|
|
/usr/local/lib/hermes-agent/venv/bin/python -c "
|
|
from faster_whisper import WhisperModel
|
|
model = WhisperModel('base', device='cpu', compute_type='int8')
|
|
segments, info = model.transcribe('test.ogg', language='de')
|
|
for s in segments:
|
|
print(f'[{s.start:.1f}s -> {s.end:.1f}s] {s.text}')
|
|
print(f'Language: {info.language} (p={info.language_probability:.2f})')
|
|
"
|
|
```
|
|
|
|
---
|
|
|
|
## TTS: Piper
|
|
|
|
### Installation
|
|
|
|
```bash
|
|
# In Hermes venv
|
|
/usr/local/lib/hermes-agent/venv/bin/python -m pip install piper-tts
|
|
```
|
|
|
|
### Voice Models
|
|
|
|
Voices auto-download on first use from: https://github.com/rhasspy/piper/blob/master/VOICES.md
|
|
|
|
**Popular voices:**
|
|
|
|
| Language | Voice | Quality | Size |
|
|
|----------|-------|---------|------|
|
|
| English (US) | `en_US-lessac-medium` | Natural, clear | ~50 MB |
|
|
| English (US) | `en_US-amy-medium` | Female, warm | ~50 MB |
|
|
| English (UK) | `en_GB-northern_english_male-medium` | British accent | ~50 MB |
|
|
| **German** | `de_DE-thorsten-medium` | **Best German male** | ~50 MB |
|
|
| **German** | `de_DE-karlsson-medium` | German male, different tone | ~50 MB |
|
|
| **German** | `de_DE-eva_k-x_low` | Female, fast (low quality) | ~10 MB |
|
|
| French | `fr_FR-siwis-medium` | Good French | ~50 MB |
|
|
| Spanish | `es_ES-sharvard-medium` | Good Spanish | ~50 MB |
|
|
|
|
### Configuration (`~/.hermes/config.yaml`)
|
|
|
|
```yaml
|
|
tts:
|
|
provider: piper
|
|
piper:
|
|
voice: en_US-lessac-medium # or de_DE-thorsten-medium for German
|
|
use_gateway: true
|
|
```
|
|
|
|
### Pre-download Voice (avoids first-request latency)
|
|
|
|
```bash
|
|
# Test download
|
|
/usr/local/lib/hermes-agent/venv/bin/python -c "
|
|
from piper.voice import PiperVoice
|
|
voice = PiperVoice.load('de_DE-thorsten-medium') # triggers download
|
|
print(f'Voice loaded: {voice.config.num_symbols} symbols, {voice.config.sample_rate} Hz')
|
|
"
|
|
|
|
# Check downloaded files
|
|
ls -la ~/.cache/piper/
|
|
```
|
|
|
|
### Testing TTS
|
|
|
|
```bash
|
|
# Via Hermes tool
|
|
text_to_speech text="Hallo, das ist ein Test auf Deutsch."
|
|
|
|
# Or direct Python
|
|
/usr/local/lib/hermes-agent/venv/bin/python -c "
|
|
from piper.voice import PiperVoice
|
|
voice = PiperVoice.load('de_DE-thorsten-medium')
|
|
import soundfile as sf
|
|
audio = voice.synthesize('Hallo, das ist ein Test.', speaker_id=0)
|
|
sf.write('/tmp/test.wav', audio, voice.config.sample_rate)
|
|
print('Generated /tmp/test.wav')
|
|
"
|
|
```
|
|
|
|
---
|
|
|
|
## Telegram Voice Integration
|
|
|
|
### Requirements
|
|
|
|
1. **ffmpeg** for audio conversion (`.ogg` → `.wav` for STT)
|
|
2. **Gateway** configured with Telegram platform
|
|
3. **STT + TTS** configured as above
|
|
|
|
```bash
|
|
apt-get install -y ffmpeg
|
|
```
|
|
|
|
### Flow Details
|
|
|
|
1. **User sends voice** → Telegram delivers `.ogg` (Opus codec) to gateway
|
|
2. **Gateway downloads** → saves to temp file
|
|
3. **ffmpeg converts** → `.wav` (16kHz mono) for STT
|
|
4. **faster-whisper transcribes** → German/English text
|
|
5. **LLM processes** → generates response
|
|
6. **Piper synthesizes** → `.wav` → converted to `.mp3`
|
|
7. **Gateway sends** → Telegram voice message
|
|
|
|
### Latency Estimates (CPU-only, 8 cores)
|
|
|
|
| Step | Time (approx) |
|
|
|------|---------------|
|
|
| Download + ffmpeg | 0.5-1s |
|
|
| STT (base model) | 1-3s |
|
|
| LLM (Ollama deepseek-r1:7b) | 3-10s |
|
|
| TTS (Piper) | 0.5-2s |
|
|
| **Total** | **5-16s** |
|
|
|
|
### Optimization Tips
|
|
|
|
```yaml
|
|
# In config.yaml - reduce STT model for speed
|
|
stt:
|
|
local:
|
|
model: tiny # fastest, lower accuracy
|
|
# OR
|
|
model: base # recommended balance
|
|
|
|
# For TTS - use lower quality voice for speed
|
|
tts:
|
|
piper:
|
|
voice: de_DE-eva_k-x_low # fast, smaller, lower quality
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### STT Not Working
|
|
|
|
```bash
|
|
# Check model downloads
|
|
ls -la ~/.cache/huggingface/hub/models--Systran--faster-whisper-*/
|
|
|
|
# Test manually
|
|
/usr/local/lib/hermes-agent/venv/bin/python -c "
|
|
from faster_whisper import WhisperModel
|
|
model = WhisperModel('base', device='cpu', compute_type='int8')
|
|
print('Model loaded OK')
|
|
"
|
|
```
|
|
|
|
### TTS Not Working / No Audio
|
|
|
|
```bash
|
|
# Check voice cache
|
|
ls -la ~/.cache/piper/
|
|
|
|
# Test direct synthesis
|
|
/usr/local/lib/hermes-agent/venv/bin/python -c "
|
|
from piper.voice import PiperVoice
|
|
voice = PiperVoice.load('en_US-lessac-medium')
|
|
audio = voice.synthesize('Test', speaker_id=0)
|
|
print(f'Generated {len(audio)} samples at {voice.config.sample_rate} Hz')
|
|
"
|
|
```
|
|
|
|
### Telegram Voice Not Sent
|
|
|
|
```bash
|
|
# Check gateway logs
|
|
journalctl -u hermes-gateway -n 50 | grep -i -e voice -e tts -e stt -e ogg
|
|
|
|
# Verify ffmpeg
|
|
ffmpeg -version
|
|
|
|
# Check Telegram bot has voice permissions
|
|
# BotFather → Bot Settings → Voice Messages: ENABLED
|
|
```
|
|
|
|
### Gateway Crash on Voice
|
|
|
|
```bash
|
|
# Increase memory limits
|
|
# In service file:
|
|
MemoryMax=4G
|
|
MemoryHigh=3G
|
|
|
|
# Or reduce model sizes (tiny STT, low-quality TTS)
|
|
```
|
|
|
|
---
|
|
|
|
## Language-Specific Notes
|
|
|
|
### German STT
|
|
|
|
- `base` model: Good for standard German
|
|
- `small` model: Better for dialects, technical terms
|
|
- Set `language: 'de'` in config for faster processing (skips detection)
|
|
|
|
### German TTS
|
|
|
|
- `de_DE-thorsten-medium`: **Best quality**, natural male voice
|
|
- `de_DE-karlsson-medium`: Alternative male voice
|
|
- `de_DE-eva_k-x_low`: Fast, robotic female — only for speed-critical
|
|
|
|
### Mixed Language
|
|
|
|
- STT auto-detects well with `language: ''`
|
|
- TTS: Use appropriate voice per language (config is global, so pick primary)
|
|
|
|
---
|
|
|
|
## Advanced: Custom Voice Training (Piper)
|
|
|
|
For custom voices: https://github.com/rhasspy/piper/blob/master/TRAINING.md
|
|
|
|
Requires: ~20min clean audio + transcripts, GPU for training (faster).
|
|
|
|
Not covered here — use pre-trained voices.
|
|
|
|
---
|
|
|
|
## Cost Summary
|
|
|
|
| Component | Cost |
|
|
|-----------|------|
|
|
| Ollama (LLM) | **Free** (local) |
|
|
| faster-whisper (STT) | **Free** (local) |
|
|
| Piper (TTS) | **Free** (local) |
|
|
| ffmpeg | **Free** |
|
|
| Telegram Bot API | **Free** |
|
|
| **Total** | **$0/month** |
|
|
|
|
No API keys, no cloud dependencies, works offline. |