mesh-configs/skills/mlops-inference/ollama-local-llm/references/voice-pipeline.md

Local Voice Pipeline (STT + TTS) for Hermes Gateway

Overview

This reference covers setting up fully local Speech-to-Text (STT) and Text-to-Speech (TTS) for Telegram voice messages on a headless server — no cloud APIs, no costs.

Architecture

Telegram Voice (.ogg) → Gateway → ffmpeg → faster-whisper (STT) → Text
                                                              ↓
LLM (OpenRouter/Ollama) → Response Text → Piper (TTS) → MP3 → Telegram Voice

All components run locally on the server.

---

STT: faster-whisper

Installation

# In Hermes venv
/usr/local/lib/hermes-agent/venv/bin/python -m pip install faster-whisper

Model Options

Model	Size	Speed	Quality	Languages	RAM
tiny	39 MB	Fastest	Low	99	~1 GB
base	74 MB	Fast	Good	99	~1 GB
small	244 MB	Medium	Better	99	~2 GB
medium	769 MB	Slow	High	99	~5 GB
large-v3	1.5 GB	Slowest	Best	99	~10 GB

Recommendation for server: base or small — good balance.

Configuration (~/.hermes/config.yaml)

stt:
  enabled: true
  provider: local
  local:
    model: base          # tiny, base, small, medium, large-v3
    language: ''         # auto-detect, or 'de', 'en', 'fr', etc.
  use_gateway: true

Compute Types (for CPU)

Type	Speed	Quality	RAM
int8	Fastest	Good	Lowest
int8_float16	Fast	Better	Medium
float16	Medium	Best	Higher
float32	Slow	Best	Highest

Default: int8 — best for CPU-only servers.

Testing STT

/usr/local/lib/hermes-agent/venv/bin/python -c "
from faster_whisper import WhisperModel
model = WhisperModel('base', device='cpu', compute_type='int8')
segments, info = model.transcribe('test.ogg', language='de')
for s in segments:
    print(f'[{s.start:.1f}s -> {s.end:.1f}s] {s.text}')
print(f'Language: {info.language} (p={info.language_probability:.2f})')
"

---

TTS: Piper

Installation

# In Hermes venv
/usr/local/lib/hermes-agent/venv/bin/python -m pip install piper-tts

Voice Models

Voices auto-download on first use from: https://github.com/rhasspy/piper/blob/master/VOICES.md

Popular voices:

Language	Voice	Quality	Size
English (US)	en_US-lessac-medium	Natural, clear	~50 MB
English (US)	en_US-amy-medium	Female, warm	~50 MB
English (UK)	en_GB-northern_english_male-medium	British accent	~50 MB
German	de_DE-thorsten-medium	Best German male	~50 MB
German	de_DE-karlsson-medium	German male, different tone	~50 MB
German	de_DE-eva_k-x_low	Female, fast (low quality)	~10 MB
French	fr_FR-siwis-medium	Good French	~50 MB
Spanish	es_ES-sharvard-medium	Good Spanish	~50 MB

Configuration (~/.hermes/config.yaml)

tts:
  provider: piper
  piper:
    voice: en_US-lessac-medium  # or de_DE-thorsten-medium for German
  use_gateway: true

Pre-download Voice (avoids first-request latency)

# Test download
/usr/local/lib/hermes-agent/venv/bin/python -c "
from piper.voice import PiperVoice
voice = PiperVoice.load('de_DE-thorsten-medium')  # triggers download
print(f'Voice loaded: {voice.config.num_symbols} symbols, {voice.config.sample_rate} Hz')
"

# Check downloaded files
ls -la ~/.cache/piper/

Testing TTS

# Via Hermes tool
text_to_speech text="Hallo, das ist ein Test auf Deutsch."

# Or direct Python
/usr/local/lib/hermes-agent/venv/bin/python -c "
from piper.voice import PiperVoice
voice = PiperVoice.load('de_DE-thorsten-medium')
import soundfile as sf
audio = voice.synthesize('Hallo, das ist ein Test.', speaker_id=0)
sf.write('/tmp/test.wav', audio, voice.config.sample_rate)
print('Generated /tmp/test.wav')
"

---

Telegram Voice Integration

Requirements

1. ffmpeg for audio conversion (.ogg → .wav for STT)
2. Gateway configured with Telegram platform
3. STT + TTS configured as above

apt-get install -y ffmpeg

Flow Details

1. User sends voice → Telegram delivers .ogg (Opus codec) to gateway
2. Gateway downloads → saves to temp file
3. ffmpeg converts → .wav (16kHz mono) for STT
4. faster-whisper transcribes → German/English text
5. LLM processes → generates response
6. Piper synthesizes → .wav → converted to .mp3
7. Gateway sends → Telegram voice message

Latency Estimates (CPU-only, 8 cores)

Step	Time (approx)
Download + ffmpeg	0.5-1s
STT (base model)	1-3s
LLM (Ollama deepseek-r1:7b)	3-10s
TTS (Piper)	0.5-2s
Total	5-16s

Optimization Tips

# In config.yaml - reduce STT model for speed
stt:
  local:
    model: tiny        # fastest, lower accuracy
    # OR
    model: base        # recommended balance

# For TTS - use lower quality voice for speed
tts:
  piper:
    voice: de_DE-eva_k-x_low  # fast, smaller, lower quality

---

Troubleshooting

STT Not Working

# Check model downloads
ls -la ~/.cache/huggingface/hub/models--Systran--faster-whisper-*/

# Test manually
/usr/local/lib/hermes-agent/venv/bin/python -c "
from faster_whisper import WhisperModel
model = WhisperModel('base', device='cpu', compute_type='int8')
print('Model loaded OK')
"

TTS Not Working / No Audio

# Check voice cache
ls -la ~/.cache/piper/

# Test direct synthesis
/usr/local/lib/hermes-agent/venv/bin/python -c "
from piper.voice import PiperVoice
voice = PiperVoice.load('en_US-lessac-medium')
audio = voice.synthesize('Test', speaker_id=0)
print(f'Generated {len(audio)} samples at {voice.config.sample_rate} Hz')
"

Telegram Voice Not Sent

# Check gateway logs
journalctl -u hermes-gateway -n 50 | grep -i -e voice -e tts -e stt -e ogg

# Verify ffmpeg
ffmpeg -version

# Check Telegram bot has voice permissions
# BotFather → Bot Settings → Voice Messages: ENABLED

Gateway Crash on Voice

# Increase memory limits
# In service file:
MemoryMax=4G
MemoryHigh=3G

# Or reduce model sizes (tiny STT, low-quality TTS)

---

Language-Specific Notes

German STT

• base model: Good for standard German
• small model: Better for dialects, technical terms
• Set language: 'de' in config for faster processing (skips detection)

German TTS

• de_DE-thorsten-medium: Best quality, natural male voice
• de_DE-karlsson-medium: Alternative male voice
• de_DE-eva_k-x_low: Fast, robotic female — only for speed-critical

Mixed Language

• STT auto-detects well with language: ''
• TTS: Use appropriate voice per language (config is global, so pick primary)

---

Advanced: Custom Voice Training (Piper)

For custom voices: https://github.com/rhasspy/piper/blob/master/TRAINING.md

Requires: ~20min clean audio + transcripts, GPU for training (faster).

Not covered here — use pre-trained voices.

---

Cost Summary

Component	Cost
Ollama (LLM)	Free (local)
faster-whisper (STT)	Free (local)
Piper (TTS)	Free (local)
ffmpeg	Free
Telegram Bot API	Free
Total	$0/month

No API keys, no cloud dependencies, works offline.