Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 root
|
83c98ff0b8 | ||
|
root
|
f2fdf95897 | ||
 mehlzoerwerandClaude Haiku 4.5
|
91eeb4834f |
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(gh auth *)",
|
||||
"PowerShell(gh auth *)",
|
||||
"PowerShell(winget install *)",
|
||||
"Bash(gh repo *)",
|
||||
"Bash(git init *)",
|
||||
"Bash(git remote *)",
|
||||
"Bash(git add *)",
|
||||
"Bash(git commit -m 'Initial commit: Clownfischserver v0.5.0 *)",
|
||||
"Bash(git config *)"
|
||||
]
|
||||
}
|
||||
}
|
||||
+10
-4
@@ -2,7 +2,13 @@
|
||||
|
||||
> Diese Datei ist der zentrale Wissensspeicher für das Clownfischserver-Projekt.
|
||||
> Lade sie in jede KI/Chat-Session um sofort produktiv zu sein.
|
||||
> Letzte Aktualisierung: v0.4.13 (18.04.2026)
|
||||
> Letzte Aktualisierung: v0.5.0 (30.05.2026)
|
||||
|
||||
> **⚠️ Bewusst entfernt (nicht wieder einbauen!):** Ein „Vault/MCP-Context"-System
|
||||
> (`_get_vault_context`, `MCP_VAULT_URL`, `MCP_API_KEY`) wurde in v0.5.0 komplett
|
||||
> entfernt. Es war toter Code, nutzte `verify=False` und schickte Chat-Nachrichten an
|
||||
> externe Endpoints – ein klarer DSGVO-Verstoß und Widerspruch zum „local-first"-Prinzip.
|
||||
> Token-Optimierung über Obsidian gehört NICHT in den Bot.
|
||||
|
||||
---
|
||||
|
||||
@@ -80,7 +86,7 @@ clownfischserver/
|
||||
- **Service:** `clownfisch.service` (Deutsch, mit "ch")
|
||||
- **Linux-User:** Beliebig (z.B. `clownfisch`, `clownfish`, `admin`) – wird aus `MGMT_USER` in `.env` gelesen
|
||||
- **Branding:** Immer "Clownfisch" (Deutsch)
|
||||
- **ZIP-Benennung:** `v0.4.13-clownfischserver-opus.zip`
|
||||
- **ZIP-Benennung:** `v0.5.0-clownfischserver.zip`
|
||||
|
||||
### Code-Stil
|
||||
- Python 3, keine Type Hints in Funktionssignaturen (außer Optional)
|
||||
@@ -388,7 +394,7 @@ bash -n install.sh
|
||||
### Schritt 2: ZIP erstellen
|
||||
```bash
|
||||
cd .. # Eine Ebene über clownfischserver/
|
||||
zip -r v0.4.13-clownfischserver-opus.zip clownfischserver/ -x "*__pycache__*" "*.pyc"
|
||||
zip -r v0.5.0-clownfischserver.zip clownfischserver/ -x "*__pycache__*" "*.pyc"
|
||||
```
|
||||
|
||||
**WICHTIG:**
|
||||
@@ -400,7 +406,7 @@ zip -r v0.4.13-clownfischserver-opus.zip clownfischserver/ -x "*__pycache__*" "*
|
||||
|
||||
### Schritt 3: Verifizieren
|
||||
```bash
|
||||
unzip -l v0.4.13-clownfischserver-opus.zip | head -30
|
||||
unzip -l v0.5.0-clownfischserver.zip | head -30
|
||||
# Muss zeigen: clownfischserver/bot/bot.py etc. (20 Dateien)
|
||||
```
|
||||
|
||||
|
||||
@@ -23,6 +23,15 @@ Danke dass du mitmachen willst! Hier sind die Regeln:
|
||||
- Kommentare auf Deutsch oder Englisch
|
||||
- Header in jeder Datei (siehe bestehende Dateien)
|
||||
|
||||
## Code-Hygiene (wichtig!)
|
||||
|
||||
- **Kein toter Code** – keine Funktionen/Module die definiert aber nie aufgerufen werden
|
||||
- **Keine versteckten Cloud-Calls** – jeder externe Request muss klar ersichtlich und opt-in sein
|
||||
- **Niemals `verify=False`** bei HTTPS-Requests
|
||||
- **Keine User-Eingaben an Dritte** ohne explizites Opt-in (DSGVO by design)
|
||||
- **Klare Conditions** – verschachtelte Ternaries (`A if B else C` innerhalb von `and`-Ketten) vermeiden
|
||||
- Selbst-Review vor jedem Release: macht jede Zeile noch das was sie soll?
|
||||
|
||||
## Issues
|
||||
|
||||
- Bug? → Issue mit Log-Output
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
# DSGVO-Konformität – Clownfischserver v0.5.0
|
||||
|
||||
> Dieser Stand bezieht sich auf v0.5.0. Frühere Versionen erfüllen nicht den vollen
|
||||
> Funktionsumfang. v0.6.0 erweitert um Transport-Verschlüsselung und Backup-Konzept.
|
||||
|
||||
Clownfischserver ist als **Gateway für lokal gehostete LLMs in Unternehmensumgebungen**
|
||||
positioniert. Datenschutz ist kein Add-on, sondern strukturelle Voraussetzung.
|
||||
|
||||
Verantwortlich für die DSGVO-konforme Konfiguration im konkreten Einsatz bleibt
|
||||
**der Betreiber**. Dieses Dokument beschreibt, welche technischen Maßnahmen die
|
||||
Software dafür bereitstellt.
|
||||
|
||||
---
|
||||
|
||||
## Art. 5 DSGVO – Grundsätze der Verarbeitung
|
||||
|
||||
### Art. 5 (1) (a) – Rechtmäßigkeit, Verarbeitung nach Treu und Glauben, Transparenz
|
||||
|
||||
Jeder privilegierte Vorgang wird im Audit-Trail (`audit.jsonl`) protokolliert:
|
||||
|
||||
- Wer hat was wann ausgeführt? (`/shell`, `/code`, `/run`, `/ja`, `/ssh`, `/snapshot`, `/rollback`)
|
||||
- Welches LLM-Modell hat eine Antwort generiert? (Audit-Event `llm.route`)
|
||||
- Welche Zugriffe wurden abgelehnt? (`result: denied`)
|
||||
|
||||
Operatoren können den Trail jederzeit via `/logs` einsehen
|
||||
(`bot/audit_query.py`, `cmd_logs` in `bot/bot.py`).
|
||||
|
||||
### Art. 5 (1) (b) – Zweckbindung
|
||||
|
||||
`/shell` und `/code` benötigen eine explizite Bestätigung durch einen Approver
|
||||
(`/ja`). Dieser Approval-Flow ist nicht umgehbar: jeder Befehl wird *vor* der
|
||||
Ausführung als Vorschlag im Chat dargestellt, der ausführende Approver
|
||||
hinterlässt seinerseits einen Audit-Eintrag (`shell.approval`).
|
||||
|
||||
### Art. 5 (1) (c) – Datenminimierung
|
||||
|
||||
- Audit-Einträge enthalten **nur die numerische Telegram User-ID** – keine
|
||||
Klarnamen, keine Telegram-Usernames, keine E-Mail-Adressen.
|
||||
- Befehlstexte werden auf `AUDIT_MAX_DETAIL_LEN` Zeichen (Default 2000) gekürzt.
|
||||
Konfigurierbar via `.env`.
|
||||
- Es findet **keine Telemetrie nach außen** statt. Sämtliche LLM-Calls können
|
||||
konfigurationsabhängig vollständig on-prem gehalten werden
|
||||
(`LLM_PRIMARY_PROVIDER=ollama`, `LLM_FALLBACK_PROVIDER=none`).
|
||||
|
||||
### Art. 5 (1) (d) – Richtigkeit
|
||||
|
||||
Audit-Einträge enthalten Eingaben (`description`, `command`) **und** Ergebnis
|
||||
(`returncode`, ggf. `stderr`). Falschmeldungen lassen sich gegen den realen
|
||||
Serverzustand verifizieren.
|
||||
|
||||
### Art. 5 (1) (e) – Speicherbegrenzung
|
||||
|
||||
`audit_log.prune_old_logs()` läuft beim Bot-Start und entfernt Einträge,
|
||||
die älter als `AUDIT_RETENTION_DAYS` (Default 90 Tage) sind.
|
||||
|
||||
- `AUDIT_RETENTION_DAYS=0` deaktiviert das Pruning bewusst (z. B. für
|
||||
forensische Anforderungen mit längerer Aufbewahrungspflicht).
|
||||
- Das Pruning nutzt einen atomic-rename, sodass auch ein Crash während des
|
||||
Bereinigungslaufs keine Audit-Lücken erzeugt.
|
||||
|
||||
### Art. 5 (1) (f) – Integrität und Vertraulichkeit
|
||||
|
||||
- Audit-Datei wird nach jedem Append auf `0600` gesetzt (nur Eigentümer
|
||||
lesbar/schreibbar). Auf Windows-Filesystemen ohne POSIX-Permissions wird
|
||||
der Schreibvorgang nicht blockiert, sondern lediglich geloggt – Produktion
|
||||
läuft per `install.sh` auf Linux.
|
||||
- Sämtliche LLM-Kommunikation kann ausschließlich lokal stattfinden
|
||||
(Ollama via `127.0.0.1:11434`). OpenRouter-Fallback ist **opt-in**.
|
||||
|
||||
---
|
||||
|
||||
## Art. 32 DSGVO – Sicherheit der Verarbeitung
|
||||
|
||||
### Art. 32 (1) (a) – Pseudonymisierung
|
||||
|
||||
Audit-Einträge speichern numerische User-IDs, keine Namen oder Handles.
|
||||
Die Zuordnung User-ID → Person verbleibt beim Betreiber, außerhalb der Software.
|
||||
|
||||
### Art. 32 (1) (a) – Verschlüsselung
|
||||
|
||||
- **In v0.5.0 nicht enthalten**: Die Audit-Datei liegt unverschlüsselt im
|
||||
Filesystem. Schutz erfolgt ausschließlich über Dateisystem-Permissions
|
||||
(`0600`, `MGMT_USER`).
|
||||
- **Plan für v0.6.0**: HTTPS-only Webhook-Endpoint (Telegram-Bot ist
|
||||
derzeit Long-Polling), at-rest-Encryption der Audit-Datei.
|
||||
|
||||
### Art. 32 (1) (b) – Vertraulichkeit, Integrität, Verfügbarkeit
|
||||
|
||||
- **Rollenbasierter Zugriff** (`bot/auth.py`):
|
||||
- `operator` – darf Befehle vorschlagen, ausführen, Code generieren,
|
||||
SSH-Firewall steuern, Snapshots verwalten, Rollback fahren.
|
||||
- `approver` – darf vorgeschlagene Befehle bestätigen (`/ja`).
|
||||
- `viewer` – darf Status und Snapshots einsehen, im Chat lesen.
|
||||
- Inheritance: operator ⊃ approver ⊃ viewer.
|
||||
- Jeder Endpunkt im Bot prüft die Rolle explizit; ein nicht autorisierter
|
||||
Zugriff wird abgelehnt **und** als `result: denied` im Audit protokolliert.
|
||||
- Verfügbarkeit: Bot läuft als `systemd`-Service (`clownfisch`); Boot-Notify-
|
||||
Service informiert beim Neustart.
|
||||
|
||||
### Art. 32 (1) (c) – Wiederherstellbarkeit
|
||||
|
||||
`bot/snapshot.py` erstellt automatisch Snapshots vor potenziell zerstörerischen
|
||||
Operationen (`/code`, ZIP-Upload, schreibende `/shell`-Befehle). `/rollback`
|
||||
stellt einen Snapshot wieder her – ebenfalls per Operator-Rolle abgesichert.
|
||||
|
||||
### Art. 32 (1) (d) – Verfahren zur Überprüfung
|
||||
|
||||
Tests unter `tests/`:
|
||||
|
||||
- `test_auth.py` – Rollenmodell + Backwards-Compat
|
||||
- `test_audit_log.py` – Schreibverhalten, Pruning, Datenminimierung
|
||||
- `test_audit_query.py` – Filterlogik, Format
|
||||
- `test_llm_router.py` – Routing-Verhalten, Audit-Hooks
|
||||
|
||||
`tests/_smoke.py`, `tests/_smoke_router.py`, `tests/_smoke_query.py` sind
|
||||
standalone-Skripte für Einsatz ohne pytest.
|
||||
|
||||
---
|
||||
|
||||
## Was Clownfischserver **nicht** automatisch erledigt
|
||||
|
||||
Diese Punkte verbleiben beim Betreiber:
|
||||
|
||||
- **Auftragsverarbeitungs-Vertrag** mit OpenRouter (falls Fallback aktiviert).
|
||||
OpenRouter-Server stehen außerhalb der EU – aktiviere den Fallback nicht
|
||||
für Daten mit Personenbezug.
|
||||
- **Verzeichnis von Verarbeitungstätigkeiten** (Art. 30 DSGVO).
|
||||
- **Datenschutz-Folgenabschätzung** bei großvolumiger Personendatenverarbeitung
|
||||
durch das gehostete LLM.
|
||||
- **Backup-Strategie** für die Audit-Datei (in v0.6.0 geplant).
|
||||
- **Schulung der Operatoren** im Umgang mit dem `/shell`-Approval-Flow.
|
||||
|
||||
---
|
||||
|
||||
## Kurz: Welche `.env`-Variablen sind DSGVO-relevant?
|
||||
|
||||
| Variable | Effekt |
|
||||
|---|---|
|
||||
| `CLOWNFISCH_OPERATORS` / `_APPROVERS` / `_VIEWERS` | Rollenzuweisung, Art. 32 (1) (b) |
|
||||
| `AUDIT_LOG_PATH` | Pfad zur Audit-Datei (sollte auf verschlüsseltem Volume liegen) |
|
||||
| `AUDIT_RETENTION_DAYS` | Speicherbegrenzung, Art. 5 (1) (e) |
|
||||
| `AUDIT_MAX_DETAIL_LEN` | Datenminimierung, Art. 5 (1) (c) |
|
||||
| `LLM_PRIMARY_PROVIDER=ollama` | On-prem-LLM, keine Cloud-Übertragung |
|
||||
| `LLM_FALLBACK_PROVIDER=none` | Cloud-Fallback explizit deaktivieren |
|
||||
| `OPENROUTER_API_KEY` | nur setzen wenn AV-Vertrag mit OpenRouter besteht |
|
||||
|
||||
---
|
||||
|
||||
**Stand:** Juni 2026 · v0.5.0 · Mehlzoerwer-Claude
|
||||
@@ -6,7 +6,7 @@ Ein selbst-hostbarer, KI-gestützter Server-Manager der ausschließlich über Te
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
---
|
||||
@@ -45,10 +45,82 @@ Das Ziel: **Server-Administration so einfach wie eine Chat-Nachricht.** Heute ei
|
||||
- 🔔 **Benachrichtigungen** – SSH-Zugang, Server-Boot, Knock-Versuche via Telegram
|
||||
- 📦 **Self-Update via Telegram** – ZIP an den Bot schicken, fertig (inkl. systemd-Services)
|
||||
- ☁️ **OpenRouter Fallback** – kostenlose Cloud-Modelle als Backup wenn Ollama nicht verfügbar
|
||||
- 👥 **Team-Rollen** *(v0.5.0)* – `operator` / `approver` / `viewer` mit Inheritance, env-gesteuert
|
||||
- 📜 **Audit-Trail** *(v0.5.0)* – jeder privilegierte Vorgang in `audit.jsonl`, abfragbar via `/logs`, Retention konfigurierbar
|
||||
- 🔀 **LLM-Router** *(v0.5.0)* – Primary/Fallback-Chain via `LLM_PRIMARY_PROVIDER` etc.; pro Provider eigenes Kontextfenster
|
||||
- 👤 **Eigener User** – beliebiger Name, passwordless sudo, wird überall korrekt eingesetzt
|
||||
- 🌍 **Mehrsprachig** – antwortet in der Sprache in der du schreibst
|
||||
- 🐧 **Multi-Distro** – Debian, Ubuntu, Fedora, Arch (Installer)
|
||||
- 🔒 **DSGVO by design** – lokal ist Standard, Cloud ist Opt-in
|
||||
- 🔒 **DSGVO by design** – lokal ist Standard, Cloud ist Opt-in (siehe [DSGVO.md](DSGVO.md))
|
||||
|
||||
---
|
||||
|
||||
## For Enterprise Teams (v0.5.0)
|
||||
|
||||
Clownfischserver v0.5.0 ist als **Gateway für Teams mit lokal gehosteten
|
||||
LLMs** positioniert. Wenn ihr Claude Opus, Qwen 32B oder Gemma 4 auf eigener
|
||||
Hardware betreibt, gibt euch v0.5.0:
|
||||
|
||||
### Rollenbasierte Zugriffskontrolle
|
||||
|
||||
Drei Rollen mit Inheritance (`operator` ⊃ `approver` ⊃ `viewer`):
|
||||
|
||||
```dotenv
|
||||
# .env
|
||||
CLOWNFISCH_OPERATORS=12345,67890 # dürfen /shell /code /run /ssh /ja /snapshot /rollback
|
||||
CLOWNFISCH_APPROVERS=11111 # dürfen /ja (Bestätigungen)
|
||||
CLOWNFISCH_VIEWERS=22222,33333 # dürfen /status /snapshots /help + Chat
|
||||
```
|
||||
|
||||
Wer keine Rolle hat, wird abgelehnt – inklusive Audit-Eintrag mit
|
||||
`result: denied`. Backwards-kompatibel: ohne Rollen-Vars wird die alte
|
||||
`TELEGRAM_CHAT_ID` automatisch zu `operator`.
|
||||
|
||||
### Audit-Trail nach DSGVO Art. 5 + 32
|
||||
|
||||
Jeder Vorgang wird in `audit.jsonl` als JSONL-Eintrag protokolliert:
|
||||
|
||||
```jsonl
|
||||
{"ts":"2026-06-04T14:21:03+00:00","user_id":"12345","action":"shell.execute","result":"ok","role":"operator","details":{"command":"systemctl restart nginx","returncode":0}}
|
||||
{"ts":"2026-06-04T14:21:07+00:00","action":"llm.route","result":"hit","details":{"tier":"primary","provider":"ollama","model":"qwen2.5-coder:7b","method":"chat"}}
|
||||
```
|
||||
|
||||
Operatoren fragen den Trail direkt aus Telegram ab:
|
||||
|
||||
```text
|
||||
/logs # letzte 20 Einträge
|
||||
/logs user 12345 days 7 # User-Aktivität, letzte 7 Tage
|
||||
/logs prefix shell. result error
|
||||
/logs action llm.route limit 50
|
||||
```
|
||||
|
||||
Retention via `AUDIT_RETENTION_DAYS` (Default 90 Tage, beim Bot-Start automatisch).
|
||||
|
||||
### Konfigurierbares LLM-Routing
|
||||
|
||||
Primary/Fallback-Chain via Env – kein Auto-Selection, keine Halluzinationen:
|
||||
|
||||
```dotenv
|
||||
# Primary: lokal, sensitive Daten bleiben on-prem
|
||||
LLM_PRIMARY_PROVIDER=ollama
|
||||
LLM_PRIMARY_MODEL=qwen2.5-coder:7b
|
||||
LLM_PRIMARY_NUM_CTX=65536 # z. B. Gemma 4 27B braucht 64K+
|
||||
|
||||
# Fallback nur für non-sensitive Tasks
|
||||
LLM_FALLBACK_PROVIDER=openrouter
|
||||
LLM_FALLBACK_MODEL=deepseek/deepseek-r1:free
|
||||
|
||||
# Fallback komplett deaktivieren:
|
||||
# LLM_FALLBACK_PROVIDER=none
|
||||
```
|
||||
|
||||
Routable Errors (Timeout, Connection-Refused) lösen Fallback aus; harte
|
||||
Fehler (JSON-Parse, Bad API Key) propagieren – ohne stille Modellwechsel.
|
||||
|
||||
### Vollständige Compliance-Dokumentation
|
||||
|
||||
Siehe **[DSGVO.md](DSGVO.md)** – aufgeschlüsselt nach Art. 5 + 32 mit
|
||||
Referenzen auf den konkreten Code.
|
||||
|
||||
---
|
||||
|
||||
@@ -235,7 +307,7 @@ Nach jedem Server-Neustart bekommst du eine Telegram-Nachricht mit Status aller
|
||||
|
||||
## Roadmap
|
||||
|
||||
### Fertig (v0.1 – v0.4.13)
|
||||
### Fertig (v0.1 – v0.5.0)
|
||||
|
||||
- [x] Basis-Installer mit geführtem Setup (8 Schritte)
|
||||
- [x] Telegram Bot mit Auth Guard (Single-User)
|
||||
@@ -265,9 +337,16 @@ Nach jedem Server-Neustart bekommst du eine Telegram-Nachricht mit Status aller
|
||||
- [x] DSGVO-Regel: `.env` nur per SSH, nie über Telegram lesbar
|
||||
- [x] Projekt-Kontext-Datei (CLOWNFISCHSERVER.md) für KI-gestütztes Pair-Programming
|
||||
|
||||
### v0.5.0 – Geplant
|
||||
### v0.5.0 – Cleanup & Stabilität (aktuelles Release)
|
||||
|
||||
**Sicherheit & DSGVO:**
|
||||
**Bugfixes & Aufräumarbeiten (dieser Release):**
|
||||
- [x] Totes Vault/MCP-Context-System komplett entfernt (war nie verdrahtet, `verify=False`-Lücke)
|
||||
- [x] Privacy-Fix: Chat-Nachrichten gingen via Vault-Suche an externe Endpoints → entfernt
|
||||
- [x] `/code` Filepath-Logik: fragiler Ternary-Ausdruck sauber umgeschrieben (kein Crash bei leerem Task)
|
||||
- [x] Aider-Model-Default vereinheitlicht (zieht sich `OLLAMA_MODEL` aus der Umgebung)
|
||||
- [x] Irreführende „Token-Optimization"-Logs (Anthropic-Kosten/Obsidian) aus dem Start-Log entfernt
|
||||
|
||||
**Sicherheit & DSGVO (geplant):**
|
||||
- [ ] Sensitive Dateien über Telegram blockieren (`.env`, `shadow`, Keys)
|
||||
- [ ] Output-Sanitizer: API Keys/Tokens in jeder Bot-Antwort maskieren
|
||||
- [ ] Log-Sanitizer: Keine Secrets im Audit-Log
|
||||
|
||||
@@ -17,7 +17,7 @@ logger = logging.getLogger(__name__)
|
||||
INSTALL_DIR = os.getenv("INSTALL_DIR", "/opt/clownfischserver")
|
||||
AIDER_WORKDIR = os.getenv("AIDER_WORKDIR", "/opt/clownfisch-workspace")
|
||||
OLLAMA_BASE_URL = os.getenv("OLLAMA_BASE_URL", "http://localhost:11434")
|
||||
OLLAMA_MODEL = os.getenv("OLLAMA_MODEL", "qwen2.5:2b")
|
||||
OLLAMA_MODEL = os.getenv("OLLAMA_MODEL", "qwen2.5-coder:7b")
|
||||
TIMEOUT = 600 # 10 minutes for aider tasks on CPU
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
🐠 Clownfischserver – Audit Trail (DSGVO Art. 5 + 32)
|
||||
Author: Mehlzoerwer-Claude (https://github.com/mehlzoerwer-claude)
|
||||
License: GPL-3.0
|
||||
|
||||
Writes JSON-line entries for every privileged action.
|
||||
Pseudonymized: stores only numeric Telegram user_ids, never usernames.
|
||||
|
||||
Compliance:
|
||||
- Art. 5(1)(a) lawfulness/transparency: every action is recorded.
|
||||
- Art. 5(1)(c) data minimization: numeric IDs only, command/result trimmed.
|
||||
- Art. 5(1)(e) storage limitation: prune_old_logs() drops entries older
|
||||
than AUDIT_RETENTION_DAYS (default 90).
|
||||
- Art. 32 integrity: append-only file, 0600 permissions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# How much of a command/result we keep – longer payloads are truncated
|
||||
# to honour data minimization. Configurable via env.
|
||||
_MAX_DETAIL_LEN = int(os.getenv("AUDIT_MAX_DETAIL_LEN", "2000"))
|
||||
|
||||
# Lock for append writes – multiple coroutines may log concurrently.
|
||||
_write_lock = threading.Lock()
|
||||
|
||||
|
||||
def _default_audit_path() -> str:
|
||||
install_dir = os.getenv("INSTALL_DIR", "/opt/clownfischserver")
|
||||
return os.getenv("AUDIT_LOG_PATH", os.path.join(install_dir, "audit.jsonl"))
|
||||
|
||||
|
||||
def _retention_days() -> int:
|
||||
try:
|
||||
return max(0, int(os.getenv("AUDIT_RETENTION_DAYS", "90")))
|
||||
except ValueError:
|
||||
return 90
|
||||
|
||||
|
||||
def _trim(value: Any) -> Any:
|
||||
if isinstance(value, str) and len(value) > _MAX_DETAIL_LEN:
|
||||
return value[:_MAX_DETAIL_LEN] + f"... (truncated, {len(value)} chars total)"
|
||||
return value
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
return datetime.now(timezone.utc).isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def log_action(
|
||||
user_id: str | int | None,
|
||||
action: str,
|
||||
*,
|
||||
role: str | None = None,
|
||||
result: str = "ok",
|
||||
details: dict[str, Any] | None = None,
|
||||
path: str | None = None,
|
||||
) -> None:
|
||||
"""Append a single audit record. Never raises – audit failure must not
|
||||
break the bot."""
|
||||
entry: dict[str, Any] = {
|
||||
"ts": _now_iso(),
|
||||
"user_id": str(user_id) if user_id is not None else None,
|
||||
"action": action,
|
||||
"result": result,
|
||||
}
|
||||
if role:
|
||||
entry["role"] = role
|
||||
if details:
|
||||
entry["details"] = {k: _trim(v) for k, v in details.items()}
|
||||
|
||||
target = path or _default_audit_path()
|
||||
try:
|
||||
Path(target).parent.mkdir(parents=True, exist_ok=True)
|
||||
line = json.dumps(entry, ensure_ascii=False) + "\n"
|
||||
with _write_lock:
|
||||
with open(target, "a", encoding="utf-8") as fh:
|
||||
fh.write(line)
|
||||
try:
|
||||
os.chmod(target, 0o600)
|
||||
except OSError:
|
||||
# chmod can fail on non-POSIX FS (e.g. tests on Windows) –
|
||||
# the JSONL content is still written.
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.error(f"Audit log write failed: {e}")
|
||||
|
||||
|
||||
def log_shell(user_id, command: str, *, role: str | None = None,
|
||||
status: str = "executed", returncode: int | None = None,
|
||||
stderr: str | None = None, path: str | None = None) -> None:
|
||||
details: dict[str, Any] = {"command": command}
|
||||
if returncode is not None:
|
||||
details["returncode"] = returncode
|
||||
if stderr:
|
||||
details["stderr"] = stderr
|
||||
log_action(user_id, "shell.execute", role=role, result=status,
|
||||
details=details, path=path)
|
||||
|
||||
|
||||
def log_shell_proposed(user_id, description: str, command: str,
|
||||
role: str | None = None, dangerous: bool = False,
|
||||
path: str | None = None) -> None:
|
||||
log_action(
|
||||
user_id,
|
||||
"shell.proposed",
|
||||
role=role,
|
||||
result="dangerous" if dangerous else "ok",
|
||||
details={"description": description, "command": command},
|
||||
path=path,
|
||||
)
|
||||
|
||||
|
||||
def log_code(user_id, task: str, *, role: str | None = None,
|
||||
status: str = "ok", path: str | None = None) -> None:
|
||||
log_action(user_id, "code.run", role=role, result=status,
|
||||
details={"task": task}, path=path)
|
||||
|
||||
|
||||
def log_approval(user_id, command: str, *, role: str | None = None,
|
||||
approved: bool = True, path: str | None = None) -> None:
|
||||
log_action(
|
||||
user_id,
|
||||
"shell.approval",
|
||||
role=role,
|
||||
result="approved" if approved else "rejected",
|
||||
details={"command": command},
|
||||
path=path,
|
||||
)
|
||||
|
||||
|
||||
def log_denied(user_id, action: str, *, role: str | None = None,
|
||||
reason: str = "insufficient_role",
|
||||
path: str | None = None) -> None:
|
||||
log_action(user_id, action, role=role, result="denied",
|
||||
details={"reason": reason}, path=path)
|
||||
|
||||
|
||||
def prune_old_logs(path: str | None = None,
|
||||
retention_days: int | None = None) -> int:
|
||||
"""Rewrite the audit file, dropping entries older than retention_days.
|
||||
|
||||
Returns the number of dropped lines. retention_days=0 disables pruning.
|
||||
Uses an atomic temp-file + rename so a crash mid-prune cannot corrupt
|
||||
the audit trail.
|
||||
"""
|
||||
target = path or _default_audit_path()
|
||||
days = retention_days if retention_days is not None else _retention_days()
|
||||
if days <= 0 or not os.path.exists(target):
|
||||
return 0
|
||||
|
||||
cutoff = time.time() - days * 86400
|
||||
tmp_path = target + ".prune.tmp"
|
||||
dropped = 0
|
||||
kept = 0
|
||||
|
||||
try:
|
||||
with _write_lock:
|
||||
with open(target, "r", encoding="utf-8") as src, \
|
||||
open(tmp_path, "w", encoding="utf-8") as dst:
|
||||
for line in src:
|
||||
raw = line.strip()
|
||||
if not raw:
|
||||
continue
|
||||
try:
|
||||
record = json.loads(raw)
|
||||
ts = record.get("ts", "")
|
||||
# ISO 8601 – parse with fromisoformat (UTC suffix safe)
|
||||
dt = datetime.fromisoformat(ts.replace("Z", "+00:00"))
|
||||
if dt.timestamp() < cutoff:
|
||||
dropped += 1
|
||||
continue
|
||||
except (ValueError, KeyError):
|
||||
# Malformed line: keep it rather than silently dropping
|
||||
# forensic evidence.
|
||||
pass
|
||||
dst.write(line if line.endswith("\n") else line + "\n")
|
||||
kept += 1
|
||||
os.replace(tmp_path, target)
|
||||
try:
|
||||
os.chmod(target, 0o600)
|
||||
except OSError:
|
||||
pass
|
||||
logger.info(f"Audit prune: kept={kept} dropped={dropped} retention={days}d")
|
||||
except Exception as e:
|
||||
logger.error(f"Audit prune failed: {e}")
|
||||
if os.path.exists(tmp_path):
|
||||
try:
|
||||
os.remove(tmp_path)
|
||||
except OSError:
|
||||
pass
|
||||
return 0
|
||||
|
||||
return dropped
|
||||
@@ -0,0 +1,189 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
🐠 Clownfischserver – Audit Trail Query (Phase 2b)
|
||||
Author: Mehlzoerwer-Claude (https://github.com/mehlzoerwer-claude)
|
||||
License: GPL-3.0
|
||||
|
||||
Read-side companion to audit_log.py. Filters/sorts JSONL audit records for
|
||||
the `/logs` admin command.
|
||||
|
||||
The query layer is intentionally separate from the writer so that:
|
||||
- Tests can exercise filtering without touching real audit data.
|
||||
- The writer stays append-only and minimal.
|
||||
- Future query backends (SQLite, OpenSearch) can swap in without touching
|
||||
the writer.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from typing import Any, Iterable
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _default_audit_path() -> str:
|
||||
install_dir = os.getenv("INSTALL_DIR", "/opt/clownfischserver")
|
||||
return os.getenv("AUDIT_LOG_PATH", os.path.join(install_dir, "audit.jsonl"))
|
||||
|
||||
|
||||
def _parse_ts(record: dict) -> datetime | None:
|
||||
"""Best-effort parse of the record's ts field. Returns None on
|
||||
malformed timestamps (caller decides how to handle)."""
|
||||
ts = record.get("ts")
|
||||
if not isinstance(ts, str):
|
||||
return None
|
||||
try:
|
||||
return datetime.fromisoformat(ts.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def read_all(path: str | None = None) -> list[dict]:
|
||||
"""Load every audit record from the JSONL file. Malformed lines are
|
||||
skipped with a debug log (the writer keeps them for forensic purposes,
|
||||
but the query layer prefers a clean result set)."""
|
||||
target = path or _default_audit_path()
|
||||
if not os.path.exists(target):
|
||||
return []
|
||||
|
||||
out: list[dict] = []
|
||||
try:
|
||||
with open(target, "r", encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
out.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
logger.debug(f"audit_query: skipping malformed line")
|
||||
continue
|
||||
except OSError as e:
|
||||
logger.error(f"audit_query: cannot read {target}: {e}")
|
||||
return []
|
||||
return out
|
||||
|
||||
|
||||
def filter_records(
|
||||
records: Iterable[dict],
|
||||
*,
|
||||
user_id: str | int | None = None,
|
||||
action: str | None = None,
|
||||
action_prefix: str | None = None,
|
||||
result: str | None = None,
|
||||
since: datetime | None = None,
|
||||
until: datetime | None = None,
|
||||
) -> list[dict]:
|
||||
"""Apply filters and return matching records, newest first.
|
||||
|
||||
Filters are AND-combined. None values are ignored. action/result match
|
||||
exactly; action_prefix matches if the record's action starts with it.
|
||||
"""
|
||||
target_user = str(user_id) if user_id is not None else None
|
||||
|
||||
matched: list[dict] = []
|
||||
for rec in records:
|
||||
if target_user is not None and rec.get("user_id") != target_user:
|
||||
continue
|
||||
if action is not None and rec.get("action") != action:
|
||||
continue
|
||||
if action_prefix is not None:
|
||||
rec_action = rec.get("action", "")
|
||||
if not isinstance(rec_action, str) or not rec_action.startswith(action_prefix):
|
||||
continue
|
||||
if result is not None and rec.get("result") != result:
|
||||
continue
|
||||
if since is not None or until is not None:
|
||||
dt = _parse_ts(rec)
|
||||
if dt is None:
|
||||
# Malformed ts – exclude when caller asked for a date window.
|
||||
continue
|
||||
if since is not None and dt < since:
|
||||
continue
|
||||
if until is not None and dt > until:
|
||||
continue
|
||||
matched.append(rec)
|
||||
|
||||
# Newest first – stable sort by ts (records without ts go to the bottom).
|
||||
matched.sort(key=lambda r: r.get("ts", ""), reverse=True)
|
||||
return matched
|
||||
|
||||
|
||||
def query(
|
||||
*,
|
||||
user_id: str | int | None = None,
|
||||
action: str | None = None,
|
||||
action_prefix: str | None = None,
|
||||
result: str | None = None,
|
||||
days: int | None = None,
|
||||
limit: int = 20,
|
||||
path: str | None = None,
|
||||
) -> list[dict]:
|
||||
"""Convenience wrapper: read + filter in one call.
|
||||
|
||||
`days` is a shortcut for `since = now - days`. limit clamps the
|
||||
returned record count (use limit=0 for no limit)."""
|
||||
since = None
|
||||
if days is not None and days > 0:
|
||||
since = datetime.now(timezone.utc) - timedelta(days=days)
|
||||
records = read_all(path=path)
|
||||
matched = filter_records(
|
||||
records,
|
||||
user_id=user_id,
|
||||
action=action,
|
||||
action_prefix=action_prefix,
|
||||
result=result,
|
||||
since=since,
|
||||
)
|
||||
if limit and limit > 0:
|
||||
matched = matched[:limit]
|
||||
return matched
|
||||
|
||||
|
||||
def format_record(rec: dict) -> str:
|
||||
"""Render one audit record as a single compact line for Telegram."""
|
||||
ts = rec.get("ts", "?")
|
||||
# Trim ts down to HH:MM:SS or YYYY-MM-DD HH:MM:SS for readability.
|
||||
if isinstance(ts, str) and len(ts) >= 19:
|
||||
ts_short = ts[:19].replace("T", " ")
|
||||
else:
|
||||
ts_short = str(ts)
|
||||
uid = rec.get("user_id") or "—"
|
||||
action = rec.get("action", "?")
|
||||
result = rec.get("result", "?")
|
||||
role = rec.get("role")
|
||||
role_part = f" [{role}]" if role else ""
|
||||
|
||||
details = rec.get("details") or {}
|
||||
detail_str = ""
|
||||
if isinstance(details, dict):
|
||||
# Pick the single most-informative key per common action.
|
||||
for key in ("command", "task", "reason", "snapshot", "method", "tier"):
|
||||
if key in details:
|
||||
val = str(details[key])
|
||||
if len(val) > 80:
|
||||
val = val[:77] + "..."
|
||||
detail_str = f" {val}"
|
||||
break
|
||||
|
||||
return f"{ts_short} {uid}{role_part} {action}={result}{detail_str}"
|
||||
|
||||
|
||||
def summarize(records: list[dict]) -> dict[str, Any]:
|
||||
"""Aggregate counts useful for the /logs header."""
|
||||
by_action: dict[str, int] = {}
|
||||
by_user: dict[str, int] = {}
|
||||
by_result: dict[str, int] = {}
|
||||
for r in records:
|
||||
act = r.get("action", "?")
|
||||
by_action[act] = by_action.get(act, 0) + 1
|
||||
uid = r.get("user_id") or "—"
|
||||
by_user[uid] = by_user.get(uid, 0) + 1
|
||||
res = r.get("result", "?")
|
||||
by_result[res] = by_result.get(res, 0) + 1
|
||||
return {"total": len(records), "by_action": by_action,
|
||||
"by_user": by_user, "by_result": by_result}
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
🐠 Clownfischserver – Role-Based Access Control
|
||||
Author: Mehlzoerwer-Claude (https://github.com/mehlzoerwer-claude)
|
||||
License: GPL-3.0
|
||||
|
||||
Implements DSGVO Art. 32 access control:
|
||||
- operator: full execution rights (/shell, /code, /run, /ja, /ssh, /snapshot)
|
||||
- approver: can approve pending operations (/ja)
|
||||
- viewer: read-only access (/status, /snapshots, /help)
|
||||
|
||||
Backwards compatible: if no role env vars are set but TELEGRAM_CHAT_ID
|
||||
is configured, that user is auto-granted the operator role.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
ROLE_OPERATOR = "operator"
|
||||
ROLE_APPROVER = "approver"
|
||||
ROLE_VIEWER = "viewer"
|
||||
|
||||
VALID_ROLES = (ROLE_OPERATOR, ROLE_APPROVER, ROLE_VIEWER)
|
||||
|
||||
# Operators implicitly hold approver + viewer rights.
|
||||
# Approvers implicitly hold viewer rights.
|
||||
_ROLE_INHERITANCE: dict[str, tuple[str, ...]] = {
|
||||
ROLE_OPERATOR: (ROLE_OPERATOR, ROLE_APPROVER, ROLE_VIEWER),
|
||||
ROLE_APPROVER: (ROLE_APPROVER, ROLE_VIEWER),
|
||||
ROLE_VIEWER: (ROLE_VIEWER,),
|
||||
}
|
||||
|
||||
|
||||
def _parse_id_list(raw: str | None) -> set[str]:
|
||||
if not raw:
|
||||
return set()
|
||||
return {part.strip() for part in raw.split(",") if part.strip()}
|
||||
|
||||
|
||||
def _load_role_map() -> dict[str, str]:
|
||||
"""Read role config from env vars.
|
||||
|
||||
Returns {user_id_str: highest_role}. If a user appears in multiple
|
||||
role lists, the strongest role wins (operator > approver > viewer).
|
||||
"""
|
||||
operators = _parse_id_list(os.getenv("CLOWNFISCH_OPERATORS"))
|
||||
approvers = _parse_id_list(os.getenv("CLOWNFISCH_APPROVERS"))
|
||||
viewers = _parse_id_list(os.getenv("CLOWNFISCH_VIEWERS"))
|
||||
|
||||
role_map: dict[str, str] = {}
|
||||
|
||||
for uid in viewers:
|
||||
role_map[uid] = ROLE_VIEWER
|
||||
for uid in approvers:
|
||||
role_map[uid] = ROLE_APPROVER
|
||||
for uid in operators:
|
||||
role_map[uid] = ROLE_OPERATOR
|
||||
|
||||
# Backwards compatibility: legacy TELEGRAM_CHAT_ID becomes operator
|
||||
# when no explicit role lists are defined.
|
||||
if not role_map:
|
||||
legacy = os.getenv("TELEGRAM_CHAT_ID", "").strip()
|
||||
if legacy:
|
||||
role_map[legacy] = ROLE_OPERATOR
|
||||
logger.info(
|
||||
"Auth: no role lists configured – granting operator to legacy "
|
||||
"TELEGRAM_CHAT_ID for backwards compatibility."
|
||||
)
|
||||
|
||||
return role_map
|
||||
|
||||
|
||||
def get_role(user_id: str | int | None) -> str | None:
|
||||
"""Return the role of a user, or None if unauthorized."""
|
||||
if user_id is None:
|
||||
return None
|
||||
return _load_role_map().get(str(user_id))
|
||||
|
||||
|
||||
def has_role(user_id: str | int | None, *required: str) -> bool:
|
||||
"""Check if the user satisfies ANY of the required roles.
|
||||
|
||||
Inheritance applies: an operator passes a check for approver/viewer.
|
||||
"""
|
||||
role = get_role(user_id)
|
||||
if role is None:
|
||||
return False
|
||||
granted = _ROLE_INHERITANCE.get(role, (role,))
|
||||
return any(r in granted for r in required)
|
||||
|
||||
|
||||
def is_authorized(user_id: str | int | None) -> bool:
|
||||
"""True if the user has any role at all."""
|
||||
return get_role(user_id) is not None
|
||||
|
||||
|
||||
def list_users_with_role(role: str) -> list[str]:
|
||||
"""Return all user_ids that hold (or inherit) the given role."""
|
||||
if role not in VALID_ROLES:
|
||||
return []
|
||||
return [uid for uid, r in _load_role_map().items()
|
||||
if role in _ROLE_INHERITANCE.get(r, (r,))]
|
||||
|
||||
|
||||
def describe_roles() -> dict[str, list[str]]:
|
||||
"""Snapshot of current role assignments – useful for /status or audits."""
|
||||
role_map = _load_role_map()
|
||||
result: dict[str, list[str]] = {r: [] for r in VALID_ROLES}
|
||||
for uid, role in role_map.items():
|
||||
result[role].append(uid)
|
||||
return result
|
||||
+372
-46
@@ -35,6 +35,10 @@ from openrouter_client import OpenRouterClient
|
||||
from shell import run_shell, execute_command
|
||||
from snapshot import SnapshotManager, should_snapshot
|
||||
from aider_wrapper import AiderWrapper
|
||||
import auth
|
||||
import audit_log
|
||||
import audit_query
|
||||
from llm_router import LLMRouter, OllamaProvider
|
||||
|
||||
# --- Load env ---
|
||||
_install_dir = os.getenv("INSTALL_DIR", "/opt/clownfischserver")
|
||||
@@ -51,15 +55,33 @@ logging.basicConfig(
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
BOT_TOKEN = os.getenv("TELEGRAM_BOT_TOKEN")
|
||||
CHAT_ID = str(os.getenv("TELEGRAM_CHAT_ID"))
|
||||
CHAT_ID = str(os.getenv("TELEGRAM_CHAT_ID") or "")
|
||||
|
||||
if not BOT_TOKEN or not CHAT_ID:
|
||||
logger.error("TELEGRAM_BOT_TOKEN oder TELEGRAM_CHAT_ID fehlt in .env!")
|
||||
# v0.5.0: auth is satisfied if EITHER legacy TELEGRAM_CHAT_ID OR any of
|
||||
# the new role envs is set. The bot only requires BOT_TOKEN to boot.
|
||||
_has_role_config = any(
|
||||
os.getenv(k) for k in
|
||||
("CLOWNFISCH_OPERATORS", "CLOWNFISCH_APPROVERS", "CLOWNFISCH_VIEWERS")
|
||||
)
|
||||
|
||||
if not BOT_TOKEN:
|
||||
logger.error("TELEGRAM_BOT_TOKEN fehlt in .env!")
|
||||
sys.exit(1)
|
||||
if not CHAT_ID and not _has_role_config:
|
||||
logger.error(
|
||||
"Weder TELEGRAM_CHAT_ID noch CLOWNFISCH_OPERATORS/APPROVERS/VIEWERS "
|
||||
"in .env gesetzt – Bot hätte keine autorisierten Nutzer!"
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
# --- Init modules ---
|
||||
# LLMRouter owns the primary/fallback chain. OpenRouterClient is kept as a
|
||||
# thin status proxy for backwards-compatible /start banner ("Fallback aktiv").
|
||||
openrouter = OpenRouterClient()
|
||||
ollama = OllamaClient(openrouter_client=openrouter)
|
||||
llm = LLMRouter()
|
||||
# Legacy alias so any third-party patches/plugins that still refer to `ollama`
|
||||
# keep working. The router exposes the same async surface.
|
||||
ollama = llm
|
||||
snapshots = SnapshotManager()
|
||||
aider = AiderWrapper()
|
||||
|
||||
@@ -67,11 +89,92 @@ aider = AiderWrapper()
|
||||
# AUTH GUARD + HELPERS
|
||||
# =============================================================================
|
||||
|
||||
def _user_id(update: Update) -> str | None:
|
||||
"""Prefer the actual user id over chat id so group chats with multiple
|
||||
members still produce distinct audit records."""
|
||||
if update.effective_user and update.effective_user.id is not None:
|
||||
return str(update.effective_user.id)
|
||||
if update.effective_chat:
|
||||
return str(update.effective_chat.id)
|
||||
return None
|
||||
|
||||
|
||||
def is_authorized(update: Update) -> bool:
|
||||
return str(update.effective_chat.id) == CHAT_ID
|
||||
"""Backwards-compatible auth check. Delegates to auth.is_authorized()
|
||||
which falls back to legacy TELEGRAM_CHAT_ID when no roles are configured."""
|
||||
return auth.is_authorized(_user_id(update))
|
||||
|
||||
|
||||
async def deny(update: Update, action: str, required_role: str):
|
||||
"""Reply with a denial message and write an audit entry."""
|
||||
uid = _user_id(update)
|
||||
role = auth.get_role(uid)
|
||||
audit_log.log_denied(uid, action, role=role,
|
||||
reason=f"requires_{required_role}")
|
||||
if update.message:
|
||||
try:
|
||||
await update.message.reply_text(
|
||||
f"⛔ Nicht erlaubt – `{action}` benötigt Rolle `{required_role}`.",
|
||||
parse_mode="Markdown",
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def require(update: Update, *roles: str) -> bool:
|
||||
"""Returns True if the user holds any of the listed roles.
|
||||
Returns False if unauthorized OR role is too weak. Caller must
|
||||
short-circuit on False (deny() should be awaited separately)."""
|
||||
uid = _user_id(update)
|
||||
if not auth.is_authorized(uid):
|
||||
return False
|
||||
return auth.has_role(uid, *roles)
|
||||
|
||||
|
||||
def sanitize_output(text: str) -> str:
|
||||
"""Mask sensitive data in bot output: API keys, tokens, passwords, secrets."""
|
||||
import re
|
||||
|
||||
# Patterns to mask (key prefix -> replacement)
|
||||
patterns = [
|
||||
# Bearer tokens (in headers and in text) - MUST come before API key patterns
|
||||
(r"Bearer\s+(?:sk-or-v1-[a-zA-Z0-9\-_.]{10,}|sk-[a-zA-Z0-9\-_.]{10,}|[a-zA-Z0-9\-_\.=]{20,})", r"Bearer ***"),
|
||||
# Authorization headers
|
||||
(r"(Authorization:\s*)[a-zA-Z0-9\-_\.=]{20,}", r"\1***"),
|
||||
# OpenRouter keys (full: sk-or-v1-..., display: sk-or-...)
|
||||
(r"(sk-or-v1-[a-zA-Z0-9\-_]{10,})", r"sk-or-v1-***"),
|
||||
(r"(sk-or-\.{3}[a-zA-Z0-9]{4,})", r"sk-or-***"),
|
||||
# Generic API keys (sk-..., pk-...)
|
||||
(r"\b(sk-[a-zA-Z0-9\-_]{20,})\b", r"sk-***"),
|
||||
(r"\b(pk-[a-zA-Z0-9\-_]{20,})\b", r"pk-***"),
|
||||
# Generic long base64-like secrets (32+ chars)
|
||||
(r"\b([A-Za-z0-9+/]{32,}={0,2})\b", r"***"),
|
||||
# Passwords in URLs
|
||||
(r"(://)([^:@/]+):([^:@/]+)@", r"\1\2:***@"),
|
||||
# Environment variable assignments with secrets (env-style: VAR=value, VAR: value)
|
||||
(r"(?m)^\s*(?:PASSWORD|SECRET|TOKEN|KEY|API_KEY)\s*[=:]?\s*\S+", "***", re.IGNORECASE),
|
||||
# Password/passwort in text with assignment (password: value, password=value)
|
||||
(r"(?i)(password|passwort|pwd)\s*[:=]\s*\S+", "***"),
|
||||
# SSH private key markers
|
||||
(r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----", "***PRIVATE KEY***"),
|
||||
]
|
||||
|
||||
result = text
|
||||
for pattern_tuple in patterns:
|
||||
if len(pattern_tuple) == 3:
|
||||
pattern, repl, flags = pattern_tuple
|
||||
result = re.sub(pattern, repl, result, flags=flags)
|
||||
else:
|
||||
pattern, repl = pattern_tuple
|
||||
result = re.sub(pattern, repl, result)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
async def safe_reply(target, text: str, parse_mode: str = "Markdown", edit: bool = False):
|
||||
"""Send or edit a message. Falls back to plain text if Markdown fails."""
|
||||
# Sanitize output to prevent leaking secrets
|
||||
text = sanitize_output(text)
|
||||
try:
|
||||
if edit:
|
||||
return await target.edit_text(text, parse_mode=parse_mode)
|
||||
@@ -90,15 +193,24 @@ async def safe_reply(target, text: str, parse_mode: str = "Markdown", edit: bool
|
||||
# =============================================================================
|
||||
|
||||
async def cmd_start(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
if not is_authorized(update):
|
||||
if not require(update, auth.ROLE_VIEWER):
|
||||
await deny(update, "/start", auth.ROLE_VIEWER)
|
||||
return
|
||||
|
||||
desc = llm.describe()
|
||||
primary_info = ""
|
||||
if desc["primary"]:
|
||||
primary_info = f"\n🧠 Primary: `{desc['primary']['provider']}` ({desc['primary']['model']})"
|
||||
fallback_info = ""
|
||||
if openrouter.is_available():
|
||||
fallback_info = "\n✅ OpenRouter Fallback aktiv"
|
||||
if desc["fallback"]:
|
||||
fallback_info = f"\n🔄 Fallback: `{desc['fallback']['provider']}` ({desc['fallback']['model']})"
|
||||
|
||||
role = auth.get_role(_user_id(update)) or "—"
|
||||
audit_log.log_action(_user_id(update), "bot.start", role=role)
|
||||
|
||||
await update.message.reply_text(
|
||||
f"🐠 *Clownfischserver v0.4.13 online.*{fallback_info}\n\n"
|
||||
f"🐠 *Clownfischserver v0.5.1 online.*{primary_info}{fallback_info}\n"
|
||||
f"👤 Deine Rolle: `{role}`\n\n"
|
||||
"Schreib mir einfach – ich antworte als Chat.\n\n"
|
||||
"*Befehle:*\n"
|
||||
"• `/shell <beschreibung>` – KI generiert Befehl, du bestätigst\n"
|
||||
@@ -111,13 +223,17 @@ async def cmd_start(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
"• `/snapshot [label]` – manuell Snapshot erstellen\n"
|
||||
"• `/ssh open [ip]` – SSH öffnen\n"
|
||||
"• `/ssh close [ip]` – SSH schließen\n"
|
||||
"• `/logs [filter]` – Audit-Trail abfragen (operator)\n"
|
||||
"• `/help` – diese Hilfe",
|
||||
parse_mode="Markdown"
|
||||
)
|
||||
|
||||
async def cmd_status(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
if not is_authorized(update):
|
||||
if not require(update, auth.ROLE_VIEWER):
|
||||
await deny(update, "/status", auth.ROLE_VIEWER)
|
||||
return
|
||||
audit_log.log_action(_user_id(update), "status.read",
|
||||
role=auth.get_role(_user_id(update)))
|
||||
cmd = (
|
||||
"echo '=== UPTIME ===' && uptime && "
|
||||
"echo '=== MEMORY ===' && free -h && "
|
||||
@@ -131,8 +247,11 @@ async def cmd_status(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
await safe_reply(update.message, f"📊 Systemstatus\n\n```\n{result}\n```")
|
||||
|
||||
async def cmd_snapshots(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
if not is_authorized(update):
|
||||
if not require(update, auth.ROLE_VIEWER):
|
||||
await deny(update, "/snapshots", auth.ROLE_VIEWER)
|
||||
return
|
||||
audit_log.log_action(_user_id(update), "snapshots.list",
|
||||
role=auth.get_role(_user_id(update)))
|
||||
snap_list = snapshots.list_snapshots()
|
||||
if not snap_list:
|
||||
await update.message.reply_text("📸 Keine Snapshots vorhanden.")
|
||||
@@ -147,7 +266,8 @@ async def cmd_snapshots(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
await update.message.reply_text(msg, parse_mode="Markdown")
|
||||
|
||||
async def cmd_rollback(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
if not is_authorized(update):
|
||||
if not require(update, auth.ROLE_OPERATOR):
|
||||
await deny(update, "/rollback", auth.ROLE_OPERATOR)
|
||||
return
|
||||
if not context.args:
|
||||
await update.message.reply_text(
|
||||
@@ -156,18 +276,37 @@ async def cmd_rollback(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
)
|
||||
return
|
||||
snap_name = context.args[0]
|
||||
audit_log.log_action(
|
||||
_user_id(update), "snapshot.rollback",
|
||||
role=auth.get_role(_user_id(update)),
|
||||
details={"snapshot": snap_name},
|
||||
)
|
||||
await update.message.reply_text(f"⏪ Rollback zu `{snap_name}` wird eingeleitet...", parse_mode="Markdown")
|
||||
result = snapshots.rollback(snap_name)
|
||||
await update.message.reply_text(result)
|
||||
|
||||
async def cmd_snapshot(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
"""Snapshot management: create, keep, delete"""
|
||||
if not is_authorized(update):
|
||||
return
|
||||
|
||||
args = context.args
|
||||
subcmd = args[0].lower() if args else "create"
|
||||
|
||||
# Read-only subcommands are viewer-accessible; everything else mutates
|
||||
# and requires operator.
|
||||
if subcmd in ("list", "ls"):
|
||||
required = auth.ROLE_VIEWER
|
||||
else:
|
||||
required = auth.ROLE_OPERATOR
|
||||
|
||||
if not require(update, required):
|
||||
await deny(update, f"/snapshot {subcmd}", required)
|
||||
return
|
||||
|
||||
audit_log.log_action(
|
||||
_user_id(update), f"snapshot.{subcmd}",
|
||||
role=auth.get_role(_user_id(update)),
|
||||
details={"args": args[1:] if len(args) > 1 else []},
|
||||
)
|
||||
|
||||
if subcmd == "keep" and len(args) > 1:
|
||||
result = snapshots.keep_snapshot(args[1])
|
||||
await update.message.reply_text(result)
|
||||
@@ -194,13 +333,106 @@ async def cmd_snapshot(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
await update.message.reply_text("❌ Snapshot fehlgeschlagen – check Logs.")
|
||||
|
||||
async def cmd_help(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
if not is_authorized(update):
|
||||
if not require(update, auth.ROLE_VIEWER):
|
||||
await deny(update, "/help", auth.ROLE_VIEWER)
|
||||
return
|
||||
await cmd_start(update, context)
|
||||
|
||||
async def cmd_logs(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
"""Operator-only audit query.
|
||||
|
||||
Usage:
|
||||
/logs → last 20 entries
|
||||
/logs user 12345 → filter by user
|
||||
/logs action shell.execute → exact action
|
||||
/logs prefix shell. → action prefix
|
||||
/logs days 7 → last 7 days only
|
||||
/logs result denied → filter by result
|
||||
/logs limit 50 → up to 50 entries
|
||||
Filters can be combined: /logs user 12345 days 1 limit 10
|
||||
"""
|
||||
if not require(update, auth.ROLE_OPERATOR):
|
||||
await deny(update, "/logs", auth.ROLE_OPERATOR)
|
||||
return
|
||||
|
||||
args = list(context.args) if context.args else []
|
||||
filters: dict[str, object] = {"limit": 20}
|
||||
i = 0
|
||||
while i < len(args) - 1:
|
||||
key = args[i].lower()
|
||||
val = args[i + 1]
|
||||
if key == "user":
|
||||
filters["user_id"] = val
|
||||
elif key == "action":
|
||||
filters["action"] = val
|
||||
elif key == "prefix":
|
||||
filters["action_prefix"] = val
|
||||
elif key == "result":
|
||||
filters["result"] = val
|
||||
elif key == "days":
|
||||
try:
|
||||
filters["days"] = int(val)
|
||||
except ValueError:
|
||||
await update.message.reply_text(f"❌ `days` muss eine Zahl sein, war `{val}`",
|
||||
parse_mode="Markdown")
|
||||
return
|
||||
elif key == "limit":
|
||||
try:
|
||||
filters["limit"] = max(1, min(200, int(val)))
|
||||
except ValueError:
|
||||
await update.message.reply_text(f"❌ `limit` muss eine Zahl sein, war `{val}`",
|
||||
parse_mode="Markdown")
|
||||
return
|
||||
else:
|
||||
await update.message.reply_text(
|
||||
f"❓ Unbekannter Filter `{key}`. Erlaubt: user, action, prefix, result, days, limit",
|
||||
parse_mode="Markdown")
|
||||
return
|
||||
i += 2
|
||||
|
||||
audit_log.log_action(
|
||||
_user_id(update), "logs.query",
|
||||
role=auth.get_role(_user_id(update)),
|
||||
details={"filters": filters},
|
||||
)
|
||||
|
||||
try:
|
||||
records = audit_query.query(**filters)
|
||||
except Exception as e:
|
||||
logger.error(f"audit query failed: {e}", exc_info=True)
|
||||
await update.message.reply_text(f"❌ Audit-Query fehlgeschlagen: `{e}`",
|
||||
parse_mode="Markdown")
|
||||
return
|
||||
|
||||
if not records:
|
||||
await update.message.reply_text("📭 Keine passenden Audit-Einträge.")
|
||||
return
|
||||
|
||||
summary = audit_query.summarize(records)
|
||||
header = f"📜 *{summary['total']} Einträge*"
|
||||
if summary["by_result"]:
|
||||
header += " · " + ", ".join(
|
||||
f"{k}={v}" for k, v in sorted(summary["by_result"].items())
|
||||
)
|
||||
|
||||
lines = [audit_query.format_record(r) for r in records]
|
||||
body = "\n".join(lines)
|
||||
# Telegram limit ~4096; chunk if necessary.
|
||||
msg = f"{header}\n```\n{body}\n```"
|
||||
if len(msg) > 4000:
|
||||
# Drop the code-fence wrapping for large results and split.
|
||||
await safe_reply(update.message, header)
|
||||
chunk_size = 3800
|
||||
for i in range(0, len(body), chunk_size):
|
||||
chunk = body[i:i + chunk_size]
|
||||
await safe_reply(update.message, f"```\n{chunk}\n```")
|
||||
else:
|
||||
await safe_reply(update.message, msg)
|
||||
|
||||
async def cmd_run(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
"""Direct shell execution – bypasses Ollama completely"""
|
||||
if not is_authorized(update):
|
||||
if not require(update, auth.ROLE_OPERATOR):
|
||||
await deny(update, "/run", auth.ROLE_OPERATOR)
|
||||
return
|
||||
if not context.args:
|
||||
await update.message.reply_text(
|
||||
@@ -209,8 +441,13 @@ async def cmd_run(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
)
|
||||
return
|
||||
cmd = " ".join(context.args)
|
||||
uid = _user_id(update)
|
||||
role = auth.get_role(uid)
|
||||
await update.message.reply_text(f"⚡ Direkt: `{cmd}`", parse_mode="Markdown")
|
||||
stdout, stderr, returncode = await run_shell(cmd)
|
||||
audit_log.log_shell(uid, cmd, role=role,
|
||||
status="ok" if returncode == 0 else "error",
|
||||
returncode=returncode, stderr=stderr or None)
|
||||
result = stdout or stderr or "✓ Fertig (keine Ausgabe)"
|
||||
if len(result) > 4000:
|
||||
result = result[:4000] + "\n... (gekürzt)"
|
||||
@@ -218,10 +455,16 @@ async def cmd_run(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
|
||||
async def cmd_ssh(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
"""Open or close SSH via ufw"""
|
||||
if not is_authorized(update):
|
||||
if not require(update, auth.ROLE_OPERATOR):
|
||||
await deny(update, "/ssh", auth.ROLE_OPERATOR)
|
||||
return
|
||||
action = context.args[0].lower() if context.args else ""
|
||||
client_ip = context.args[1] if len(context.args) > 1 else None
|
||||
audit_log.log_action(
|
||||
_user_id(update), "ssh.toggle",
|
||||
role=auth.get_role(_user_id(update)),
|
||||
details={"action": action, "client_ip": client_ip},
|
||||
)
|
||||
|
||||
if action == "open":
|
||||
if client_ip:
|
||||
@@ -254,7 +497,8 @@ async def cmd_ssh(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
# =============================================================================
|
||||
|
||||
async def cmd_shell(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
if not is_authorized(update):
|
||||
if not require(update, auth.ROLE_OPERATOR):
|
||||
await deny(update, "/shell", auth.ROLE_OPERATOR)
|
||||
return
|
||||
if not context.args:
|
||||
await update.message.reply_text(
|
||||
@@ -264,7 +508,9 @@ async def cmd_shell(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
)
|
||||
return
|
||||
|
||||
if (ollama_ready is None or not ollama_ready.is_set()) and not openrouter.is_available():
|
||||
primary_is_ollama = isinstance(llm.primary, OllamaProvider)
|
||||
ollama_warming = ollama_ready is None or not ollama_ready.is_set()
|
||||
if primary_is_ollama and ollama_warming and not llm.has_fallback():
|
||||
await update.message.reply_text("⏳ Ollama lädt noch – bitte kurz warten.")
|
||||
return
|
||||
|
||||
@@ -284,7 +530,12 @@ async def cmd_shell(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
await safe_reply(thinking_msg, "❌ Konnte keinen Befehl generieren. Versuch es anders zu formulieren.", edit=True)
|
||||
return
|
||||
|
||||
uid = _user_id(update)
|
||||
role = auth.get_role(uid)
|
||||
|
||||
if result.get("dangerous", False):
|
||||
audit_log.log_shell_proposed(uid, description, command,
|
||||
role=role, dangerous=True)
|
||||
await safe_reply(thinking_msg,
|
||||
f"⚠️ *Gefährlicher Befehl verweigert!*\n\n"
|
||||
f"Befehl: `{command}`\n"
|
||||
@@ -293,6 +544,7 @@ async def cmd_shell(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
)
|
||||
return
|
||||
|
||||
audit_log.log_shell_proposed(uid, description, command, role=role)
|
||||
context.user_data["pending_cmd"] = command
|
||||
await safe_reply(thinking_msg,
|
||||
f"🔧 Vorgeschlagener Befehl:\n\n"
|
||||
@@ -309,7 +561,9 @@ async def cmd_shell(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
# =============================================================================
|
||||
|
||||
async def cmd_ja(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
if not is_authorized(update):
|
||||
# /ja is the approval gate – approver inherits to operator.
|
||||
if not require(update, auth.ROLE_APPROVER):
|
||||
await deny(update, "/ja", auth.ROLE_APPROVER)
|
||||
return
|
||||
|
||||
pending = context.user_data.get("pending_cmd")
|
||||
@@ -319,6 +573,9 @@ async def cmd_ja(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
|
||||
command = pending
|
||||
context.user_data.pop("pending_cmd", None)
|
||||
uid = _user_id(update)
|
||||
role = auth.get_role(uid)
|
||||
audit_log.log_approval(uid, command, role=role, approved=True)
|
||||
|
||||
snap_name = None
|
||||
if should_snapshot(command):
|
||||
@@ -331,6 +588,7 @@ async def cmd_ja(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
|
||||
try:
|
||||
result = await execute_command(command, ollama)
|
||||
audit_log.log_shell(uid, command, role=role, status="ok")
|
||||
if len(result) > 4000:
|
||||
chunks = [result[i:i+4000] for i in range(0, len(result), 4000)]
|
||||
await safe_reply(status_msg, f"✅ Ergebnis (Teil 1/{len(chunks)}):\n\n```\n{chunks[0]}\n```", edit=True)
|
||||
@@ -340,6 +598,8 @@ async def cmd_ja(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
await safe_reply(status_msg, f"✅ Ergebnis:\n\n```\n{result}\n```", edit=True)
|
||||
except Exception as e:
|
||||
logger.error(f"cmd_ja execution failed: {e}", exc_info=True)
|
||||
audit_log.log_shell(uid, command, role=role, status="error",
|
||||
stderr=str(e))
|
||||
await safe_reply(status_msg,
|
||||
f"❌ Fehler:\n`{e}`\n\nSnapshot: `{snap_name or 'keiner'}`",
|
||||
edit=True
|
||||
@@ -350,7 +610,8 @@ async def cmd_ja(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
# =============================================================================
|
||||
|
||||
async def cmd_code(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
if not is_authorized(update):
|
||||
if not require(update, auth.ROLE_OPERATOR):
|
||||
await deny(update, "/code", auth.ROLE_OPERATOR)
|
||||
return
|
||||
if not context.args:
|
||||
await update.message.reply_text(
|
||||
@@ -367,7 +628,16 @@ async def cmd_code(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
workspace = os.getenv("AIDER_WORKDIR", "/opt/clownfisch-workspace")
|
||||
task_lower = task.lower()
|
||||
|
||||
if workspace not in task and "/opt/" not in task and "/" not in task.split()[0] if task.split() else True:
|
||||
# Only auto-generate a filepath if the task doesn't already point at one.
|
||||
words = task.split()
|
||||
first_word_is_path = bool(words) and "/" in words[0]
|
||||
needs_filepath = (
|
||||
workspace not in task
|
||||
and "/opt/" not in task
|
||||
and not first_word_is_path
|
||||
)
|
||||
|
||||
if needs_filepath:
|
||||
file_match = re.search(r'(\S+\.(sh|py|js|ts|html|css|yaml|yml|json|conf|cfg|txt))', task)
|
||||
if file_match:
|
||||
filename = file_match.group(1)
|
||||
@@ -387,13 +657,18 @@ async def cmd_code(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
if snap_name:
|
||||
await update.message.reply_text(f"📸 Snapshot: `{snap_name}`", parse_mode="Markdown")
|
||||
|
||||
uid = _user_id(update)
|
||||
role = auth.get_role(uid)
|
||||
|
||||
try:
|
||||
result = await aider.run(task)
|
||||
audit_log.log_code(uid, task, role=role, status="ok")
|
||||
if len(result) > 4000:
|
||||
result = result[:4000] + "\n... (gekürzt)"
|
||||
await safe_reply(status_msg, f"🛠️ Aider Ergebnis:\n\n```\n{result}\n```", edit=True)
|
||||
except Exception as e:
|
||||
logger.error(f"cmd_code failed: {e}", exc_info=True)
|
||||
audit_log.log_code(uid, task, role=role, status="error")
|
||||
await safe_reply(status_msg,
|
||||
f"❌ Aider Fehler:\n`{e}`\n\nSnapshot: `{snap_name or 'keiner'}`",
|
||||
edit=True
|
||||
@@ -405,7 +680,8 @@ async def cmd_code(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
|
||||
async def handle_file(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
"""Handle file uploads – ZIP = self-update"""
|
||||
if not is_authorized(update):
|
||||
if not require(update, auth.ROLE_OPERATOR):
|
||||
await deny(update, "file.upload", auth.ROLE_OPERATOR)
|
||||
return
|
||||
|
||||
doc = update.message.document
|
||||
@@ -440,10 +716,29 @@ async def handle_file(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
await tg_file.download_to_drive(zip_path)
|
||||
|
||||
# Extract
|
||||
# PowerShell's Compress-Archive writes path entries with backslashes,
|
||||
# which Python's zipfile does NOT split on under POSIX – without this
|
||||
# normalization the archive extracts as flat files named
|
||||
# "clownfischserver\bot\bot.py", and os.walk never finds bot.py.
|
||||
await status_msg.edit_text("📂 Entpacke ZIP...")
|
||||
extract_dir = os.path.join(tmp_dir, "extracted")
|
||||
os.makedirs(extract_dir, exist_ok=True)
|
||||
with zipfile.ZipFile(zip_path, "r") as z:
|
||||
z.extractall(extract_dir)
|
||||
for info in z.infolist():
|
||||
normalized = info.filename.replace("\\", "/")
|
||||
if not normalized or normalized.endswith("/"):
|
||||
if normalized:
|
||||
os.makedirs(os.path.join(extract_dir, normalized.rstrip("/")),
|
||||
exist_ok=True)
|
||||
continue
|
||||
target_path = os.path.join(extract_dir, *normalized.split("/"))
|
||||
# Zip-Slip guard: refuse entries that escape extract_dir.
|
||||
if not os.path.abspath(target_path).startswith(
|
||||
os.path.abspath(extract_dir) + os.sep):
|
||||
raise ValueError(f"Unsicherer Pfad im ZIP: {info.filename}")
|
||||
os.makedirs(os.path.dirname(target_path), exist_ok=True)
|
||||
with z.open(info) as src, open(target_path, "wb") as dst:
|
||||
shutil.copyfileobj(src, dst)
|
||||
|
||||
# Find bot/ directory
|
||||
bot_dir = None
|
||||
@@ -583,8 +878,10 @@ async def _perform_update(update, context, tmp_dir, bot_dir, extract_dir, openro
|
||||
|
||||
async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
"""Normal messages: either handle update dialog or go to Ollama chat."""
|
||||
if not is_authorized(update):
|
||||
logger.warning(f"Unauthorized access attempt from chat_id: {update.effective_chat.id}")
|
||||
uid = _user_id(update)
|
||||
if not require(update, auth.ROLE_VIEWER):
|
||||
logger.warning(f"Unauthorized access attempt from user_id: {uid}")
|
||||
audit_log.log_denied(uid, "chat.message", reason="unauthorized")
|
||||
return
|
||||
|
||||
user_message = update.message.text.strip()
|
||||
@@ -598,7 +895,9 @@ async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
# --- Normal chat ---
|
||||
logger.info(f"Chat message: {user_message[:100]}")
|
||||
|
||||
if (ollama_ready is None or not ollama_ready.is_set()) and not openrouter.is_available():
|
||||
primary_is_ollama = isinstance(llm.primary, OllamaProvider)
|
||||
ollama_warming = ollama_ready is None or not ollama_ready.is_set()
|
||||
if primary_is_ollama and ollama_warming and not llm.has_fallback():
|
||||
await update.message.reply_text(
|
||||
"⏳ Ollama lädt noch...\n\n"
|
||||
"Direkte Befehle funktionieren sofort:\n"
|
||||
@@ -615,6 +914,9 @@ async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
history = context.user_data.get("chat_history", [])
|
||||
result = await ollama.chat(user_message, history=history)
|
||||
|
||||
# Sanitize LLM response before sending
|
||||
result = sanitize_output(result)
|
||||
|
||||
history.append({"role": "user", "content": user_message})
|
||||
history.append({"role": "assistant", "content": result})
|
||||
if len(history) > 20:
|
||||
@@ -631,7 +933,8 @@ async def handle_message(update: Update, context: ContextTypes.DEFAULT_TYPE):
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Chat error: {e}", exc_info=True)
|
||||
await thinking_msg.edit_text(f"❌ Chat-Fehler: `{e}`", parse_mode="Markdown")
|
||||
error_msg = sanitize_output(f"❌ Chat-Fehler: `{e}`")
|
||||
await thinking_msg.edit_text(error_msg, parse_mode="Markdown")
|
||||
|
||||
|
||||
async def _handle_update_dialog(update, context, user_input, step):
|
||||
@@ -679,15 +982,16 @@ async def _handle_update_dialog(update, context, user_input, step):
|
||||
# =============================================================================
|
||||
|
||||
def main():
|
||||
logger.info("🐠 Clownfischserver v0.4.13 Bot startet...")
|
||||
logger.info("🐠 Clownfischserver v0.5.1 Bot startet...")
|
||||
|
||||
import requests as req
|
||||
import time
|
||||
import threading
|
||||
|
||||
model = os.getenv("OLLAMA_MODEL", "qwen2.5-coder:7b")
|
||||
model_fast = os.getenv("OLLAMA_MODEL_FAST", "")
|
||||
base_url = os.getenv("OLLAMA_BASE_URL", "http://localhost:11434")
|
||||
# Pull Ollama models from the router – respects LLM_*_MODEL overrides
|
||||
# and falls back to OLLAMA_MODEL/OLLAMA_MODEL_FAST for backwards compat.
|
||||
base_url = llm.ollama_base_url
|
||||
ollama_models = llm.ollama_models
|
||||
|
||||
global ollama_ready
|
||||
ollama_ready = threading.Event()
|
||||
@@ -711,23 +1015,36 @@ def main():
|
||||
return False
|
||||
|
||||
def warmup_ollama():
|
||||
logger.info("Ollama Warmup gestartet (Hintergrund)...")
|
||||
models_to_warm = [(model, "Hauptmodell")]
|
||||
if model_fast and model_fast != model:
|
||||
models_to_warm.insert(0, (model_fast, "Fast-Modell"))
|
||||
logger.info(f"Dual-Model: fast={model_fast}, main={model}")
|
||||
for m, label in models_to_warm:
|
||||
if not ollama_models:
|
||||
logger.info("Kein Ollama-Provider konfiguriert – Warmup übersprungen.")
|
||||
ollama_ready.set()
|
||||
return
|
||||
logger.info(f"Ollama Warmup gestartet (Hintergrund): {ollama_models}")
|
||||
for idx, m in enumerate(ollama_models):
|
||||
label = "Fast-Modell" if idx == 0 and len(ollama_models) > 1 else "Hauptmodell"
|
||||
warmup_model(m, label)
|
||||
ollama_ready.set()
|
||||
|
||||
threading.Thread(target=warmup_ollama, daemon=True).start()
|
||||
if model_fast and model_fast != model:
|
||||
logger.info(f"🐠 Bot startet – 2 Modelle laden ({model_fast} + {model})")
|
||||
if ollama_models:
|
||||
logger.info(f"🐠 Bot startet – Ollama warmup für {len(ollama_models)} Modell(e)")
|
||||
else:
|
||||
logger.info("🐠 Bot startet – Ollama lädt im Hintergrund.")
|
||||
logger.info("🐠 Bot startet – kein Ollama-Provider, kein Warmup")
|
||||
ollama_ready.set()
|
||||
|
||||
if openrouter.is_available():
|
||||
logger.info("✅ OpenRouter Fallback aktiv")
|
||||
route_desc = llm.describe()
|
||||
logger.info(f"LLM Routing: {route_desc}")
|
||||
|
||||
# Audit-Trail Setup: prune old logs on boot, log the role roster.
|
||||
dropped = audit_log.prune_old_logs()
|
||||
if dropped:
|
||||
logger.info(f"Audit: {dropped} alte Einträge entfernt (Retention)")
|
||||
roster = auth.describe_roles()
|
||||
logger.info(
|
||||
f"Rollen: operators={len(roster['operator'])} "
|
||||
f"approvers={len(roster['approver'])} viewers={len(roster['viewer'])}"
|
||||
)
|
||||
audit_log.log_action(None, "bot.boot", details={"roster": roster})
|
||||
|
||||
app = Application.builder().token(BOT_TOKEN).build()
|
||||
|
||||
@@ -743,13 +1060,22 @@ def main():
|
||||
app.add_handler(CommandHandler("shell", cmd_shell))
|
||||
app.add_handler(CommandHandler("ja", cmd_ja))
|
||||
app.add_handler(CommandHandler("code", cmd_code))
|
||||
app.add_handler(CommandHandler("logs", cmd_logs))
|
||||
|
||||
# Messages – single handler for chat AND update dialog (no double-processing)
|
||||
app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, handle_message))
|
||||
app.add_handler(MessageHandler(filters.Document.ALL, handle_file))
|
||||
|
||||
logger.info(f"Bot läuft. Authorized Chat-ID: {CHAT_ID}")
|
||||
app.run_polling(allowed_updates=Update.ALL_TYPES)
|
||||
legacy_info = f" legacy_chat={CHAT_ID}" if CHAT_ID else ""
|
||||
logger.info(f"Bot läuft. Rollen geladen.{legacy_info}")
|
||||
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
loop.run_until_complete(app.initialize())
|
||||
app.run_polling(allowed_updates=Update.ALL_TYPES)
|
||||
finally:
|
||||
loop.run_until_complete(app.shutdown())
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@@ -0,0 +1,322 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
🐠 Clownfischserver – LLM Provider Router (Phase 3)
|
||||
Author: Mehlzoerwer-Claude (https://github.com/mehlzoerwer-claude)
|
||||
License: GPL-3.0
|
||||
|
||||
Replaces the previously hard-wired Ollama+OpenRouter fallback logic with a
|
||||
config-driven primary/fallback chain.
|
||||
|
||||
Env vars:
|
||||
LLM_PRIMARY_PROVIDER = "ollama" | "openrouter" (default: ollama)
|
||||
LLM_PRIMARY_MODEL = model id for the primary provider
|
||||
LLM_PRIMARY_MODEL_FAST = optional faster model for chat replies (ollama only)
|
||||
LLM_PRIMARY_NUM_CTX = context window override (ollama)
|
||||
LLM_FALLBACK_PROVIDER = "openrouter" | "ollama" | "" ("" disables fallback)
|
||||
LLM_FALLBACK_MODEL = model id for fallback provider
|
||||
TRACK_LLM_COSTS = "true"|"false" – currently informational only
|
||||
|
||||
Backwards compatibility:
|
||||
- If LLM_*_PROVIDER vars are absent we still default to Ollama primary and,
|
||||
when OPENROUTER_API_KEY exists, OpenRouter as fallback. Existing
|
||||
OLLAMA_MODEL / OLLAMA_MODEL_FAST / OLLAMA_NUM_CTX / OPENROUTER_API_KEY
|
||||
env vars are honoured.
|
||||
|
||||
Routing policy (intentionally explicit, not "self-improving"):
|
||||
- Primary is tried first. Only routable errors (network, timeout, provider
|
||||
unavailable) trigger fallback. Hard errors (invalid JSON, malformed prompt,
|
||||
bad API key) propagate so callers see them.
|
||||
- Every routing decision is written to the audit trail as `llm.route` so
|
||||
admins can see which model produced which output (DSGVO Art. 5(1)(a)
|
||||
transparency).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Any, Awaitable, Callable, Optional
|
||||
|
||||
from ollama_client import OllamaClient
|
||||
from openrouter_client import OpenRouterClient
|
||||
|
||||
try: # audit is optional – router must work even if audit module is missing
|
||||
import audit_log
|
||||
except ImportError: # pragma: no cover
|
||||
audit_log = None # type: ignore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
PROVIDER_OLLAMA = "ollama"
|
||||
PROVIDER_OPENROUTER = "openrouter"
|
||||
|
||||
# Error substrings that should trigger a fallback rather than propagate.
|
||||
# Kept narrow on purpose – we don't want to mask programmer errors.
|
||||
_ROUTABLE_ERROR_FRAGMENTS = (
|
||||
"Timeout",
|
||||
"nicht erreichbar",
|
||||
"ConnectionError",
|
||||
"Connection refused",
|
||||
"API Key nicht konfiguriert",
|
||||
)
|
||||
|
||||
|
||||
def _is_routable_error(exc: BaseException) -> bool:
|
||||
msg = str(exc)
|
||||
return any(frag in msg for frag in _ROUTABLE_ERROR_FRAGMENTS)
|
||||
|
||||
|
||||
class LLMProvider:
|
||||
"""Thin interface every provider satisfies."""
|
||||
|
||||
name: str = "base"
|
||||
model: str = ""
|
||||
|
||||
def is_available(self) -> bool:
|
||||
raise NotImplementedError
|
||||
|
||||
async def chat(self, message: str, history: list | None = None) -> str:
|
||||
raise NotImplementedError
|
||||
|
||||
async def generate_shell_command(self, description: str) -> dict:
|
||||
raise NotImplementedError
|
||||
|
||||
async def explain_output(self, command: str, output: str,
|
||||
error: str = "") -> str:
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class OllamaProvider(LLMProvider):
|
||||
name = PROVIDER_OLLAMA
|
||||
|
||||
def __init__(self, model: Optional[str] = None,
|
||||
model_fast: Optional[str] = None,
|
||||
num_ctx: Optional[int] = None,
|
||||
base_url: Optional[str] = None):
|
||||
# Pass openrouter_client=None so the underlying client does NOT
|
||||
# perform its own fallback – the router owns that decision now.
|
||||
self.client = OllamaClient(
|
||||
openrouter_client=None,
|
||||
model=model,
|
||||
model_fast=model_fast,
|
||||
num_ctx=num_ctx,
|
||||
base_url=base_url,
|
||||
)
|
||||
self.model = self.client.model
|
||||
|
||||
def is_available(self) -> bool:
|
||||
# Connection health is checked at call time; we always advertise
|
||||
# ourselves as available so the router will attempt and trigger
|
||||
# fallback on actual failure.
|
||||
return True
|
||||
|
||||
async def chat(self, message, history=None):
|
||||
return await self.client.chat(message, history)
|
||||
|
||||
async def generate_shell_command(self, description):
|
||||
return await self.client.generate_shell_command(description)
|
||||
|
||||
async def explain_output(self, command, output, error=""):
|
||||
return await self.client.explain_output(command, output, error)
|
||||
|
||||
|
||||
class OpenRouterProvider(LLMProvider):
|
||||
name = PROVIDER_OPENROUTER
|
||||
|
||||
def __init__(self, model: Optional[str] = None):
|
||||
self.client = OpenRouterClient()
|
||||
# OpenRouterClient reads OPENROUTER_API_KEY at import time. We
|
||||
# only override the model id, leaving the API key handling alone.
|
||||
self.model = model or "openrouter/free"
|
||||
if model:
|
||||
# Patch the FREE_MODELS dict so chat/coder both use the
|
||||
# configured model. Limited blast radius – it only affects
|
||||
# this OpenRouterClient instance, not the module constant.
|
||||
self._patch_model(model)
|
||||
|
||||
def _patch_model(self, model: str) -> None:
|
||||
# The underlying OpenRouterClient hard-codes FREE_MODELS["coder"]/
|
||||
# ["chat"] at call sites. Wrap _chat_sync so every call resolves
|
||||
# to the user-pinned model regardless of what callers pass in.
|
||||
pinned = model
|
||||
original_chat_sync = self.client._chat_sync
|
||||
|
||||
def wrapper(messages, system=None, model=None):
|
||||
return original_chat_sync(messages, system=system, model=pinned)
|
||||
|
||||
self.client._chat_sync = wrapper # type: ignore[assignment]
|
||||
|
||||
def is_available(self) -> bool:
|
||||
return self.client.is_available()
|
||||
|
||||
async def chat(self, message, history=None):
|
||||
return await self.client.chat(message, history)
|
||||
|
||||
async def generate_shell_command(self, description):
|
||||
return await self.client.generate_shell_command(description)
|
||||
|
||||
async def explain_output(self, command, output, error=""):
|
||||
return await self.client.explain_output(command, output, error)
|
||||
|
||||
|
||||
def _build_provider(kind: str) -> Optional[LLMProvider]:
|
||||
"""Build the provider for kind ∈ {"primary", "fallback"}.
|
||||
|
||||
Returns None if the configuration disables this tier (e.g. fallback
|
||||
requested but no key available).
|
||||
"""
|
||||
explicit = os.getenv(f"LLM_{kind.upper()}_PROVIDER", "").strip().lower()
|
||||
|
||||
if kind == "primary":
|
||||
provider_name = explicit or PROVIDER_OLLAMA
|
||||
else:
|
||||
# Fallback default: openrouter only if API key is configured.
|
||||
if explicit:
|
||||
provider_name = explicit
|
||||
elif os.getenv("OPENROUTER_API_KEY"):
|
||||
provider_name = PROVIDER_OPENROUTER
|
||||
else:
|
||||
return None
|
||||
|
||||
if provider_name in ("", "none", "off", "disabled"):
|
||||
return None
|
||||
|
||||
model = os.getenv(f"LLM_{kind.upper()}_MODEL", "").strip() or None
|
||||
|
||||
if provider_name == PROVIDER_OLLAMA:
|
||||
model_fast = os.getenv(f"LLM_{kind.upper()}_MODEL_FAST", "").strip() or None
|
||||
num_ctx_raw = os.getenv(f"LLM_{kind.upper()}_NUM_CTX", "").strip()
|
||||
num_ctx = int(num_ctx_raw) if num_ctx_raw.isdigit() else None
|
||||
return OllamaProvider(model=model, model_fast=model_fast,
|
||||
num_ctx=num_ctx)
|
||||
if provider_name == PROVIDER_OPENROUTER:
|
||||
provider = OpenRouterProvider(model=model)
|
||||
# Skip the tier if OpenRouter is configured but key is missing.
|
||||
if not provider.is_available():
|
||||
logger.warning(
|
||||
f"LLM router: {kind} provider 'openrouter' has no API key – disabled"
|
||||
)
|
||||
return None
|
||||
return provider
|
||||
|
||||
logger.error(f"LLM router: unknown {kind} provider '{provider_name}'")
|
||||
return None
|
||||
|
||||
|
||||
_BUILD_FROM_ENV = object() # sentinel: distinguish "not passed" from "None"
|
||||
|
||||
|
||||
class LLMRouter:
|
||||
"""Drop-in replacement for the old OllamaClient surface.
|
||||
|
||||
Exposes the same async methods (`chat`, `generate_shell_command`,
|
||||
`explain_output`) plus a small status surface (`primary`, `fallback`,
|
||||
`describe()`).
|
||||
|
||||
Pass `primary=None` / `fallback=None` to explicitly disable a tier;
|
||||
omit the kwarg entirely to build the tier from environment.
|
||||
"""
|
||||
|
||||
def __init__(self, primary=_BUILD_FROM_ENV, fallback=_BUILD_FROM_ENV):
|
||||
self.primary = _build_provider("primary") if primary is _BUILD_FROM_ENV else primary
|
||||
self.fallback = _build_provider("fallback") if fallback is _BUILD_FROM_ENV else fallback
|
||||
if self.primary is None and self.fallback is None:
|
||||
logger.error("LLMRouter: neither primary nor fallback configured!")
|
||||
|
||||
# ---- public describe ------------------------------------------------
|
||||
def describe(self) -> dict:
|
||||
def _info(p: LLMProvider | None) -> dict | None:
|
||||
if p is None:
|
||||
return None
|
||||
return {"provider": p.name, "model": p.model,
|
||||
"available": p.is_available()}
|
||||
return {"primary": _info(self.primary), "fallback": _info(self.fallback)}
|
||||
|
||||
# ---- dispatcher -----------------------------------------------------
|
||||
async def _dispatch(self, method_name: str, *args, **kwargs) -> Any:
|
||||
attempts: list[tuple[str, LLMProvider]] = []
|
||||
if self.primary is not None:
|
||||
attempts.append(("primary", self.primary))
|
||||
if self.fallback is not None:
|
||||
attempts.append(("fallback", self.fallback))
|
||||
|
||||
if not attempts:
|
||||
raise RuntimeError(
|
||||
"Kein LLM-Provider konfiguriert (LLM_PRIMARY_PROVIDER fehlt)"
|
||||
)
|
||||
|
||||
last_error: Optional[BaseException] = None
|
||||
for tier, provider in attempts:
|
||||
if not provider.is_available():
|
||||
self._audit("skip", tier, provider, method_name,
|
||||
reason="provider_unavailable")
|
||||
continue
|
||||
try:
|
||||
result = await getattr(provider, method_name)(*args, **kwargs)
|
||||
self._audit("hit", tier, provider, method_name)
|
||||
return result
|
||||
except Exception as e:
|
||||
last_error = e
|
||||
routable = _is_routable_error(e)
|
||||
self._audit("error", tier, provider, method_name,
|
||||
reason=str(e)[:200],
|
||||
routable=routable)
|
||||
if not routable:
|
||||
# Hard error – don't mask it with a fallback.
|
||||
raise
|
||||
# Otherwise loop to next tier.
|
||||
continue
|
||||
|
||||
# All tiers exhausted.
|
||||
raise RuntimeError(
|
||||
f"Alle LLM-Provider fehlgeschlagen für {method_name}: {last_error}"
|
||||
)
|
||||
|
||||
def _audit(self, status: str, tier: str, provider: LLMProvider,
|
||||
method: str, **extra) -> None:
|
||||
if audit_log is None:
|
||||
return
|
||||
try:
|
||||
details = {"tier": tier, "provider": provider.name,
|
||||
"model": provider.model, "method": method}
|
||||
details.update(extra)
|
||||
audit_log.log_action(None, "llm.route", result=status,
|
||||
details=details)
|
||||
except Exception as e: # pragma: no cover – audit must never fail
|
||||
logger.debug(f"audit emit failed: {e}")
|
||||
|
||||
# ---- public surface -------------------------------------------------
|
||||
async def chat(self, message: str, history: list | None = None) -> str:
|
||||
return await self._dispatch("chat", message, history)
|
||||
|
||||
async def generate_shell_command(self, description: str) -> dict:
|
||||
return await self._dispatch("generate_shell_command", description)
|
||||
|
||||
async def explain_output(self, command: str, output: str,
|
||||
error: str = "") -> str:
|
||||
return await self._dispatch("explain_output", command, output, error)
|
||||
|
||||
# ---- convenience for the warmup path in bot.py ----------------------
|
||||
@property
|
||||
def ollama_models(self) -> list[str]:
|
||||
"""Return distinct Ollama model ids configured across tiers –
|
||||
used by the warmup thread in bot.py."""
|
||||
models: list[str] = []
|
||||
for p in (self.primary, self.fallback):
|
||||
if isinstance(p, OllamaProvider):
|
||||
if p.client.model and p.client.model not in models:
|
||||
models.append(p.client.model)
|
||||
if (p.client.model_fast
|
||||
and p.client.model_fast not in models):
|
||||
models.append(p.client.model_fast)
|
||||
return models
|
||||
|
||||
@property
|
||||
def ollama_base_url(self) -> str:
|
||||
for p in (self.primary, self.fallback):
|
||||
if isinstance(p, OllamaProvider):
|
||||
return p.client.base_url
|
||||
return os.getenv("OLLAMA_BASE_URL", "http://localhost:11434")
|
||||
|
||||
def has_fallback(self) -> bool:
|
||||
return self.fallback is not None and self.fallback.is_available()
|
||||
+10
-6
@@ -81,10 +81,14 @@ def _extract_json(raw: str) -> dict:
|
||||
|
||||
|
||||
class OllamaClient:
|
||||
def __init__(self, openrouter_client=None):
|
||||
self.base_url = OLLAMA_BASE_URL
|
||||
self.model = OLLAMA_MODEL
|
||||
self.model_fast = OLLAMA_MODEL_FAST or OLLAMA_MODEL
|
||||
def __init__(self, openrouter_client=None, model: Optional[str] = None,
|
||||
model_fast: Optional[str] = None,
|
||||
num_ctx: Optional[int] = None,
|
||||
base_url: Optional[str] = None):
|
||||
self.base_url = base_url or OLLAMA_BASE_URL
|
||||
self.model = model or OLLAMA_MODEL
|
||||
self.model_fast = model_fast or OLLAMA_MODEL_FAST or self.model
|
||||
self.num_ctx = num_ctx if num_ctx is not None else OLLAMA_NUM_CTX
|
||||
self.openrouter = openrouter_client
|
||||
|
||||
def _chat_sync(self, messages: list, system: Optional[str] = None, think: Optional[bool] = None, use_fast: bool = False) -> str:
|
||||
@@ -95,7 +99,7 @@ class OllamaClient:
|
||||
"messages": messages,
|
||||
"stream": False,
|
||||
"keep_alive": -1,
|
||||
"options": {"num_ctx": OLLAMA_NUM_CTX},
|
||||
"options": {"num_ctx": self.num_ctx},
|
||||
}
|
||||
if system:
|
||||
payload["system"] = system
|
||||
@@ -152,7 +156,7 @@ class OllamaClient:
|
||||
except RuntimeError as e:
|
||||
if "Timeout" not in str(e) and "nicht erreichbar" not in str(e):
|
||||
raise
|
||||
logger.warning(f"Ollama nicht verfügbar für /shell – trying OpenRouter fallback")
|
||||
logger.warning("Ollama nicht verfügbar für /shell – trying OpenRouter fallback")
|
||||
|
||||
# 2. Fallback to OpenRouter
|
||||
if self.openrouter and self.openrouter.is_available():
|
||||
|
||||
@@ -53,7 +53,9 @@ class OpenRouterClient:
|
||||
"""OpenRouter Free Models Client – fallback when Ollama times out."""
|
||||
|
||||
def __init__(self):
|
||||
self.api_key = OPENROUTER_API_KEY
|
||||
# Read at construction time, not import time, so env changes between
|
||||
# boot and config reloads (or test monkeypatching) take effect.
|
||||
self.api_key = os.getenv("OPENROUTER_API_KEY", "") or OPENROUTER_API_KEY
|
||||
self.base_url = OPENROUTER_BASE_URL
|
||||
if self.api_key:
|
||||
logger.info("OpenRouter Fallback konfiguriert")
|
||||
@@ -103,6 +105,8 @@ class OpenRouterClient:
|
||||
"""Generate shell command via OpenRouter."""
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
system = "Server: Ubuntu 24.04, Clownfischserver setup."
|
||||
|
||||
prompt = (
|
||||
"You are a Linux bash command generator for Ubuntu 24.04.\n"
|
||||
"Convert the request into a single non-interactive bash command.\n"
|
||||
@@ -116,6 +120,7 @@ class OpenRouterClient:
|
||||
None,
|
||||
lambda: self._chat_sync(
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
system=system,
|
||||
model=FREE_MODELS["coder"],
|
||||
),
|
||||
)
|
||||
|
||||
@@ -4,8 +4,29 @@
|
||||
|
||||
# Telegram
|
||||
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrSTUvwxYZ
|
||||
# Legacy single-user mode: dieser Nutzer bekommt automatisch die operator-Rolle,
|
||||
# wenn keine CLOWNFISCH_* Rollen-Variablen gesetzt sind (v0.4-Kompatibilität).
|
||||
TELEGRAM_CHAT_ID=987654321
|
||||
|
||||
# === Team-Rollen (v0.5.0) ===========================================
|
||||
# Kommagetrennte Telegram User-IDs (nicht Chat-IDs!).
|
||||
# Operator = darf /shell /code /run /ja /ssh /snapshot /rollback ausführen
|
||||
# Approver = darf /ja (Bestätigungen) ausführen
|
||||
# Viewer = darf /start /help /status /snapshots /chat lesen
|
||||
# Inheritance: operator > approver > viewer (höhere Rolle erbt niedrigere Rechte).
|
||||
# CLOWNFISCH_OPERATORS=12345,67890
|
||||
# CLOWNFISCH_APPROVERS=11111
|
||||
# CLOWNFISCH_VIEWERS=22222,33333
|
||||
|
||||
# === Audit-Trail (DSGVO Art. 5, 32) =================================
|
||||
# Pfad zur JSON-Lines-Audit-Datei. Default: $INSTALL_DIR/audit.jsonl
|
||||
# AUDIT_LOG_PATH=/opt/clownfischserver/audit.jsonl
|
||||
# Aufbewahrungsdauer in Tagen (Art. 5(1)(e) – Storage Limitation).
|
||||
# Beim Bot-Start werden ältere Einträge automatisch entfernt. 0 = nie löschen.
|
||||
AUDIT_RETENTION_DAYS=90
|
||||
# Maximale Länge eines Audit-Detailfeldes (Datenminimierung).
|
||||
AUDIT_MAX_DETAIL_LEN=2000
|
||||
|
||||
# Ollama
|
||||
OLLAMA_BASE_URL=http://localhost:11434
|
||||
OLLAMA_MODEL=qwen2.5-coder:7b
|
||||
@@ -16,10 +37,36 @@ OLLAMA_NUM_CTX=16384
|
||||
# API Key von openrouter.ai (kostenlos, keine Kreditkarte)
|
||||
# OPENROUTER_API_KEY=sk-or-v1-...
|
||||
|
||||
# === LLM Routing (v0.5.0, Phase 3) ==================================
|
||||
# Konfiguriert welches Modell als Primary und welches als Fallback genutzt wird.
|
||||
# Wenn nicht gesetzt: defaultet auf Ollama primary, OpenRouter fallback (sofern
|
||||
# OPENROUTER_API_KEY gesetzt ist). Damit bleibt das v0.4-Verhalten erhalten.
|
||||
#
|
||||
# Primary (für sensible Daten – läuft lokal, keine Cloud)
|
||||
# LLM_PRIMARY_PROVIDER=ollama
|
||||
# LLM_PRIMARY_MODEL=qwen2.5-coder:7b
|
||||
# LLM_PRIMARY_MODEL_FAST=gemma3:4b
|
||||
# Gemma 4 27B benötigt mindestens 64K Token Kontext – sonst halluziniert das Modell.
|
||||
# Override pro Provider (überschreibt OLLAMA_NUM_CTX).
|
||||
# LLM_PRIMARY_NUM_CTX=65536
|
||||
#
|
||||
# Fallback (nur falls Primary nicht erreichbar / Timeout)
|
||||
# LLM_FALLBACK_PROVIDER=openrouter
|
||||
# LLM_FALLBACK_MODEL=deepseek/deepseek-r1:free
|
||||
# Wert "none" oder leer = Fallback explizit deaktivieren
|
||||
# LLM_FALLBACK_PROVIDER=none
|
||||
#
|
||||
# Optional: Kostentracking für Admins (informativ, keine Limit-Erzwingung).
|
||||
# TRACK_LLM_COSTS=true
|
||||
|
||||
# Snapshots
|
||||
SNAPSHOT_DIR=/opt/clownfisch-snapshots
|
||||
SNAPSHOT_METHOD=tar
|
||||
|
||||
# HomeIP Auto-Update (optional – myfritz.net Dynamic DNS)
|
||||
# Domain für automatische Firewall-Regel-Updates bei IP-Wechsel
|
||||
# CLOWNFISCHSERVER_HOME_IP_DOMAIN=vorname.myfritz.net
|
||||
|
||||
# Paths
|
||||
INSTALL_DIR=/opt/clownfischserver
|
||||
MGMT_USER=clownfish
|
||||
|
||||
+65
-3
@@ -29,8 +29,13 @@ MGMT_USER=""
|
||||
PKG_MANAGER=""
|
||||
DISTRO=""
|
||||
OLLAMA_MODEL=""
|
||||
OLLAMA_MODEL_FAST=""
|
||||
SNAPSHOT_METHOD="tar"
|
||||
OLLAMA_NUM_CTX=4096
|
||||
MYFRITZ_DOMAIN=""
|
||||
BOT_TOKEN=""
|
||||
CHAT_ID=""
|
||||
OPENROUTER_API_KEY=""
|
||||
|
||||
# HELPERS
|
||||
|
||||
@@ -105,7 +110,7 @@ print_banner() {
|
||||
echo -e "${CYAN}${BOLD}"
|
||||
echo " ╔══════════════════════════════════════════════════════╗"
|
||||
echo " ║ 🐠 CLOWNFISCHSERVER INSTALLER ║"
|
||||
echo " ║ v0.4.13 | GPL-3.0 ║"
|
||||
echo " ║ v0.5.0 | GPL-3.0 ║"
|
||||
echo " ╚══════════════════════════════════════════════════════╝"
|
||||
echo -e "${NC}"
|
||||
echo -e " Willkommen! Dieses Script richtet deinen"
|
||||
@@ -335,6 +340,34 @@ KNOCKEOF
|
||||
warn "Bitte Firewall manuell einrichten (iptables/nftables)"
|
||||
;;
|
||||
esac
|
||||
|
||||
echo ""
|
||||
echo -e " ${BOLD}${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||
echo -e " ${BOLD} myfritz – Home-IP Auto-Update (optional)${NC}"
|
||||
echo -e " ${CYAN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||
echo -e " Wenn du ein AVM-Gerät (Fritzbox) mit myfritz.net hast,"
|
||||
echo -e " kann der Bot deine Home-IP automatisch ins Firewall-Regelwerk"
|
||||
echo -e " schreiben. Das bedeutet: SSH bleibt immer erreichbar, auch"
|
||||
echo -e " wenn sich deine IP ändert (z.B. ISP, mobil, VPN)."
|
||||
echo ""
|
||||
echo -e " ${YELLOW}?${NC} ${BOLD}Hast du ein AVM myfritz.net Gerät?${NC} [j/n]"
|
||||
read -r -p " -> " USE_MYFRITZ
|
||||
|
||||
if [[ "$USE_MYFRITZ" =~ ^[jJ]$ ]]; then
|
||||
echo ""
|
||||
echo -e " ${YELLOW}?${NC} ${BOLD}Deine myfritz-Domain:${NC}"
|
||||
echo -e " ${CYAN} Format: vorname.myfritz.net${NC}"
|
||||
read -r -p " -> " MYFRITZ_DOMAIN
|
||||
if [[ -z "$MYFRITZ_DOMAIN" ]]; then
|
||||
MYFRITZ_DOMAIN=""
|
||||
warn "myfritz übersprungen"
|
||||
else
|
||||
ok "myfritz Domain: $MYFRITZ_DOMAIN"
|
||||
fi
|
||||
else
|
||||
MYFRITZ_DOMAIN=""
|
||||
info "myfritz übersprungen"
|
||||
fi
|
||||
}
|
||||
|
||||
# HELPER: Pull an Ollama model with progress bar
|
||||
@@ -681,6 +714,7 @@ SNAPSHOT_METHOD=$SNAPSHOT_METHOD
|
||||
INSTALL_DIR=$INSTALL_DIR
|
||||
MGMT_USER=$MGMT_USER
|
||||
AIDER_WORKDIR=$WORKSPACE_DIR
|
||||
CLOWNFISCHSERVER_HOME_IP_DOMAIN=$MYFRITZ_DOMAIN
|
||||
ENVEOF
|
||||
|
||||
chmod 600 "$INSTALL_DIR/config/.env"
|
||||
@@ -696,6 +730,13 @@ step_service() {
|
||||
cp -r "$SCRIPT_DIR/bot/"* "$INSTALL_DIR/bot/" || fail "Bot-Dateien nicht gefunden – ist der bot/ Ordner dabei?"
|
||||
ok "Bot-Dateien kopiert"
|
||||
|
||||
mkdir -p "$INSTALL_DIR/scripts"
|
||||
if [ -d "$SCRIPT_DIR/scripts" ]; then
|
||||
cp -r "$SCRIPT_DIR/scripts/"* "$INSTALL_DIR/scripts/" 2>/dev/null || true
|
||||
chmod +x "$INSTALL_DIR/scripts/"*.sh 2>/dev/null || true
|
||||
ok "Scripts kopiert"
|
||||
fi
|
||||
|
||||
chown -R "$MGMT_USER:$MGMT_USER" "$INSTALL_DIR" "$SNAPSHOT_DIR" "$WORKSPACE_DIR"
|
||||
chmod -R u+w "$INSTALL_DIR/bot"
|
||||
ok "Verzeichnis-Rechte gesetzt"
|
||||
@@ -762,6 +803,27 @@ BOOTEOF
|
||||
|
||||
run_with_spinner "Boot-Notification aktivieren" bash -c "systemctl daemon-reload && systemctl enable clownfisch-boot"
|
||||
ok "Boot-Notification: Telegram-Meldung 2 Min nach Serverstart"
|
||||
|
||||
# HomeIP Updater – falls myfritz konfiguriert
|
||||
if [[ -n "$MYFRITZ_DOMAIN" ]]; then
|
||||
echo ""
|
||||
info "Installiere HomeIP Auto-Updater (5-Min Cron)..."
|
||||
|
||||
# Copy the homeip-updater script if it exists
|
||||
if [ -f "$SCRIPT_DIR/scripts/clownfischserver-homeip-updater.sh" ]; then
|
||||
mkdir -p /opt/clownfischserver-scripts
|
||||
cp "$SCRIPT_DIR/scripts/clownfischserver-homeip-updater.sh" /opt/clownfischserver/scripts/
|
||||
chmod +x /opt/clownfischserver/scripts/clownfischserver-homeip-updater.sh
|
||||
|
||||
# Add cron job (every 5 minutes)
|
||||
CRON_JOB="*/5 * * * * /opt/clownfischserver/scripts/clownfischserver-homeip-updater.sh >> /var/log/clownfischserver-homeip-updater.log 2>&1"
|
||||
(crontab -l 2>/dev/null | grep -v "clownfischserver-homeip-updater" ; echo "$CRON_JOB") | crontab - 2>/dev/null || true
|
||||
|
||||
ok "HomeIP Auto-Updater installiert (myfritz: $MYFRITZ_DOMAIN)"
|
||||
else
|
||||
warn "homeip-updater Script nicht gefunden – übersprungen"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# ZUSAMMENFASSUNG
|
||||
@@ -771,7 +833,7 @@ print_summary() {
|
||||
echo ""
|
||||
echo -e "${GREEN}${BOLD}"
|
||||
echo " +======================================================+"
|
||||
echo " | CLOWNFISCHSERVER BEREIT! v0.4.13 |"
|
||||
echo " | CLOWNFISCHSERVER BEREIT! v0.5.0 |"
|
||||
echo " +======================================================+"
|
||||
echo -e "${NC}"
|
||||
echo -e " ${BOLD}Server:${NC} $SERVER_IP"
|
||||
@@ -862,7 +924,7 @@ main() {
|
||||
esac
|
||||
fi
|
||||
|
||||
log "=== Clownfischserver v0.4.13 Installation gestartet ==="
|
||||
log "=== Clownfischserver v0.5.0 Installation gestartet ==="
|
||||
step_checks
|
||||
step_packages
|
||||
step_user
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
#!/bin/bash
|
||||
# =============================================================================
|
||||
# 🐠 CLOWNFISCHSERVER – HomeIP Auto-Update
|
||||
# Purpose: Auto-update firewall rule for dynamic home IP (myfritz.net)
|
||||
# Runs: Every 5 minutes (via cron)
|
||||
# Author: Mehlzoerwer-Claude
|
||||
# License: GPL-3.0
|
||||
# =============================================================================
|
||||
|
||||
set -e
|
||||
|
||||
# --- Configuration ---
|
||||
INSTALL_DIR="/opt/clownfischserver"
|
||||
ENV_FILE="$INSTALL_DIR/config/.env"
|
||||
STATE_FILE="/var/lib/clownfischserver-homeip-state"
|
||||
LOG_FILE="/var/log/clownfischserver-homeip-updater.log"
|
||||
SSH_PORT=22
|
||||
|
||||
# --- Helpers ---
|
||||
log() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$LOG_FILE"
|
||||
}
|
||||
|
||||
# --- Load .env ---
|
||||
load_env() {
|
||||
if [ ! -f "$ENV_FILE" ]; then
|
||||
log "ERROR: .env not found at $ENV_FILE"
|
||||
exit 1
|
||||
fi
|
||||
# Load and trim CRLF/LF
|
||||
MYFRITZ_DOMAIN=$(grep "^CLOWNFISCHSERVER_HOME_IP_DOMAIN=" "$ENV_FILE" | cut -d= -f2 | tr -d '\r\n')
|
||||
if [ -z "$MYFRITZ_DOMAIN" ]; then
|
||||
log "ERROR: CLOWNFISCHSERVER_HOME_IP_DOMAIN not set in .env"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Detect Firewall ---
|
||||
detect_firewall() {
|
||||
if systemctl is-active --quiet ufw; then
|
||||
echo "ufw"
|
||||
elif systemctl is-active --quiet firewalld; then
|
||||
echo "firewalld"
|
||||
elif command -v iptables &>/dev/null; then
|
||||
echo "iptables"
|
||||
else
|
||||
log "ERROR: No firewall detected (ufw/firewalld/iptables)"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Resolve DNS with fallbacks ---
|
||||
resolve_ip() {
|
||||
local domain="$1"
|
||||
|
||||
# Try dig first
|
||||
if command -v dig &>/dev/null; then
|
||||
IP=$(dig +short "$domain" A | grep -oE '^[0-9.]+$' | head -1)
|
||||
[ -n "$IP" ] && echo "$IP" && return 0
|
||||
fi
|
||||
|
||||
# Try nslookup
|
||||
if command -v nslookup &>/dev/null; then
|
||||
IP=$(nslookup "$domain" 2>/dev/null | grep "Address:" | tail -1 | awk '{print $2}' | grep -oE '^[0-9.]+$')
|
||||
[ -n "$IP" ] && echo "$IP" && return 0
|
||||
fi
|
||||
|
||||
# Try getent
|
||||
if command -v getent &>/dev/null; then
|
||||
IP=$(getent hosts "$domain" | awk '{print $1}' | grep -oE '^[0-9.]+$' | head -1)
|
||||
[ -n "$IP" ] && echo "$IP" && return 0
|
||||
fi
|
||||
|
||||
log "ERROR: Could not resolve $domain"
|
||||
return 1
|
||||
}
|
||||
|
||||
# --- Load previous state ---
|
||||
load_state() {
|
||||
if [ -f "$STATE_FILE" ]; then
|
||||
PREVIOUS_IP=$(cat "$STATE_FILE" | grep "^IP=" | cut -d= -f2)
|
||||
echo "$PREVIOUS_IP"
|
||||
else
|
||||
echo ""
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Save state ---
|
||||
save_state() {
|
||||
local ip="$1"
|
||||
echo "IP=$ip" > "$STATE_FILE"
|
||||
chmod 600 "$STATE_FILE"
|
||||
}
|
||||
|
||||
# --- UFW Functions ---
|
||||
ufw_add_rule() {
|
||||
local ip="$1"
|
||||
ufw allow from "$ip" to any port "$SSH_PORT" comment "clownfisch-homeip" >/dev/null 2>&1
|
||||
log "UFW: Added rule for $ip:$SSH_PORT"
|
||||
}
|
||||
|
||||
ufw_remove_rule() {
|
||||
local ip="$1"
|
||||
echo "y" | ufw delete allow from "$ip" to any port "$SSH_PORT" >/dev/null 2>&1
|
||||
log "UFW: Deleted rule for $ip"
|
||||
}
|
||||
|
||||
# --- Firewalld Functions ---
|
||||
firewalld_add_rule() {
|
||||
local ip="$1"
|
||||
firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='$ip' port protocol='tcp' port='$SSH_PORT' accept" >/dev/null 2>&1
|
||||
firewall-cmd --reload >/dev/null 2>&1
|
||||
log "firewalld: Added rule for $ip:$SSH_PORT"
|
||||
}
|
||||
|
||||
firewalld_remove_rule() {
|
||||
local ip="$1"
|
||||
firewall-cmd --permanent --remove-rich-rule="rule family='ipv4' source address='$ip' port protocol='tcp' port='$SSH_PORT' accept" >/dev/null 2>&1
|
||||
firewall-cmd --reload >/dev/null 2>&1
|
||||
log "firewalld: Deleted rule for $ip"
|
||||
}
|
||||
|
||||
# --- iptables Functions ---
|
||||
iptables_add_rule() {
|
||||
local ip="$1"
|
||||
iptables -I INPUT -s "$ip" -p tcp --dport "$SSH_PORT" -j ACCEPT
|
||||
log "iptables: Added rule for $ip:$SSH_PORT"
|
||||
}
|
||||
|
||||
iptables_remove_rule() {
|
||||
local ip="$1"
|
||||
iptables -D INPUT -s "$ip" -p tcp --dport "$SSH_PORT" -j ACCEPT 2>/dev/null || true
|
||||
log "iptables: Deleted rule for $ip"
|
||||
}
|
||||
|
||||
# --- Main ---
|
||||
main() {
|
||||
log "=== HomeIP Auto-Update started ==="
|
||||
|
||||
load_env
|
||||
FIREWALL=$(detect_firewall)
|
||||
NEW_IP=$(resolve_ip "$MYFRITZ_DOMAIN") || exit 1
|
||||
PREVIOUS_IP=$(load_state)
|
||||
|
||||
if [ "$NEW_IP" = "$PREVIOUS_IP" ]; then
|
||||
log "No change: $NEW_IP (same as previous)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
log "IP change detected: $PREVIOUS_IP → $NEW_IP (firewall: $FIREWALL)"
|
||||
|
||||
# Remove old rule
|
||||
if [ -n "$PREVIOUS_IP" ]; then
|
||||
case "$FIREWALL" in
|
||||
ufw) ufw_remove_rule "$PREVIOUS_IP" ;;
|
||||
firewalld) firewalld_remove_rule "$PREVIOUS_IP" ;;
|
||||
iptables) iptables_remove_rule "$PREVIOUS_IP" ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
# Add new rule
|
||||
case "$FIREWALL" in
|
||||
ufw) ufw_add_rule "$NEW_IP" ;;
|
||||
firewalld) firewalld_add_rule "$NEW_IP" ;;
|
||||
iptables) iptables_add_rule "$NEW_IP" ;;
|
||||
esac
|
||||
|
||||
# Save state
|
||||
save_state "$NEW_IP"
|
||||
|
||||
log "=== HomeIP Auto-Update completed successfully ==="
|
||||
}
|
||||
|
||||
main "$@"
|
||||
Reference in New Issue
Block a user