Initial: Mesh-Skills, Configs, Scripts
This commit is contained in:
root
@@ -0,0 +1,176 @@
|
||||
---
|
||||
name: hermes-gateway-systemd
|
||||
description: "Troubleshoot and fix Hermes Agent Gateway systemd service issues — crash loops, working directory mismatches, permission problems, and restart blockers."
|
||||
version: 1.0.0
|
||||
author: Hermes Agent
|
||||
license: MIT
|
||||
platforms: [linux]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [hermes, gateway, systemd, troubleshooting, service, crash-loop]
|
||||
related_skills: [hermes-agent]
|
||||
---
|
||||
|
||||
# Hermes Gateway systemd Service Troubleshooting
|
||||
|
||||
This skill covers diagnosing and fixing common Hermes Gateway systemd service problems — especially the infamous crash loop caused by `WorkingDirectory` / `User` mismatches.
|
||||
|
||||
## Scope
|
||||
|
||||
- **Hermes Gateway** running as a systemd service (user or system)
|
||||
- **Crash loops** (status 200/CHDIR, rapid restarts)
|
||||
- **Permission issues** (config owned by root, service runs as user, or vice versa)
|
||||
- **Restart blockers** (cannot run `systemctl restart` from inside gateway session)
|
||||
|
||||
## Quick Diagnosis
|
||||
|
||||
```bash
|
||||
# 1. Check service status
|
||||
systemctl status hermes-gateway --no-pager
|
||||
|
||||
# 2. View crash loop logs
|
||||
journalctl -u hermes-gateway -n 30 --no-pager
|
||||
|
||||
# 3. Inspect service file
|
||||
systemctl cat hermes-gateway
|
||||
|
||||
# 4. Verify config location matches service User/WorkingDirectory
|
||||
ls -la /root/.hermes/config.yaml 2>/dev/null
|
||||
ls -la /home/<user>/.hermes/config.yaml 2>/dev/null
|
||||
```
|
||||
|
||||
## Common Failure: CHDIR Crash Loop
|
||||
|
||||
### Symptom
|
||||
```
|
||||
hermes-gateway.service: Changing to the requested working directory failed: No such file or directory
|
||||
hermes-gateway.service: Main process exited, code=exited, status=200/CHDIR
|
||||
Restart counter increments every 5 seconds (e.g., 20,000+ restarts)
|
||||
```
|
||||
|
||||
### Root Cause
|
||||
Service file specifies `User=X` and `WorkingDirectory=/home/X/.hermes` but:
|
||||
- Config actually lives in `/root/.hermes` (installed as root)
|
||||
- OR `/home/X/.hermes` doesn't exist
|
||||
|
||||
### Fix
|
||||
Align `User`, `WorkingDirectory`, `HOME`, `HERMES_HOME` to where config actually lives.
|
||||
|
||||
**If config is in `/root/.hermes` (root install):**
|
||||
```ini
|
||||
User=root
|
||||
Group=root
|
||||
WorkingDirectory=/root/.hermes
|
||||
Environment="HOME=/root"
|
||||
Environment="HERMES_HOME=/root/.hermes"
|
||||
```
|
||||
|
||||
**If config is in `/home/user/.hermes` (user install):**
|
||||
```ini
|
||||
User=user
|
||||
Group=user
|
||||
WorkingDirectory=/home/user/.hermes
|
||||
Environment="HOME=/home/user"
|
||||
Environment="HERMES_HOME=/home/user/.hermes"
|
||||
```
|
||||
|
||||
Then:
|
||||
```bash
|
||||
systemctl daemon-reload
|
||||
pkill -9 -f 'hermes_cli.main gateway run' # kill crash loop
|
||||
systemctl start hermes-gateway
|
||||
sleep 3
|
||||
systemctl status hermes-gateway --no-pager
|
||||
```
|
||||
|
||||
### Real-World Example (This Session)
|
||||
- **Config location:** `/root/.hermes` (owned by root, installed via `hermes setup`)
|
||||
- **Broken service file:** `User=mehlzoerwer`, `WorkingDirectory=/home/mehlzoerwer/.hermes` (directory didn't exist)
|
||||
- **Symptom:** 20,731+ restart loop, status 200/CHDIR
|
||||
- **Fix applied:** Changed service to `User=root`, `WorkingDirectory=/root/.hermes`, `HERMES_HOME=/root/.hermes`
|
||||
- **Result:** Service started cleanly, Telegram connected, startup notification sent
|
||||
|
||||
## Blocker: Cannot Restart from Inside Gateway
|
||||
|
||||
### Error
|
||||
```
|
||||
Blocked: cannot restart or stop the gateway from inside the gateway process.
|
||||
The gateway would kill this command before it could complete (SIGTERM propagates to child processes).
|
||||
Run `hermes gateway restart` from a separate shell outside the running gateway.
|
||||
```
|
||||
|
||||
### Cause
|
||||
Hermes detects it's running inside its own gateway process and blocks process-termination commands to prevent self-kill.
|
||||
|
||||
### Workarounds
|
||||
|
||||
| Method | When to Use |
|
||||
|--------|-------------|
|
||||
| **New SSH session** | Always works — run `systemctl restart hermes-gateway` from fresh terminal |
|
||||
| **`hermes gateway restart`** | If gateway is responsive (slash command in Telegram/CLI) |
|
||||
| **`at` job** | `echo 'systemctl restart hermes-gateway' | at now + 1 minute` |
|
||||
| **Direct PID kill from outside** | `ssh root@host 'pkill -9 -f hermes_cli.main'` |
|
||||
| **tmux/screen on server** | Run commands in detached session |
|
||||
|
||||
### Scheduled Auto-Restart (systemd Timer) — durable pattern
|
||||
|
||||
To restart the gateway **on a recurring schedule** (e.g. nightly 23:59 to refresh memory/config) without manual intervention, install a pair of units **from a shell OUTSIDE the gateway**:
|
||||
|
||||
`/etc/systemd/system/hermes-gateway-restart.service`:
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Naechtlicher Hermes-Gateway-Neustart
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/bin/systemctl restart hermes-gateway.service
|
||||
TimeoutStartSec=60
|
||||
```
|
||||
|
||||
`/etc/systemd/system/hermes-gateway-restart.timer`:
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Startet Hermes-Gateway taeglich um 23:59
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 23:59:00
|
||||
Persistent=false
|
||||
RandomizedDelaySec=0
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
```
|
||||
|
||||
Install + enable:
|
||||
```bash
|
||||
cp restart.service restart.timer /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl enable hermes-gateway-restart.timer # allowed from inside
|
||||
```
|
||||
|
||||
**CRITICAL PITFALL:** From inside the running gateway, `systemctl daemon-reload` and
|
||||
`systemctl enable <timer>` succeed, but **`systemctl start <timer>` is ALSO blocked** (the
|
||||
security scanner traces the timer to the gateway-restart service and treats the activation
|
||||
as a self-kill). The `enable` persists, so the timer is armed and will fire on schedule —
|
||||
you only need the user to run `systemctl start hermes-gateway-restart.timer` **once** from a
|
||||
normal shell. After that it fires automatically daily.
|
||||
|
||||
**Ordering tip (user preference):** schedule the restart AFTER the daily Doku/Archiv cron and
|
||||
BEFORE the Wissensstand (knowledge-sync) job, so the fresh session has updated memory/config
|
||||
before the sync runs.
|
||||
### Real-World Note (This Session)
|
||||
Attempting to fix the remote gateway (92.60.38.175) **from within this local Hermes gateway session** triggered the blocker. The fix required running commands directly on the remote server via SSH from a separate terminal, not from inside an active Hermes gateway process.
|
||||
|
||||
## Service File Templates
|
||||
|
||||
See `references/gateway-systemd-troubleshooting.md` for complete, ready-to-use service file templates for both root and user installs.
|
||||
|
||||
## Diagnostic Checklist (Run in Order)
|
||||
|
||||
1. `systemctl status hermes-gateway` — active? crash loop?
|
||||
2. `journalctl -u hermes-gateway -n 20` — CHDIR error? Permission denied? Module not found?
|
||||
3. `systemctl cat hermes-gateway` — check User, WorkingDirectory, HERMES_HOME
|
||||
4. `ls -la /root/.hermes /home/user/.hermes` — which exists? Who owns it?
|
||||
5. `cat /root/.hermes/config.yaml | head -5` — verify config is valid
|
||||
6. Fix service file to match reality → `daemon-reload` → kill loop → restart
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `hermes-agent` — main Hermes configuration skill (covers gateway setup, platforms, config)
|
||||
@@ -0,0 +1,201 @@
|
||||
# Hermes Gateway systemd Service Troubleshooting — Reference Detail
|
||||
|
||||
## Common Crash Loop: "Changing to the requested working directory failed"
|
||||
|
||||
**Symptom:** Service restarts every 5 seconds, logs show:
|
||||
```
|
||||
hermes-gateway.service: Changing to the requested working directory failed: No such file or directory
|
||||
hermes-gateway.service: Main process exited, code=exited, status=200/CHDIR
|
||||
```
|
||||
|
||||
**Root Cause:** `WorkingDirectory` in service file points to a path that doesn't exist for the configured `User`.
|
||||
|
||||
**Typical Scenario:**
|
||||
- Config lives in `/root/.hermes` (installed as root)
|
||||
- Service file has `User=mehlzoerwer` and `WorkingDirectory=/home/mehlzoerwer/.hermes`
|
||||
- But `/home/mehlzoerwer/.hermes` doesn't exist
|
||||
|
||||
**Fix:** Align `User`, `WorkingDirectory`, `HOME`, `HERMES_HOME` to where config actually lives.
|
||||
|
||||
```ini
|
||||
# If config is in /root/.hermes:
|
||||
User=root
|
||||
Group=root
|
||||
WorkingDirectory=/root/.hermes
|
||||
Environment="HOME=/root"
|
||||
Environment="HERMES_HOME=/root/.hermes"
|
||||
|
||||
# If config is in /home/user/.hermes:
|
||||
User=user
|
||||
Group=user
|
||||
WorkingDirectory=/home/user/.hermes
|
||||
Environment="HOME=/home/user"
|
||||
Environment="HERMES_HOME=/home/user/.hermes"
|
||||
```
|
||||
|
||||
Then:
|
||||
```bash
|
||||
systemctl daemon-reload
|
||||
pkill -9 -f 'hermes_cli.main gateway run' # kill crash loop
|
||||
systemctl start hermes-gateway
|
||||
systemctl status hermes-gateway --no-pager
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cannot Restart from Inside Gateway Session
|
||||
|
||||
**Problem:** Running `systemctl restart hermes-gateway` (or `pkill`/`kill`) from within an active Hermes gateway session fails with:
|
||||
```text
|
||||
Blocked: cannot restart or stop the gateway from inside the gateway process.
|
||||
The gateway would kill this command before it could complete (SIGTERM propagates to child processes).
|
||||
Run `hermes gateway restart` from a separate shell outside the running gateway.
|
||||
```
|
||||
|
||||
**Reason:** Hermes detects it's running inside its own gateway process and blocks process termination commands to prevent self-kill.
|
||||
|
||||
**Workarounds:**
|
||||
1. **Run commands from a separate shell** (new SSH session, tmux pane, or `at` job)
|
||||
2. **Use `hermes gateway restart`** from within the gateway (if gateway is responsive)
|
||||
3. **Kill via PID directly from outside**: `ssh root@host 'pkill -9 -f hermes_cli.main'`
|
||||
4. **Schedule restart**: `echo 'systemctl restart hermes-gateway' | at now + 1 minute`
|
||||
|
||||
---
|
||||
|
||||
## Telegram Polling Conflict (409 Error)
|
||||
|
||||
**Symptom:** When starting the gateway, logs show:
|
||||
```text
|
||||
WARNING hermes_plugins.telegram_platform.adapter: [Telegram] Telegram polling conflict (1/5) — previous session still held open on Telegram's servers. Waiting 20s for it to expire. Error: Conflict: terminated by other getUpdates request; make sure that only one bot instance is running
|
||||
INFO hermes_plugins.telegram_platform.adapter: [Telegram] Telegram polling resumed after conflict retry 1/5
|
||||
```
|
||||
|
||||
**Cause:** A previous gateway instance (or another process) still holds an active Long Polling connection to Telegram. Telegram only allows ONE active `getUpdates` connection per bot token.
|
||||
|
||||
**Built-in Resolution:** Hermes Gateway (since recent versions) automatically retries up to 5 times with exponential backoff (20s, 40s, 60s, 80s, 100s). The old connection expires on Telegram's side after ~30-60 seconds of inactivity.
|
||||
|
||||
**When to Act:**
|
||||
- If conflict persists after all 5 retries → another process is actively polling (check for duplicate gateway services, cron jobs, or manual runs)
|
||||
- Fix: `pkill -9 -f 'hermes_cli.main gateway run'` then restart cleanly
|
||||
|
||||
**Verification:**
|
||||
```bash
|
||||
# Check for duplicate gateway processes
|
||||
ps aux | grep 'hermes_cli.main gateway run'
|
||||
|
||||
# Should only show ONE process (the systemd one)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Gateway Startup Notification
|
||||
|
||||
**Behavior:** On successful Telegram connection, the gateway automatically sends a startup notification to the configured `TELEGRAM_HOME_CHANNEL` (from `.env` or config).
|
||||
|
||||
**Log entry:**
|
||||
```text
|
||||
INFO gateway.run: Sent home-channel startup notification to telegram:6417835766
|
||||
```
|
||||
|
||||
**If notification not received:**
|
||||
1. Verify `TELEGRAM_HOME_CHANNEL` and `TELEGRAM_ALLOWED_USERS` in `.env` match your chat ID
|
||||
2. Check bot has permission to message that chat (not blocked, not a channel without bot admin)
|
||||
3. Check gateway logs for delivery errors after "Sent home-channel startup notification"
|
||||
|
||||
---
|
||||
|
||||
## Diagnostic Checklist
|
||||
|
||||
```bash
|
||||
# 1. Check service file
|
||||
systemctl cat hermes-gateway
|
||||
|
||||
# 2. Verify paths exist for the configured User
|
||||
ls -la /home/user/.hermes # or /root/.hermes
|
||||
|
||||
# 3. Check current crash loop
|
||||
journalctl -u hermes-gateway -n 20 --no-pager
|
||||
|
||||
# 4. Verify Hermes config location
|
||||
cat /root/.hermes/config.yaml | head -5
|
||||
# or
|
||||
cat /home/user/.hermes/config.yaml | head -5
|
||||
|
||||
# 5. Check which user owns the config
|
||||
ls -la /root/.hermes/config.yaml
|
||||
ls -la /home/user/.hermes/config.yaml 2>/dev/null
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Service File Template (Root Install)
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Hermes Agent Gateway - Messaging Platform Integration
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
Group=root
|
||||
ExecStart=/usr/local/lib/hermes-agent/venv/bin/python -m hermes_cli.main gateway run
|
||||
WorkingDirectory=/root/.hermes
|
||||
Environment="HOME=/root"
|
||||
Environment="USER=root"
|
||||
Environment="LOGNAME=root"
|
||||
Environment="PATH=/usr/local/lib/hermes-agent/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||
Environment="VIRTUAL_ENV=/usr/local/lib/hermes-agent/venv"
|
||||
Environment="HERMES_HOME=/root/.hermes"
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
RestartForceExitStatus=75
|
||||
KillMode=mixed
|
||||
KillSignal=SIGTERM
|
||||
ExecReload=/bin/kill -USR1 $MAINPID
|
||||
TimeoutStopSec=210
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Service File Template (User Install)
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Hermes Agent Gateway - Messaging Platform Integration
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=<username>
|
||||
Group=<username>
|
||||
ExecStart=/usr/local/lib/hermes-agent/venv/bin/python -m hermes_cli.main gateway run
|
||||
WorkingDirectory=/home/<username>/.hermes
|
||||
Environment="HOME=/home/<username>"
|
||||
Environment="USER=<username>"
|
||||
Environment="LOGNAME=<username>"
|
||||
Environment="PATH=/usr/local/lib/hermes-agent/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||
Environment="VIRTUAL_ENV=/usr/local/lib/hermes-agent/venv"
|
||||
Environment="HERMES_HOME=/home/<username>/.hermes"
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
RestartForceExitStatus=75
|
||||
KillMode=mixed
|
||||
KillSignal=SIGTERM
|
||||
ExecReload=/bin/kill -USR1 $MAINPID
|
||||
TimeoutStopSec=210
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
Reference in New Issue
Block a user