mesh-configs/skills/devops/hermes-gateway-systemd/SKILL.md

---
name: hermes-gateway-systemd
description: "Troubleshoot and fix Hermes Agent Gateway systemd service issues — crash loops, working directory mismatches, permission problems, and restart blockers."
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux]
metadata:
  hermes:
    tags: [hermes, gateway, systemd, troubleshooting, service, crash-loop]
    related_skills: [hermes-agent]
---

# Hermes Gateway systemd Service Troubleshooting

This skill covers diagnosing and fixing common Hermes Gateway systemd service problems — especially the infamous crash loop caused by `WorkingDirectory` / `User` mismatches.

## Scope

- **Hermes Gateway** running as a systemd service (user or system)
- **Crash loops** (status 200/CHDIR, rapid restarts)
- **Permission issues** (config owned by root, service runs as user, or vice versa)
- **Restart blockers** (cannot run `systemctl restart` from inside gateway session)

## Quick Diagnosis

```bash
# 1. Check service status
systemctl status hermes-gateway --no-pager

# 2. View crash loop logs
journalctl -u hermes-gateway -n 30 --no-pager

# 3. Inspect service file
systemctl cat hermes-gateway

# 4. Verify config location matches service User/WorkingDirectory
ls -la /root/.hermes/config.yaml 2>/dev/null
ls -la /home/<user>/.hermes/config.yaml 2>/dev/null
```

## Common Failure: CHDIR Crash Loop

### Symptom
```
hermes-gateway.service: Changing to the requested working directory failed: No such file or directory
hermes-gateway.service: Main process exited, code=exited, status=200/CHDIR
Restart counter increments every 5 seconds (e.g., 20,000+ restarts)
```

### Root Cause
Service file specifies `User=X` and `WorkingDirectory=/home/X/.hermes` but:
- Config actually lives in `/root/.hermes` (installed as root)
- OR `/home/X/.hermes` doesn't exist

### Fix
Align `User`, `WorkingDirectory`, `HOME`, `HERMES_HOME` to where config actually lives.

**If config is in `/root/.hermes` (root install):**
```ini
User=root
Group=root
WorkingDirectory=/root/.hermes
Environment="HOME=/root"
Environment="HERMES_HOME=/root/.hermes"
```

**If config is in `/home/user/.hermes` (user install):**
```ini
User=user
Group=user
WorkingDirectory=/home/user/.hermes
Environment="HOME=/home/user"
Environment="HERMES_HOME=/home/user/.hermes"
```

Then:
```bash
systemctl daemon-reload
pkill -9 -f 'hermes_cli.main gateway run'  # kill crash loop
systemctl start hermes-gateway
sleep 3
systemctl status hermes-gateway --no-pager
```

### Real-World Example (This Session)
- **Config location:** `/root/.hermes` (owned by root, installed via `hermes setup`)
- **Broken service file:** `User=mehlzoerwer`, `WorkingDirectory=/home/mehlzoerwer/.hermes` (directory didn't exist)
- **Symptom:** 20,731+ restart loop, status 200/CHDIR
- **Fix applied:** Changed service to `User=root`, `WorkingDirectory=/root/.hermes`, `HERMES_HOME=/root/.hermes`
- **Result:** Service started cleanly, Telegram connected, startup notification sent

## Blocker: Cannot Restart from Inside Gateway

### Error
```
Blocked: cannot restart or stop the gateway from inside the gateway process.
The gateway would kill this command before it could complete (SIGTERM propagates to child processes).
Run `hermes gateway restart` from a separate shell outside the running gateway.
```

### Cause
Hermes detects it's running inside its own gateway process and blocks process-termination commands to prevent self-kill.

### Workarounds

| Method | When to Use |
|--------|-------------|
| **New SSH session** | Always works — run `systemctl restart hermes-gateway` from fresh terminal |
| **`hermes gateway restart`** | If gateway is responsive (slash command in Telegram/CLI) |
| **`at` job** | `echo 'systemctl restart hermes-gateway' | at now + 1 minute` |
| **Direct PID kill from outside** | `ssh root@host 'pkill -9 -f hermes_cli.main'` |
| **tmux/screen on server** | Run commands in detached session |

### Scheduled Auto-Restart (systemd Timer) — durable pattern

To restart the gateway **on a recurring schedule** (e.g. nightly 23:59 to refresh memory/config) without manual intervention, install a pair of units **from a shell OUTSIDE the gateway**:

`/etc/systemd/system/hermes-gateway-restart.service`:
```ini
[Unit]
Description=Naechtlicher Hermes-Gateway-Neustart
[Service]
Type=oneshot
ExecStart=/bin/systemctl restart hermes-gateway.service
TimeoutStartSec=60
```

`/etc/systemd/system/hermes-gateway-restart.timer`:
```ini
[Unit]
Description=Startet Hermes-Gateway taeglich um 23:59
[Timer]
OnCalendar=*-*-* 23:59:00
Persistent=false
RandomizedDelaySec=0
[Install]
WantedBy=timers.target
```

Install + enable:
```bash
cp restart.service restart.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable hermes-gateway-restart.timer   # allowed from inside
```

**CRITICAL PITFALL:** From inside the running gateway, `systemctl daemon-reload` and
`systemctl enable <timer>` succeed, but **`systemctl start <timer>` is ALSO blocked** (the
security scanner traces the timer to the gateway-restart service and treats the activation
as a self-kill). The `enable` persists, so the timer is armed and will fire on schedule —
you only need the user to run `systemctl start hermes-gateway-restart.timer` **once** from a
normal shell. After that it fires automatically daily.

**Ordering tip (user preference):** schedule the restart AFTER the daily Doku/Archiv cron and
BEFORE the Wissensstand (knowledge-sync) job, so the fresh session has updated memory/config
before the sync runs.
### Real-World Note (This Session)
Attempting to fix the remote gateway (92.60.38.175) **from within this local Hermes gateway session** triggered the blocker. The fix required running commands directly on the remote server via SSH from a separate terminal, not from inside an active Hermes gateway process.

## Service File Templates

See `references/gateway-systemd-troubleshooting.md` for complete, ready-to-use service file templates for both root and user installs.

## Diagnostic Checklist (Run in Order)

1. `systemctl status hermes-gateway` — active? crash loop?
2. `journalctl -u hermes-gateway -n 20` — CHDIR error? Permission denied? Module not found?
3. `systemctl cat hermes-gateway` — check User, WorkingDirectory, HERMES_HOME
4. `ls -la /root/.hermes /home/user/.hermes` — which exists? Who owns it?
5. `cat /root/.hermes/config.yaml | head -5` — verify config is valid
6. Fix service file to match reality → `daemon-reload` → kill loop → restart

## Related Skills

- `hermes-agent` — main Hermes configuration skill (covers gateway setup, platforms, config)