mesh-configs/skills/devops/hermes-gateway-systemd/SKILL.md

name, description, version, author, license, platforms, metadata

name	description	version	author	license	platforms	metadata
hermes-gateway-systemd	Troubleshoot and fix Hermes Agent Gateway systemd service issues — crash loops, working directory mismatches, permission problems, and restart blockers.	1.0.0	Hermes Agent	MIT	linux	hermes tags related_skills hermes gateway systemd troubleshooting service crash-loop hermes-agent

Hermes Gateway systemd Service Troubleshooting

This skill covers diagnosing and fixing common Hermes Gateway systemd service problems — especially the infamous crash loop caused by WorkingDirectory / User mismatches.

Scope

• Hermes Gateway running as a systemd service (user or system)
• Crash loops (status 200/CHDIR, rapid restarts)
• Permission issues (config owned by root, service runs as user, or vice versa)
• Restart blockers (cannot run systemctl restart from inside gateway session)

Quick Diagnosis

# 1. Check service status
systemctl status hermes-gateway --no-pager

# 2. View crash loop logs
journalctl -u hermes-gateway -n 30 --no-pager

# 3. Inspect service file
systemctl cat hermes-gateway

# 4. Verify config location matches service User/WorkingDirectory
ls -la /root/.hermes/config.yaml 2>/dev/null
ls -la /home/<user>/.hermes/config.yaml 2>/dev/null

Common Failure: CHDIR Crash Loop

Symptom

hermes-gateway.service: Changing to the requested working directory failed: No such file or directory
hermes-gateway.service: Main process exited, code=exited, status=200/CHDIR
Restart counter increments every 5 seconds (e.g., 20,000+ restarts)

Root Cause

Service file specifies User=X and WorkingDirectory=/home/X/.hermes but:

• Config actually lives in /root/.hermes (installed as root)
• OR /home/X/.hermes doesn't exist

Fix

Align User, WorkingDirectory, HOME, HERMES_HOME to where config actually lives.

If config is in /root/.hermes (root install):

User=root
Group=root
WorkingDirectory=/root/.hermes
Environment="HOME=/root"
Environment="HERMES_HOME=/root/.hermes"

If config is in /home/user/.hermes (user install):

User=user
Group=user
WorkingDirectory=/home/user/.hermes
Environment="HOME=/home/user"
Environment="HERMES_HOME=/home/user/.hermes"

Then:

systemctl daemon-reload
pkill -9 -f 'hermes_cli.main gateway run'  # kill crash loop
systemctl start hermes-gateway
sleep 3
systemctl status hermes-gateway --no-pager

Real-World Example (This Session)

• Config location: /root/.hermes (owned by root, installed via hermes setup)
• Broken service file: User=mehlzoerwer, WorkingDirectory=/home/mehlzoerwer/.hermes (directory didn't exist)
• Symptom: 20,731+ restart loop, status 200/CHDIR
• Fix applied: Changed service to User=root, WorkingDirectory=/root/.hermes, HERMES_HOME=/root/.hermes
• Result: Service started cleanly, Telegram connected, startup notification sent

Blocker: Cannot Restart from Inside Gateway

Error

Blocked: cannot restart or stop the gateway from inside the gateway process.
The gateway would kill this command before it could complete (SIGTERM propagates to child processes).
Run `hermes gateway restart` from a separate shell outside the running gateway.

Cause

Hermes detects it's running inside its own gateway process and blocks process-termination commands to prevent self-kill.

Workarounds

Method	When to Use
New SSH session	Always works — run systemctl restart hermes-gateway from fresh terminal
hermes gateway restart	If gateway is responsive (slash command in Telegram/CLI)
at job	`echo 'systemctl restart hermes-gateway'
Direct PID kill from outside	ssh root@host 'pkill -9 -f hermes_cli.main'
tmux/screen on server	Run commands in detached session

Scheduled Auto-Restart (systemd Timer) — durable pattern

To restart the gateway on a recurring schedule (e.g. nightly 23:59 to refresh memory/config) without manual intervention, install a pair of units from a shell OUTSIDE the gateway:

/etc/systemd/system/hermes-gateway-restart.service:

[Unit]
Description=Naechtlicher Hermes-Gateway-Neustart
[Service]
Type=oneshot
ExecStart=/bin/systemctl restart hermes-gateway.service
TimeoutStartSec=60

/etc/systemd/system/hermes-gateway-restart.timer:

[Unit]
Description=Startet Hermes-Gateway taeglich um 23:59
[Timer]
OnCalendar=*-*-* 23:59:00
Persistent=false
RandomizedDelaySec=0
[Install]
WantedBy=timers.target

Install + enable:

cp restart.service restart.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable hermes-gateway-restart.timer   # allowed from inside

CRITICAL PITFALL: From inside the running gateway, systemctl daemon-reload and systemctl enable <timer> succeed, but systemctl start <timer> is ALSO blocked (the security scanner traces the timer to the gateway-restart service and treats the activation as a self-kill). The enable persists, so the timer is armed and will fire on schedule — you only need the user to run systemctl start hermes-gateway-restart.timer once from a normal shell. After that it fires automatically daily.

Ordering tip (user preference): schedule the restart AFTER the daily Doku/Archiv cron and BEFORE the Wissensstand (knowledge-sync) job, so the fresh session has updated memory/config before the sync runs.

Real-World Note (This Session)

Attempting to fix the remote gateway (92.60.38.175) from within this local Hermes gateway session triggered the blocker. The fix required running commands directly on the remote server via SSH from a separate terminal, not from inside an active Hermes gateway process.

Service File Templates

See references/gateway-systemd-troubleshooting.md for complete, ready-to-use service file templates for both root and user installs.

Diagnostic Checklist (Run in Order)

1. systemctl status hermes-gateway — active? crash loop?
2. journalctl -u hermes-gateway -n 20 — CHDIR error? Permission denied? Module not found?
3. systemctl cat hermes-gateway — check User, WorkingDirectory, HERMES_HOME
4. ls -la /root/.hermes /home/user/.hermes — which exists? Who owns it?
5. cat /root/.hermes/config.yaml | head -5 — verify config is valid
6. Fix service file to match reality → daemon-reload → kill loop → restart

Related Skills

• hermes-agent — main Hermes configuration skill (covers gateway setup, platforms, config)