Hermes Gateway systemd Service Troubleshooting — Reference Detail

Common Crash Loop: "Changing to the requested working directory failed"

Symptom: Service restarts every 5 seconds, logs show:

hermes-gateway.service: Changing to the requested working directory failed: No such file or directory
hermes-gateway.service: Main process exited, code=exited, status=200/CHDIR

Root Cause: WorkingDirectory in service file points to a path that doesn't exist for the configured User.

Typical Scenario:

Config lives in /root/.hermes (installed as root)
Service file has User=mehlzoerwer and WorkingDirectory=/home/mehlzoerwer/.hermes
But /home/mehlzoerwer/.hermes doesn't exist

Fix: Align User, WorkingDirectory, HOME, HERMES_HOME to where config actually lives.

# If config is in /root/.hermes:
User=root
Group=root
WorkingDirectory=/root/.hermes
Environment="HOME=/root"
Environment="HERMES_HOME=/root/.hermes"

# If config is in /home/user/.hermes:
User=user
Group=user
WorkingDirectory=/home/user/.hermes
Environment="HOME=/home/user"
Environment="HERMES_HOME=/home/user/.hermes"

Then:

systemctl daemon-reload
pkill -9 -f 'hermes_cli.main gateway run'  # kill crash loop
systemctl start hermes-gateway
systemctl status hermes-gateway --no-pager

Cannot Restart from Inside Gateway Session

Problem: Running systemctl restart hermes-gateway (or pkill/kill) from within an active Hermes gateway session fails with:

Blocked: cannot restart or stop the gateway from inside the gateway process.
The gateway would kill this command before it could complete (SIGTERM propagates to child processes).
Run `hermes gateway restart` from a separate shell outside the running gateway.

Reason: Hermes detects it's running inside its own gateway process and blocks process termination commands to prevent self-kill.

Workarounds:

Run commands from a separate shell (new SSH session, tmux pane, or at job)
Use hermes gateway restart from within the gateway (if gateway is responsive)
Kill via PID directly from outside: ssh root@host 'pkill -9 -f hermes_cli.main'
Schedule restart: echo 'systemctl restart hermes-gateway' | at now + 1 minute

Telegram Polling Conflict (409 Error)

Symptom: When starting the gateway, logs show:

WARNING hermes_plugins.telegram_platform.adapter: [Telegram] Telegram polling conflict (1/5) — previous session still held open on Telegram's servers. Waiting 20s for it to expire. Error: Conflict: terminated by other getUpdates request; make sure that only one bot instance is running
INFO hermes_plugins.telegram_platform.adapter: [Telegram] Telegram polling resumed after conflict retry 1/5

Cause: A previous gateway instance (or another process) still holds an active Long Polling connection to Telegram. Telegram only allows ONE active getUpdates connection per bot token.

Built-in Resolution: Hermes Gateway (since recent versions) automatically retries up to 5 times with exponential backoff (20s, 40s, 60s, 80s, 100s). The old connection expires on Telegram's side after ~30-60 seconds of inactivity.

When to Act:

If conflict persists after all 5 retries → another process is actively polling (check for duplicate gateway services, cron jobs, or manual runs)
Fix: pkill -9 -f 'hermes_cli.main gateway run' then restart cleanly

Verification:

# Check for duplicate gateway processes
ps aux | grep 'hermes_cli.main gateway run'

# Should only show ONE process (the systemd one)

Gateway Startup Notification

Behavior: On successful Telegram connection, the gateway automatically sends a startup notification to the configured TELEGRAM_HOME_CHANNEL (from .env or config).

Log entry:

INFO gateway.run: Sent home-channel startup notification to telegram:6417835766

If notification not received:

Verify TELEGRAM_HOME_CHANNEL and TELEGRAM_ALLOWED_USERS in .env match your chat ID
Check bot has permission to message that chat (not blocked, not a channel without bot admin)
Check gateway logs for delivery errors after "Sent home-channel startup notification"

Diagnostic Checklist

# 1. Check service file
systemctl cat hermes-gateway

# 2. Verify paths exist for the configured User
ls -la /home/user/.hermes   # or /root/.hermes

# 3. Check current crash loop
journalctl -u hermes-gateway -n 20 --no-pager

# 4. Verify Hermes config location
cat /root/.hermes/config.yaml | head -5
# or
cat /home/user/.hermes/config.yaml | head -5

# 5. Check which user owns the config
ls -la /root/.hermes/config.yaml
ls -la /home/user/.hermes/config.yaml 2>/dev/null

Service File Template (Root Install)

[Unit]
Description=Hermes Agent Gateway - Messaging Platform Integration
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=0

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/lib/hermes-agent/venv/bin/python -m hermes_cli.main gateway run
WorkingDirectory=/root/.hermes
Environment="HOME=/root"
Environment="USER=root"
Environment="LOGNAME=root"
Environment="PATH=/usr/local/lib/hermes-agent/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Environment="VIRTUAL_ENV=/usr/local/lib/hermes-agent/venv"
Environment="HERMES_HOME=/root/.hermes"
Restart=always
RestartSec=5
RestartForceExitStatus=75
KillMode=mixed
KillSignal=SIGTERM
ExecReload=/bin/kill -USR1 $MAINPID
TimeoutStopSec=210
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Service File Template (User Install)

[Unit]
Description=Hermes Agent Gateway - Messaging Platform Integration
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=0

[Service]
Type=simple
User=<username>
Group=<username>
ExecStart=/usr/local/lib/hermes-agent/venv/bin/python -m hermes_cli.main gateway run
WorkingDirectory=/home/<username>/.hermes
Environment="HOME=/home/<username>"
Environment="USER=<username>"
Environment="LOGNAME=<username>"
Environment="PATH=/usr/local/lib/hermes-agent/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Environment="VIRTUAL_ENV=/usr/local/lib/hermes-agent/venv"
Environment="HERMES_HOME=/home/<username>/.hermes"
Restart=always
RestartSec=5
RestartForceExitStatus=75
KillMode=mixed
KillSignal=SIGTERM
ExecReload=/bin/kill -USR1 $MAINPID
TimeoutStopSec=210
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

6.4 KiB Raw Blame History