201 lines
6.4 KiB
Markdown
201 lines
6.4 KiB
Markdown
# Hermes Gateway systemd Service Troubleshooting — Reference Detail
|
|
|
|
## Common Crash Loop: "Changing to the requested working directory failed"
|
|
|
|
**Symptom:** Service restarts every 5 seconds, logs show:
|
|
```
|
|
hermes-gateway.service: Changing to the requested working directory failed: No such file or directory
|
|
hermes-gateway.service: Main process exited, code=exited, status=200/CHDIR
|
|
```
|
|
|
|
**Root Cause:** `WorkingDirectory` in service file points to a path that doesn't exist for the configured `User`.
|
|
|
|
**Typical Scenario:**
|
|
- Config lives in `/root/.hermes` (installed as root)
|
|
- Service file has `User=mehlzoerwer` and `WorkingDirectory=/home/mehlzoerwer/.hermes`
|
|
- But `/home/mehlzoerwer/.hermes` doesn't exist
|
|
|
|
**Fix:** Align `User`, `WorkingDirectory`, `HOME`, `HERMES_HOME` to where config actually lives.
|
|
|
|
```ini
|
|
# If config is in /root/.hermes:
|
|
User=root
|
|
Group=root
|
|
WorkingDirectory=/root/.hermes
|
|
Environment="HOME=/root"
|
|
Environment="HERMES_HOME=/root/.hermes"
|
|
|
|
# If config is in /home/user/.hermes:
|
|
User=user
|
|
Group=user
|
|
WorkingDirectory=/home/user/.hermes
|
|
Environment="HOME=/home/user"
|
|
Environment="HERMES_HOME=/home/user/.hermes"
|
|
```
|
|
|
|
Then:
|
|
```bash
|
|
systemctl daemon-reload
|
|
pkill -9 -f 'hermes_cli.main gateway run' # kill crash loop
|
|
systemctl start hermes-gateway
|
|
systemctl status hermes-gateway --no-pager
|
|
```
|
|
|
|
---
|
|
|
|
## Cannot Restart from Inside Gateway Session
|
|
|
|
**Problem:** Running `systemctl restart hermes-gateway` (or `pkill`/`kill`) from within an active Hermes gateway session fails with:
|
|
```text
|
|
Blocked: cannot restart or stop the gateway from inside the gateway process.
|
|
The gateway would kill this command before it could complete (SIGTERM propagates to child processes).
|
|
Run `hermes gateway restart` from a separate shell outside the running gateway.
|
|
```
|
|
|
|
**Reason:** Hermes detects it's running inside its own gateway process and blocks process termination commands to prevent self-kill.
|
|
|
|
**Workarounds:**
|
|
1. **Run commands from a separate shell** (new SSH session, tmux pane, or `at` job)
|
|
2. **Use `hermes gateway restart`** from within the gateway (if gateway is responsive)
|
|
3. **Kill via PID directly from outside**: `ssh root@host 'pkill -9 -f hermes_cli.main'`
|
|
4. **Schedule restart**: `echo 'systemctl restart hermes-gateway' | at now + 1 minute`
|
|
|
|
---
|
|
|
|
## Telegram Polling Conflict (409 Error)
|
|
|
|
**Symptom:** When starting the gateway, logs show:
|
|
```text
|
|
WARNING hermes_plugins.telegram_platform.adapter: [Telegram] Telegram polling conflict (1/5) — previous session still held open on Telegram's servers. Waiting 20s for it to expire. Error: Conflict: terminated by other getUpdates request; make sure that only one bot instance is running
|
|
INFO hermes_plugins.telegram_platform.adapter: [Telegram] Telegram polling resumed after conflict retry 1/5
|
|
```
|
|
|
|
**Cause:** A previous gateway instance (or another process) still holds an active Long Polling connection to Telegram. Telegram only allows ONE active `getUpdates` connection per bot token.
|
|
|
|
**Built-in Resolution:** Hermes Gateway (since recent versions) automatically retries up to 5 times with exponential backoff (20s, 40s, 60s, 80s, 100s). The old connection expires on Telegram's side after ~30-60 seconds of inactivity.
|
|
|
|
**When to Act:**
|
|
- If conflict persists after all 5 retries → another process is actively polling (check for duplicate gateway services, cron jobs, or manual runs)
|
|
- Fix: `pkill -9 -f 'hermes_cli.main gateway run'` then restart cleanly
|
|
|
|
**Verification:**
|
|
```bash
|
|
# Check for duplicate gateway processes
|
|
ps aux | grep 'hermes_cli.main gateway run'
|
|
|
|
# Should only show ONE process (the systemd one)
|
|
```
|
|
|
|
---
|
|
|
|
## Gateway Startup Notification
|
|
|
|
**Behavior:** On successful Telegram connection, the gateway automatically sends a startup notification to the configured `TELEGRAM_HOME_CHANNEL` (from `.env` or config).
|
|
|
|
**Log entry:**
|
|
```text
|
|
INFO gateway.run: Sent home-channel startup notification to telegram:6417835766
|
|
```
|
|
|
|
**If notification not received:**
|
|
1. Verify `TELEGRAM_HOME_CHANNEL` and `TELEGRAM_ALLOWED_USERS` in `.env` match your chat ID
|
|
2. Check bot has permission to message that chat (not blocked, not a channel without bot admin)
|
|
3. Check gateway logs for delivery errors after "Sent home-channel startup notification"
|
|
|
|
---
|
|
|
|
## Diagnostic Checklist
|
|
|
|
```bash
|
|
# 1. Check service file
|
|
systemctl cat hermes-gateway
|
|
|
|
# 2. Verify paths exist for the configured User
|
|
ls -la /home/user/.hermes # or /root/.hermes
|
|
|
|
# 3. Check current crash loop
|
|
journalctl -u hermes-gateway -n 20 --no-pager
|
|
|
|
# 4. Verify Hermes config location
|
|
cat /root/.hermes/config.yaml | head -5
|
|
# or
|
|
cat /home/user/.hermes/config.yaml | head -5
|
|
|
|
# 5. Check which user owns the config
|
|
ls -la /root/.hermes/config.yaml
|
|
ls -la /home/user/.hermes/config.yaml 2>/dev/null
|
|
```
|
|
|
|
---
|
|
|
|
## Service File Template (Root Install)
|
|
|
|
```ini
|
|
[Unit]
|
|
Description=Hermes Agent Gateway - Messaging Platform Integration
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
StartLimitIntervalSec=0
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=root
|
|
Group=root
|
|
ExecStart=/usr/local/lib/hermes-agent/venv/bin/python -m hermes_cli.main gateway run
|
|
WorkingDirectory=/root/.hermes
|
|
Environment="HOME=/root"
|
|
Environment="USER=root"
|
|
Environment="LOGNAME=root"
|
|
Environment="PATH=/usr/local/lib/hermes-agent/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
|
Environment="VIRTUAL_ENV=/usr/local/lib/hermes-agent/venv"
|
|
Environment="HERMES_HOME=/root/.hermes"
|
|
Restart=always
|
|
RestartSec=5
|
|
RestartForceExitStatus=75
|
|
KillMode=mixed
|
|
KillSignal=SIGTERM
|
|
ExecReload=/bin/kill -USR1 $MAINPID
|
|
TimeoutStopSec=210
|
|
StandardOutput=journal
|
|
StandardError=journal
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
---
|
|
|
|
## Service File Template (User Install)
|
|
|
|
```ini
|
|
[Unit]
|
|
Description=Hermes Agent Gateway - Messaging Platform Integration
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
StartLimitIntervalSec=0
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=<username>
|
|
Group=<username>
|
|
ExecStart=/usr/local/lib/hermes-agent/venv/bin/python -m hermes_cli.main gateway run
|
|
WorkingDirectory=/home/<username>/.hermes
|
|
Environment="HOME=/home/<username>"
|
|
Environment="USER=<username>"
|
|
Environment="LOGNAME=<username>"
|
|
Environment="PATH=/usr/local/lib/hermes-agent/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
|
Environment="VIRTUAL_ENV=/usr/local/lib/hermes-agent/venv"
|
|
Environment="HERMES_HOME=/home/<username>/.hermes"
|
|
Restart=always
|
|
RestartSec=5
|
|
RestartForceExitStatus=75
|
|
KillMode=mixed
|
|
KillSignal=SIGTERM
|
|
ExecReload=/bin/kill -USR1 $MAINPID
|
|
TimeoutStopSec=210
|
|
StandardOutput=journal
|
|
StandardError=journal
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
``` |