gb/AGENTS-PROD.md

General Bots Cloud — Production Operations Guide

Infrastructure Overview

• Host OS: Ubuntu 24.04 LTS, LXD (snap)
• SSH: Key auth only, sudoer user in lxd group
• Container engine: LXD with ZFS storage pool

LXC Container Architecture

Container	Purpose	Exposed Ports
<tenant>-proxy	Caddy reverse proxy	80, 443
<tenant>-system	botserver + botui (privileged!)	internal only
<tenant>-alm	Forgejo (ALM/Git)	internal only
<tenant>-alm-ci	Forgejo CI runner	none
<tenant>-email	Stalwart mail server	25,465,587,993,995,143,110
<tenant>-dns	CoreDNS	53
<tenant>-drive	MinIO S3	internal only
<tenant>-tables	PostgreSQL	internal only
<tenant>-table-editor	NocoDB	internal only
<tenant>-webmail	Roundcube	internal only

Key Rules

• <tenant>-system must be privileged (security.privileged: true) — required for botserver to own /opt/gbo/ mounts
• All containers use LXD proxy devices for port forwarding (network forwards don't work when external IP is on host NIC, not bridge)
• Never remove proxy devices for ports: 80, 443, 25, 465, 587, 993, 995, 143, 110, 4190, 53
• CI runner (alm-ci) must NOT have cross-container disk device mounts — deploy via SSH instead

Firewall (host)

• ufw with DEFAULT_FORWARD_POLICY=ACCEPT (needed for container internet)
• LXD forward rule must persist via systemd service
• fail2ban on host (SSH jail) and in email container (mail jail)

---

🔧 Common Production Issues & Fixes

Issue: Valkey/Redis Connection Timeout

Symptom: botserver logs show Connection timed out (os error 110) when connecting to cache at localhost:6379

Root Cause: iptables DROP rule for port 6379 blocks loopback traffic because no ACCEPT rule for lo interface exists before the DROP rules.

Fix:

# Insert loopback ACCEPT at top of INPUT chain
lxc exec <tenant>-system -- iptables -I INPUT 1 -i lo -j ACCEPT

# Persist the rule
lxc exec <tenant>-system -- bash -c 'iptables-save > /etc/iptables/rules.v4'

# Verify Valkey responds
lxc exec <tenant>-system -- /opt/gbo/bin/botserver-stack/bin/cache/bin/valkey-cli ping
# Should return: PONG

# Restart botserver to pick up working cache
lxc exec <tenant>-system -- systemctl restart system.service ui.service

Prevention: Always ensure loopback ACCEPT rule is at the top of iptables INPUT chain before any DROP rules.

Issue: Suggestions Not Showing in Frontend

Symptom: Bot's start.bas has ADD_SUGGESTION_TOOL calls but suggestions don't appear in the UI.

Diagnosis:

# Get bot ID
lxc exec <tenant>-system -- /opt/gbo/bin/botserver-stack/bin/tables/bin/psql -h localhost -U gbuser -d botserver -t -c "SELECT id, name FROM bots WHERE name = 'botname';"

# Check if suggestions exist in cache with correct bot_id
lxc exec <tenant>-system -- /opt/gbo/bin/botserver-stack/bin/cache/bin/valkey-cli --scan --pattern "suggestions:<bot_id>:*"

# If no keys found, check logs for wrong bot_id being used
lxc exec <tenant>-system -- grep "Adding suggestion to Redis key" /opt/gbo/logs/error.log | tail -5

Fix: This was a code bug where suggestions were stored with user_id instead of bot_id. After deploying the fix:

1. Wait for CI/CD to build and deploy new binary (~10 minutes)
2. Service auto-restarts on binary update
3. Test by opening a new session (old sessions may have stale keys)

Deployment Workflow:

# 1. Fix code in dev environment
# 2. Commit and push to ALM
cd botserver && git push alm main

# 3. Update root gb repository
cd .. && git add botserver && git commit -m "Update submodule" && git push alm main

# 4. Wait 10 minutes for CI/CD pipeline
# 5. Verify deployment
lxc exec <tenant>-system -- ls -lh /opt/gbo/bin/botserver
lxc exec <tenant>-system -- systemctl status system.service

# 6. Test the fix
# Open new session at https://chat.<domain>/<botname>
# Suggestions should now appear

---

⚠️ Caddy Config — CRITICAL RULES

NEVER replace the Caddyfile with a minimal/partial config. The full config has ~25 vhosts. If you only see 1-2 vhosts, you are looking at a broken/partial config.

Before ANY change:

1. Backup: cp /opt/gbo/conf/config /opt/gbo/conf/config.bak-$(date +%Y%m%d%H%M)
2. Validate: caddy validate --config /opt/gbo/conf/config --adapter caddyfile
3. Reload (not restart): caddy reload --config /opt/gbo/conf/config --adapter caddyfile

Caddy storage must be explicitly set in the global block, otherwise Caddy uses ~/.local/share/caddy and loses existing certificates on restart:

{
    storage file_system {
        root /opt/gbo/data/caddy
    }
}

Dead domains cause ERR_SSL_PROTOCOL_ERROR — if a domain in the Caddyfile has no DNS record, Caddy loops trying to get a certificate and pollutes TLS state. Remove dead domains immediately.

After removing domains from config, restart Caddy (not just reload) to clear in-memory ACME state from old domains.

---

botserver / botui

• botserver: system.service on port 5858
• botui: ui.service on port 5859
• BOTSERVER_URL in ui.service must point to http://localhost:5858 (not HTTPS external URL) — using external URL causes WebSocket disconnect before TALK executes
• Valkey/Redis bound to 127.0.0.1:6379 — iptables rules must allow loopback on this port or suggestions/cache won't work
• Vault unseal keys stored in /opt/gbo/bin/botserver-stack/conf/vault/init.json (production only - never commit to git)

iptables loopback rule (required)

Internal services (Valkey, MinIO) are protected by DROP rules. Loopback must be explicitly allowed before the DROP rules:

iptables -I INPUT -i lo -j ACCEPT
iptables -A INPUT -p tcp --dport 6379 -j DROP  # external only

---

CoreDNS Hardening

Corefile must include acl plugin to prevent DNS amplification attacks:

zone.example.com:53 {
    file /opt/gbo/data/zone.example.com.zone
    acl {
        allow type ANY net 10.0.0.0/8 127.0.0.0/8
        allow type A net 0.0.0.0/0
        allow type AAAA net 0.0.0.0/0
        allow type MX net 0.0.0.0/0
        block
    }
    cache
    errors
}

Reload with SIGHUP: pkill -HUP coredns

---

fail2ban in Proxy Container

Proxy container needs its own fail2ban for HTTP flood protection:

• Filter: match 4xx errors from Caddy JSON access log
• Jail: caddy-http-flood — 100 errors/60s → ban 1h
• Disable default sshd jail (no SSH in proxy container) via jail.d/defaults-debian.conf

---

CI/CD (Forgejo Runner)

• Runner container must have no cross-container disk mounts
• Deploy via SSH: scp binary <system-container>:/opt/gbo/bin/botserver
• SSH key from runner → system container must be pre-authorized
• sccache + cargo registry cache accumulates — daily cleanup cron required
• ZFS snapshots of CI container can be huge if taken while cross-mounts were active — delete stale snapshots after removing mounts

---

ZFS Disk Space

• Check snapshots: zfs list -t snapshot -o name,used | sort -k2 -rh
• Snapshots retain data from device mounts at time of snapshot — removing mounts doesn't free space until snapshot is deleted
• Delete snapshot: zfs destroy <pool>/containers/<name>@<snapshot>
• Daily rolling snapshots (7-day retention) via cron

---

Git Workflow

Push to both remotes after every change:

cd <submodule>
git push origin main
git push alm main
cd ..
git add <submodule>
git commit -m "Update submodule"
git push alm main

Failure to push the root gb repo will not trigger CI/CD pipelines.

---

Useful Commands

# Check all containers
lxc list

# Check disk device mounts per container
for c in $(lxc list --format csv -c n); do
  devices=$(lxc config device show $c | grep 'type: disk' | grep -v 'pool:' | wc -l)
  [ $devices -gt 0 ] && echo "=== $c ===" && lxc config device show $c | grep -E 'source:|path:' | grep -v pool
done

# Tail Caddy errors
lxc exec <tenant>-proxy -- tail -f /opt/gbo/logs/access.log

# Restart botserver + botui
lxc exec <tenant>-system -- systemctl restart system.service ui.service

# Check iptables in system container
lxc exec <tenant>-system -- iptables -L -n | grep -E 'DROP|ACCEPT.*lo'

# ZFS snapshot usage
zfs list -t snapshot -o name,used | sort -k2 -rh | head -20

# Unseal Vault (use actual unseal key from init.json)
lxc exec <tenant>-system -- bash -c "
  export VAULT_ADDR=https://127.0.0.1:8200 VAULT_SKIP_VERIFY=true
  /opt/gbo/bin/botserver-stack/bin/vault/vault operator unseal \$UNSEAL_KEY
"