gb/AGENTS-PROD.md

General Bots Cloud — Production Operations Guide

Infrastructure Overview

• Host OS: Ubuntu 24.04 LTS
• SSH: Key auth only
• Container engine: LXC (Linux Containers)
• Tenant: pragmatismo

---

PostgreSQL Container (tables) Management

Container Info

• Container Name: pragmatismo-tables
• Database: PostgreSQL 14
• Access: Inside LXC container on port 5432
• Data Location: /opt/gbo/tables/ inside container

Common Operations

# Check container status
lxc list | grep pragmatismo-tables

# Exec into container
lxc exec pragmatismo-tables -- bash

# Check PostgreSQL status
lxc exec pragmatismo-tables -- pg_isready

# Query version
lxc exec pragmatismo-tables -- psql -U postgres -c 'SELECT version();'

# Restart PostgreSQL
lxc exec pragmatismo-tables -- systemctl restart postgresql

Backup PostgreSQL

# Create database dump
lxc exec pragmatismo-tables -- pg_dump -U postgres -F c -f /tmp/backup.dump pragmatismo

# Copy backup to host
lxc file pull pragmatismo-tables/tmp/backup.dump ~/backups/postgresql-$(date +%Y%m%d).dump

---

LXC Container Status (82.29.59.188)

Container	Status	Purpose	Notes
pragmatismo-dns	✅ RUNNING	CoreDNS	DNS server
pragmatismo-proxy	✅ RUNNING	Caddy	Reverse proxy on :80/:443
pragmatismo-tables	✅ RUNNING	PostgreSQL 14	Database
pragmatismo-system	✅ RUNNING	botserver	Bot system
pragmatismo-email	✅ RUNNING	Stalwart	Email server
pragmatismo-webmail	✅ RUNNING	Roundcube	Webmail interface
pragmatismo-alm	✅ RUNNING	Forgejo	Git/Code ALM
pragmatismo-alm-ci	✅ RUNNING	Runner	CI/CD runner
pragmatismo-drive	✅ RUNNING	MinIO	S3-compatible storage

---

LXC Container Management

Common Commands

# List all containers
lxc list

# Show container details
lxc info pragmatismo-dns

# Exec into container
lxc exec pragmatismo-dns -- bash
lxc exec pragmatismo-dns -- /bin/sh

# Start/Stop/Restart containers
lxc start pragmatismo-dns
lxc stop pragmatismo-dns
lxc restart pragmatismo-dns

# View logs
lxc log pragmatismo-dns
lxc log pragmatismo-dns --show-log

# Copy files to/from container
lxc file push localfile pragmatismo-dns/opt/gbo/conf/
lxc file pull pragmatismo-dns/opt/gbo/logs/output.log .

---

SSH Setup

On Production Server

# Add SSH public key for access
mkdir -p /root/.ssh
echo "<your-public-key>" >> /root/.ssh/authorized_keys
chmod 700 /root/.ssh
chmod 600 /root/.ssh/authorized_keys

From This Machine

# SSH key for passwordless access to production
ssh-copy-id root@82.29.59.188

# Test connection
ssh root@82.29.59.188 "lxc list"

---

LXC Container Data Sync (rsync)

Prerequisites

• Source: SSH access as root to source host
• Network: Same LAN or VPN connection between hosts

Copy Individual Container Data

# Copy specific container data (e.g., dns container)
sudo rsync -avz --progress -e ssh \
    root@<source-host>:/opt/gbo/tenants/pragmatismo/dns/ \
    /opt/gbo/dns/

# Copy tables (PostgreSQL data)
sudo rsync -avz --progress -e ssh \
    root@<source-host>:/opt/gbo/tenants/pragmatismo/tables/ \
    /opt/gbo/tables/

# Copy drive (MinIO/S3 data)
sudo rsync -avz --progress -e ssh \
    root@<source-host>:/opt/gbo/tenants/pragmatismo/drive/ \
    /opt/gbo/drive/

Dry Run Before Copy

# Preview what will be copied (no changes made)
sudo rsync -avzn --progress -e ssh \
    root@<source-host>:/opt/gbo/tenants/pragmatismo/dns/ \
    /opt/gbo/dns/

Exclude Patterns

# Exclude logs, temp files, and system directories
sudo rsync -avz --progress -e ssh \
    --exclude='*.log' \
    --exclude='*.tmp' \
    --exclude='.git/' \
    --exclude='__pycache__/' \
    root@<source-host>:/opt/gbo/tenants/pragmatismo/ \
    /opt/gbo/

---

DNS Management (pragmatismo-dns)

Container Info

• Container Name: pragmatismo-dns
• Service: CoreDNS
• Access: Inside LXC container on port 53
• Config: /opt/gbo/dns/conf/Corefile inside container

Common Operations

# Check container status
lxc list | grep pragmatismo-dns

# View logs
lxc log pragmatismo-dns
lxc log pragmatismo-dns --show-log

# Restart DNS
lxc restart pragmatismo-dns

# Exec into container
lxc exec pragmatismo-dns -- /bin/sh

# Test DNS
dig @localhost pragmatismo.com.br SOA +short
dig @localhost ddsites.com.br SOA +short
dig @localhost chat.pragmatismo.com.br A +short

Update DNS Records

# Edit Corefile inside container
lxc exec pragmatismo-dns -- nano /opt/gbo/dns/conf/Corefile

# Edit zone files inside container
lxc exec pragmatismo-dns -- nano /opt/gbo/dns/data/pragmatismo.com.br.zone

# Restart CoreDNS after changes
lxc exec pragmatismo-dns -- systemctl restart coredns
# or if running as systemd service:
lxc restart pragmatismo-dns

---

DNS Cutover

Update NS/A records at your registrar:

• NS records: Point to ns1.pragmatismo.com.br / ns2.pragmatismo.com.br
• A records: Update to 82.29.59.188

---

Proxy/Caddy Management (pragmatismo-proxy)

Container Info

• Container Name: pragmatismo-proxy
• Service: Caddy reverse proxy
• Access: Inside LXC container on ports 80/443
• Config: /opt/gbo/proxy/conf/config inside container

Common Operations

# Check container status
lxc list | grep pragmatismo-proxy

# View logs
lxc log pragmatismo-proxy
lxc log pragmatismo-proxy --show-log

# Restart proxy
lxc restart pragmatismo-proxy

# Exec into container
lxc exec pragmatismo-proxy -- bash

Update Caddy Configuration

# Edit Caddyfile inside container
lxc exec pragmatismo-proxy -- nano /opt/gbo/proxy/conf/config

# Validate configuration
lxc exec pragmatismo-proxy -- caddy validate --config /opt/gbo/proxy/conf/config --adapter caddyfile

# Reload Caddy after changes
lxc exec pragmatismo-proxy -- caddy reload --config /opt/gbo/proxy/conf/config --adapter caddyfile

# Or restart the entire container
lxc restart pragmatismo-proxy

---

⚠️ Caddy Config — CRITICAL RULES

NEVER replace the Caddyfile with a minimal/partial config. The full config has ~25 vhosts.

Before ANY change:

1. Backup: lxc exec pragmatismo-proxy -- cp /opt/gbo/proxy/conf/config /opt/gbo/proxy/conf/config.bak-$(date +%Y%m%d%H%M)
2. Validate: lxc exec pragmatismo-proxy -- caddy validate --config /opt/gbo/proxy/conf/config --adapter caddyfile
3. Reload: lxc exec pragmatismo-proxy -- caddy reload --config /opt/gbo/proxy/conf/config --adapter caddyfile

Caddy storage must be explicitly set:

{
    storage file_system {
        root /opt/gbo/data/caddy
    }
}

---

---

Backup Strategy

Location: Host machine /opt/gbo/backups/

Format: container-name-YYYYMMDD-HHMM.tar.gz

Schedule: Twice daily (6am and 6pm) via cron

Backup Individual Container

# Backup container to tar.gz
lxc snapshot pragmatismo-tables backup-$(date +%Y%m%d-%H%M)
lxc publish pragmatismo-tables/backup-$(date +%Y%m%d-%H%M) --alias tables-backup-$(date +%Y%m%d-%H%M)
lxc image export tables-backup-$(date +%Y%m%d-%H%M) /opt/gbo/backups/tables-$(date +%Y%m%d-%H%M).tar.gz

# List backups
ls -la /opt/gbo/backups/

# Restore from backup
lxc image import /opt/gbo/backups/tables-20260326-1227.tar.gz --alias tables-restore
lxc init tables-restore pragmatismo-tables-restored
lxc start pragmatismo-tables-restored

Backup PostgreSQL Database

# Create database dump
lxc exec pragmatismo-tables -- pg_dump -U postgres -F c -f /tmp/backup.dump pragmatismo

# Copy backup to host
lxc file pull pragmatismo-tables/tmp/backup.dump /opt/gbo/backups/postgresql-$(date +%Y%m%d).dump

Retention: Last 7 days

---

Git Workflow

Push to both remotes:

cd <submodule>
git push origin main
git push alm main
cd ..
git add <submodule>
git commit -m "Update submodule"
git push alm main

---

Git Workflow

Push to both remotes:

cd <submodule>
git push origin main
git push alm main
cd ..
git add <submodule>
git commit -m "Update submodule"
git push alm main

---

Troubleshooting Common Issues

Container Not Starting

# Check container status
lxc list

# View logs for startup issues
lxc log pragmatismo-<container-name>
lxc log pragmatismo-<container-name> --show-log

# Check resource usage
lxc info pragmatismo-<container-name>

# Try manual start
lxc start pragmatismo-<container-name>

Network Issues

# Check container IP
lxc list -c n,4

# Test connectivity from host
lxc exec pragmatismo-dns -- ping -c 3 8.8.8.8

# Check DNS resolution
lxc exec pragmatismo-system -- dig pragmatismo.com.br

Service Issues Inside Container

# Check service status inside container
lxc exec pragmatismo-<container> -- systemctl status <service>

# Restart service inside container
lxc exec pragmatismo-<container> -- systemctl restart <service>

# View service logs
lxc exec pragmatismo-<container> -- journalctl -u <service> -f

---

(End of file)