docs: add prod.md
This commit is contained in:
parent
82f5f24b44
commit
f9aa5dacd0
1 changed files with 259 additions and 0 deletions
259
prompts/prod.md
Normal file
259
prompts/prod.md
Normal file
|
|
@ -0,0 +1,259 @@
|
||||||
|
# Production Environment Guide
|
||||||
|
|
||||||
|
## Infrastructure
|
||||||
|
|
||||||
|
### Servers
|
||||||
|
|
||||||
|
| Host | IP | Purpose |
|
||||||
|
|------|-----|---------|
|
||||||
|
| `system` | `10.157.134.196` | Main botserver + botui container |
|
||||||
|
| `alm-ci` | `10.157.134.200` | CI/CD runner (Forgejo Actions) |
|
||||||
|
| `alm` | `10.157.134.34` | Forgejo git server |
|
||||||
|
| `dns` | `10.157.134.214` | DNS container |
|
||||||
|
| `drive` | `10.157.134.206` | Drive storage |
|
||||||
|
| `email` | `10.157.134.40` | Email service |
|
||||||
|
| `proxy` | `10.157.134.241` | Reverse proxy |
|
||||||
|
| `tables` | `10.157.134.174` | PostgreSQL |
|
||||||
|
| `table-editor` | `10.157.134.184` | Table editor |
|
||||||
|
| `webmail` | `10.157.134.86` | Webmail |
|
||||||
|
|
||||||
|
### Port Mapping (system container)
|
||||||
|
|
||||||
|
| Service | Internal Port | External URL |
|
||||||
|
|---------|--------------|--------------|
|
||||||
|
| botserver | `5858` | `https://system.pragmatismo.com.br` |
|
||||||
|
| botui | `5859` | `https://chat.pragmatismo.com.br` |
|
||||||
|
|
||||||
|
### Access
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# SSH to host
|
||||||
|
ssh administrator@63.141.255.9
|
||||||
|
|
||||||
|
# Execute inside system container
|
||||||
|
sudo incus exec system -- bash -c 'command'
|
||||||
|
|
||||||
|
# SSH from host to container (used by CI)
|
||||||
|
ssh -o StrictHostKeyChecking=no system "command"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Services
|
||||||
|
|
||||||
|
### botserver.service
|
||||||
|
|
||||||
|
- **Binary**: `/opt/gbo/bin/botserver`
|
||||||
|
- **Port**: `5858`
|
||||||
|
- **User**: `gbuser`
|
||||||
|
- **Logs**: `/opt/gbo/logs/out.log`, `/opt/gbo/logs/err.log`
|
||||||
|
- **Config**: `/etc/systemd/system/botserver.service`
|
||||||
|
- **Env**: `PORT=5858`
|
||||||
|
|
||||||
|
### ui.service
|
||||||
|
|
||||||
|
- **Binary**: `/opt/gbo/bin/botui`
|
||||||
|
- **Port**: `5859`
|
||||||
|
- **Config**: `/etc/systemd/system/ui.service`
|
||||||
|
- **Env**: `BOTSERVER_URL=http://localhost:5858`
|
||||||
|
- ⚠️ MUST be `http://localhost:5858` — NOT `https://system.pragmatismo.com.br`
|
||||||
|
- Rust proxy runs server-side, needs direct localhost access
|
||||||
|
- JS client uses relative URLs through `chat.pragmatismo.com.br`
|
||||||
|
|
||||||
|
### Data Directory
|
||||||
|
|
||||||
|
- **Path**: `/opt/gbo/data/`
|
||||||
|
- **Structure**: `<botname>.gbai/<botname>.gbdialog/*.bas`
|
||||||
|
- **Bots**: cristo, fema, jucees, oerlabs, poupatempo, pragmatismogb, salesianos, sentient, seplagse
|
||||||
|
- **Work dir**: `/opt/gbo/work/` (compiled .ast cache)
|
||||||
|
|
||||||
|
### Stack Services (managed by botserver bootstrap)
|
||||||
|
|
||||||
|
- **Vault**: Secrets management
|
||||||
|
- **PostgreSQL**: Database (port 5432)
|
||||||
|
- **Valkey**: Cache (port 6379, password auth)
|
||||||
|
- **MinIO**: Object storage
|
||||||
|
- **Zitadel**: Identity provider
|
||||||
|
- **LLM**: llama.cpp
|
||||||
|
|
||||||
|
## CI/CD Pipeline
|
||||||
|
|
||||||
|
### Repositories
|
||||||
|
|
||||||
|
| Repo | ALM URL | GitHub URL |
|
||||||
|
|------|---------|------------|
|
||||||
|
| gb | `https://alm.pragmatismo.com.br/GeneralBots/gb.git` | `git@github.com:GeneralBots/gb.git` |
|
||||||
|
| botserver | `https://alm.pragmatismo.com.br/GeneralBots/BotServer.git` | `git@github.com:GeneralBots/botserver.git` |
|
||||||
|
| botui | `https://alm.pragmatismo.com.br/GeneralBots/BotUI.git` | `git@github.com:GeneralBots/botui.git` |
|
||||||
|
| botlib | `https://alm.pragmatismo.com.br/GeneralBots/botlib.git` | `git@github.com:GeneralBots/botlib.git` |
|
||||||
|
|
||||||
|
### Push Order
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Push submodules first
|
||||||
|
cd botserver && git push alm main && git push origin main && cd ..
|
||||||
|
cd botui && git push alm main && git push origin main && cd ..
|
||||||
|
|
||||||
|
# 2. Update root workspace references
|
||||||
|
git add botserver botui botlib
|
||||||
|
git commit -m "Update submodules: <description>"
|
||||||
|
git push alm main && git push origin main
|
||||||
|
```
|
||||||
|
|
||||||
|
### Build Environment
|
||||||
|
|
||||||
|
- **CI runner**: `alm-ci` container (Debian Trixie, glibc 2.41)
|
||||||
|
- **Target**: `system` container (Debian 12 Bookworm, glibc 2.36)
|
||||||
|
- **⚠️ GLIBC MISMATCH**: Building on CI runner produces binaries incompatible with system container
|
||||||
|
- **Solution**: CI workflow transfers source to system container and builds there via SSH
|
||||||
|
|
||||||
|
### Workflow File
|
||||||
|
|
||||||
|
- **Location**: `botserver/.forgejo/workflows/botserver.yaml`
|
||||||
|
- **Triggers**: Push to `main` branch
|
||||||
|
- **Steps**:
|
||||||
|
1. Setup workspace on CI runner (clone repos)
|
||||||
|
2. Transfer source to system container via `tar | ssh`
|
||||||
|
3. Build inside system container (matches glibc 2.36)
|
||||||
|
4. Deploy binary inside container
|
||||||
|
5. Verify botserver is running
|
||||||
|
|
||||||
|
## Common Operations
|
||||||
|
|
||||||
|
### Check Service Status
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# From host
|
||||||
|
sudo incus exec system -- systemctl status botserver --no-pager
|
||||||
|
sudo incus exec system -- systemctl status ui --no-pager
|
||||||
|
|
||||||
|
# Check if running
|
||||||
|
sudo incus exec system -- pgrep -f botserver
|
||||||
|
sudo incus exec system -- pgrep -f botui
|
||||||
|
```
|
||||||
|
|
||||||
|
### View Logs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Systemd journal
|
||||||
|
sudo incus exec system -- journalctl -u botserver --no-pager -n 50
|
||||||
|
sudo incus exec system -- journalctl -u ui --no-pager -n 50
|
||||||
|
|
||||||
|
# Application logs
|
||||||
|
sudo incus exec system -- tail -50 /opt/gbo/logs/out.log
|
||||||
|
sudo incus exec system -- tail -50 /opt/gbo/logs/err.log
|
||||||
|
|
||||||
|
# Live tail
|
||||||
|
sudo incus exec system -- tail -f /opt/gbo/logs/out.log
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restart Services
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo incus exec system -- systemctl restart botserver
|
||||||
|
sudo incus exec system -- systemctl restart ui
|
||||||
|
```
|
||||||
|
|
||||||
|
### Manual Deploy (emergency)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Kill old process
|
||||||
|
sudo incus exec system -- killall botserver
|
||||||
|
|
||||||
|
# Copy binary (from host CI workspace or local)
|
||||||
|
sudo incus exec system -- cp /opt/gbo/ci/botserver/target/debug/botserver /opt/gbo/bin/botserver
|
||||||
|
sudo incus exec system -- chmod +x /opt/gbo/bin/botserver
|
||||||
|
sudo incus exec system -- chown gbuser:gbuser /opt/gbo/bin/botserver
|
||||||
|
|
||||||
|
# Start service
|
||||||
|
sudo incus exec system -- systemctl start botserver
|
||||||
|
```
|
||||||
|
|
||||||
|
### Transfer Bot Files to Production
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# From local to prod host
|
||||||
|
tar czf /tmp/bots.tar.gz -C /opt/gbo/data <botname>.gbai
|
||||||
|
scp /tmp/bots.tar.gz administrator@63.141.255.9:/tmp/
|
||||||
|
|
||||||
|
# From host to container
|
||||||
|
sudo incus exec system -- bash -c 'tar xzf /tmp/bots.tar.gz -C /opt/gbo/data/'
|
||||||
|
|
||||||
|
# Clear compiled cache
|
||||||
|
sudo incus exec system -- find /opt/gbo/data -name "*.ast" -delete
|
||||||
|
sudo incus exec system -- find /opt/gbo/work -name "*.ast" -delete
|
||||||
|
```
|
||||||
|
|
||||||
|
### Snapshots
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# List snapshots
|
||||||
|
sudo incus snapshot list system
|
||||||
|
|
||||||
|
# Restore snapshot
|
||||||
|
sudo incus snapshot restore system <snapshot-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### GLIBC Version Mismatch
|
||||||
|
|
||||||
|
**Symptom**: `GLIBC_2.39 not found` or `GLIBC_2.38 not found`
|
||||||
|
|
||||||
|
**Cause**: Binary compiled on CI runner (glibc 2.41) but runs in system container (glibc 2.36)
|
||||||
|
|
||||||
|
**Fix**: CI workflow must build inside the system container. Check `botserver.yaml` uses SSH to build in container.
|
||||||
|
|
||||||
|
### botserver Not Starting
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check binary
|
||||||
|
sudo incus exec system -- ldd /opt/gbo/bin/botserver | grep "not found"
|
||||||
|
|
||||||
|
# Check direct execution
|
||||||
|
sudo incus exec system -- timeout 10 /opt/gbo/bin/botserver 2>&1
|
||||||
|
|
||||||
|
# Check data directory
|
||||||
|
sudo incus exec system -- ls -la /opt/gbo/data/
|
||||||
|
```
|
||||||
|
|
||||||
|
### botui Can't Reach botserver
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check BOTSERVER_URL
|
||||||
|
sudo incus exec system -- grep BOTSERVER_URL /etc/systemd/system/ui.service
|
||||||
|
|
||||||
|
# Must be http://localhost:5858, NOT https://system.pragmatismo.com.br
|
||||||
|
# Fix:
|
||||||
|
sudo incus exec system -- sed -i 's|BOTSERVER_URL=.*|BOTSERVER_URL=http://localhost:5858|' /etc/systemd/system/ui.service
|
||||||
|
sudo incus exec system -- systemctl daemon-reload
|
||||||
|
sudo incus exec system -- systemctl restart ui
|
||||||
|
```
|
||||||
|
|
||||||
|
### Suggestions Not Showing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check bot files exist
|
||||||
|
sudo incus exec system -- ls -la /opt/gbo/data/<bot>.gbai/<bot>.gbdialog/
|
||||||
|
|
||||||
|
# Check for compilation errors
|
||||||
|
sudo incus exec system -- tail -50 /opt/gbo/logs/out.log | grep -i "error\|fail\|compile"
|
||||||
|
|
||||||
|
# Clear cache and restart
|
||||||
|
sudo incus exec system -- find /opt/gbo/work -name "*.ast" -delete
|
||||||
|
sudo incus exec system -- systemctl restart botserver
|
||||||
|
```
|
||||||
|
|
||||||
|
### IPv6 DNS Issues
|
||||||
|
|
||||||
|
**Symptom**: External API calls (Groq, Cloudflare) timeout
|
||||||
|
|
||||||
|
**Cause**: Container DNS returns AAAA records but no IPv6 connectivity
|
||||||
|
|
||||||
|
**Fix**: Container has `IPV6=no` in network config and `gai.conf` labels. If issues persist, check `RES_OPTIONS=inet4` in botserver.service.
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
- **NEVER** push secrets to git
|
||||||
|
- **NEVER** commit files to root with credentials
|
||||||
|
- **Vault** is single source of truth for secrets
|
||||||
|
- **CI/CD** is the only deployment method — never manually scp binaries
|
||||||
|
- **ALM** is production — ask before pushing
|
||||||
Loading…
Add table
Reference in a new issue