GeneralBots/gb
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix DriveMonitor dispatch failure - avoid double Arc in tokio::spawn

Browse source 
- Added static save_file_states_static() helper method
- Changed tokio::spawn calls to use Arc::clone instead of Arc::new(self.clone())
- This prevents double Arc wrapping which causes 'dispatch failure' errors
- Fixes config.csv not syncing from bucket to database for salesianos/default bots
This commit is contained in:
Rodrigo Rodriguez (Pragmatismo) 2026-04-10 11:20:00 -03:00
parent c405f18efe
commit cee8aeee34
9 changed files with 1405 additions and 353 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

BIN
.goutputstream-0HCON3 Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 59 KiB

33
AGENTS.md
View file

@ -6,6 +6,39 @@
- If in trouble with a tool, go to the official website for install instructions.
- See `botserver/src/drive/local_file_monitor.rs` to load bots from `/opt/gbo/data`.
## 🚨 CRITICAL PRODUCTION RULES
### NEVER Start Services Directly in Production
When working with the production environment (63.141.255.9), **NEVER** start botserver or botui directly:
```bash
# ❌ NEVER DO THIS IN PRODUCTION:
sudo incus exec system -- /opt/gbo/bin/botserver # Wrong
sudo incus exec system -- ./botserver # Wrong
sudo incus exec system -- /opt/gbo/bin/botserver & # Wrong
# ✅ ALWAYS USE THIS:
sudo incus exec system -- systemctl start botserver
sudo incus exec system -- systemctl restart botserver
sudo incus exec system -- systemctl stop botserver
sudo incus exec system -- systemctl status botserver
```
**Why:**
- `systemctl` loads `/opt/gbo/bin/.env` (Vault credentials, paths, etc.)
- Direct execution skips environment variables → Vault connection fails → services break
- `systemctl` manages auto-restart, logging, and process lifecycle
### Development vs Production
| Environment | Start Method | Env File |
|-------------|--------------|-----------|
| **Development** | `cargo run` or `./target/debug/botserver` | `botserver/.env` |
| **Production** | `systemctl start botserver` | `/opt/gbo/bin/.env` |
**Only use `cargo run` or direct execution in development!**
---
## 📁 Workspace Structure

1336
PROD.md Normal file
View file

File diff suppressed because it is too large Load diff

2
botserver

@ -1 +1 @@
Subproject commit dc933c22e4a14dd1ee2acfe342e6b5e7f54c802a
Subproject commit 918cb623a17db5f35aaf56ff49945f1052b1e73a

1
prompts/htmlview.md Normal file
View file

@ -0,0 +1 @@
AYVRWxru3Ciwlw7E GXmWnXQYXjn1OoK4kWnY3579FJVYTGBT

259
prompts/prod.md
View file

@ -1,259 +0,0 @@
# Production Environment Guide
## Infrastructure
### Servers
| Host | IP | Purpose |
|------|-----|---------|
| `system` | `10.157.134.196` | Main botserver + botui container |
| `alm-ci` | `10.157.134.200` | CI/CD runner (Forgejo Actions) |
| `alm` | `10.157.134.34` | Forgejo git server |
| `dns` | `10.157.134.214` | DNS container |
| `drive` | `10.157.134.206` | Drive storage |
| `email` | `10.157.134.40` | Email service |
| `proxy` | `10.157.134.241` | Reverse proxy |
| `tables` | `10.157.134.174` | PostgreSQL |
| `table-editor` | `10.157.134.184` | Table editor |
| `webmail` | `10.157.134.86` | Webmail |
### Port Mapping (system container)
| Service | Internal Port | External URL |
|---------|--------------|--------------|
| botserver | `5858` | `https://system.pragmatismo.com.br` |
| botui | `5859` | `https://chat.pragmatismo.com.br` |
### Access
```bash
# SSH to host
ssh administrator@63.141.255.9
# Execute inside system container
sudo incus exec system -- bash -c 'command'
# SSH from host to container (used by CI)
ssh -o StrictHostKeyChecking=no system "command"
```
## Services
### botserver.service
- **Binary**: `/opt/gbo/bin/botserver`
- **Port**: `5858`
- **User**: `gbuser`
- **Logs**: `/opt/gbo/logs/out.log`, `/opt/gbo/logs/err.log`
- **Config**: `/etc/systemd/system/botserver.service`
- **Env**: `PORT=5858`
### ui.service
- **Binary**: `/opt/gbo/bin/botui`
- **Port**: `5859`
- **Config**: `/etc/systemd/system/ui.service`
- **Env**: `BOTSERVER_URL=http://localhost:5858`
- ⚠️ MUST be `http://localhost:5858` — NOT `https://system.pragmatismo.com.br`
- Rust proxy runs server-side, needs direct localhost access
- JS client uses relative URLs through `chat.pragmatismo.com.br`
### Data Directory
- **Path**: `/opt/gbo/data/`
- **Structure**: `<botname>.gbai/<botname>.gbdialog/*.bas`
- **Bots**: cristo, fema, jucees, oerlabs, poupatempo, pragmatismogb, salesianos, sentient, seplagse
- **Work dir**: `/opt/gbo/work/` (compiled .ast cache)
### Stack Services (managed by botserver bootstrap)
- **Vault**: Secrets management
- **PostgreSQL**: Database (port 5432)
- **Valkey**: Cache (port 6379, password auth)
- **MinIO**: Object storage
- **Zitadel**: Identity provider
- **LLM**: llama.cpp
## CI/CD Pipeline
### Repositories
| Repo | ALM URL | GitHub URL |
|------|---------|------------|
| gb | `https://alm.pragmatismo.com.br/GeneralBots/gb.git` | `git@github.com:GeneralBots/gb.git` |
| botserver | `https://alm.pragmatismo.com.br/GeneralBots/BotServer.git` | `git@github.com:GeneralBots/botserver.git` |
| botui | `https://alm.pragmatismo.com.br/GeneralBots/BotUI.git` | `git@github.com:GeneralBots/botui.git` |
| botlib | `https://alm.pragmatismo.com.br/GeneralBots/botlib.git` | `git@github.com:GeneralBots/botlib.git` |
### Push Order
```bash
# 1. Push submodules first
cd botserver && git push alm main && git push origin main && cd ..
cd botui && git push alm main && git push origin main && cd ..
# 2. Update root workspace references
git add botserver botui botlib
git commit -m "Update submodules: <description>"
git push alm main && git push origin main
```
### Build Environment
- **CI runner**: `alm-ci` container (Debian Trixie, glibc 2.41)
- **Target**: `system` container (Debian 12 Bookworm, glibc 2.36)
- **⚠️ GLIBC MISMATCH**: Building on CI runner produces binaries incompatible with system container
- **Solution**: CI workflow transfers source to system container and builds there via SSH
### Workflow File
- **Location**: `botserver/.forgejo/workflows/botserver.yaml`
- **Triggers**: Push to `main` branch
- **Steps**:
1. Setup workspace on CI runner (clone repos)
2. Transfer source to system container via `tar | ssh`
3. Build inside system container (matches glibc 2.36)
4. Deploy binary inside container
5. Verify botserver is running
## Common Operations
### Check Service Status
```bash
# From host
sudo incus exec system -- systemctl status botserver --no-pager
sudo incus exec system -- systemctl status ui --no-pager
# Check if running
sudo incus exec system -- pgrep -f botserver
sudo incus exec system -- pgrep -f botui
```
### View Logs
```bash
# Systemd journal
sudo incus exec system -- journalctl -u botserver --no-pager -n 50
sudo incus exec system -- journalctl -u ui --no-pager -n 50
# Application logs
sudo incus exec system -- tail -50 /opt/gbo/logs/out.log
sudo incus exec system -- tail -50 /opt/gbo/logs/err.log
# Live tail
sudo incus exec system -- tail -f /opt/gbo/logs/out.log
```
### Restart Services
```bash
sudo incus exec system -- systemctl restart botserver
sudo incus exec system -- systemctl restart ui
```
### Manual Deploy (emergency)
```bash
# Kill old process
sudo incus exec system -- killall botserver
# Copy binary (from host CI workspace or local)
sudo incus exec system -- cp /opt/gbo/ci/botserver/target/debug/botserver /opt/gbo/bin/botserver
sudo incus exec system -- chmod +x /opt/gbo/bin/botserver
sudo incus exec system -- chown gbuser:gbuser /opt/gbo/bin/botserver
# Start service
sudo incus exec system -- systemctl start botserver
```
### Transfer Bot Files to Production
```bash
# From local to prod host
tar czf /tmp/bots.tar.gz -C /opt/gbo/data <botname>.gbai
scp /tmp/bots.tar.gz administrator@63.141.255.9:/tmp/
# From host to container
sudo incus exec system -- bash -c 'tar xzf /tmp/bots.tar.gz -C /opt/gbo/data/'
# Clear compiled cache
sudo incus exec system -- find /opt/gbo/data -name "*.ast" -delete
sudo incus exec system -- find /opt/gbo/work -name "*.ast" -delete
```
### Snapshots
```bash
# List snapshots
sudo incus snapshot list system
# Restore snapshot
sudo incus snapshot restore system <snapshot-name>
```
## Troubleshooting
### GLIBC Version Mismatch
**Symptom**: `GLIBC_2.39 not found` or `GLIBC_2.38 not found`
**Cause**: Binary compiled on CI runner (glibc 2.41) but runs in system container (glibc 2.36)
**Fix**: CI workflow must build inside the system container. Check `botserver.yaml` uses SSH to build in container.
### botserver Not Starting
```bash
# Check binary
sudo incus exec system -- ldd /opt/gbo/bin/botserver | grep "not found"
# Check direct execution
sudo incus exec system -- timeout 10 /opt/gbo/bin/botserver 2>&1
# Check data directory
sudo incus exec system -- ls -la /opt/gbo/data/
```
### botui Can't Reach botserver
```bash
# Check BOTSERVER_URL
sudo incus exec system -- grep BOTSERVER_URL /etc/systemd/system/ui.service
# Must be http://localhost:5858, NOT https://system.pragmatismo.com.br
# Fix:
sudo incus exec system -- sed -i 's|BOTSERVER_URL=.*|BOTSERVER_URL=http://localhost:5858|' /etc/systemd/system/ui.service
sudo incus exec system -- systemctl daemon-reload
sudo incus exec system -- systemctl restart ui
```
### Suggestions Not Showing
```bash
# Check bot files exist
sudo incus exec system -- ls -la /opt/gbo/data/<bot>.gbai/<bot>.gbdialog/
# Check for compilation errors
sudo incus exec system -- tail -50 /opt/gbo/logs/out.log | grep -i "error\|fail\|compile"
# Clear cache and restart
sudo incus exec system -- find /opt/gbo/work -name "*.ast" -delete
sudo incus exec system -- systemctl restart botserver
```
### IPv6 DNS Issues
**Symptom**: External API calls (Groq, Cloudflare) timeout
**Cause**: Container DNS returns AAAA records but no IPv6 connectivity
**Fix**: Container has `IPV6=no` in network config and `gai.conf` labels. If issues persist, check `RES_OPTIONS=inet4` in botserver.service.
## Security
- **NEVER** push secrets to git
- **NEVER** commit files to root with credentials
- **Vault** is single source of truth for secrets
- **CI/CD** is the only deployment method — never manually scp binaries
- **ALM** is production — ask before pushing

49
restart-fast.sh
View file

@ -1,49 +0,0 @@
#!/bin/bash
echo "=== Fast Restart: botserver + botmodels only ==="
# Kill only the app services, keep infra running
pkill -f "botserver --noconsole" || true
pkill -f "botmodels" || true
# Clean logs
rm -f botserver.log botmodels.log
# Build only botserver (botui likely already built)
cargo build -p botserver
# Start botmodels
cd botmodels
source venv/bin/activate
uvicorn src.main:app --host 0.0.0.0 --port 8085 > ../botmodels.log 2>&1 &
echo " botmodels PID: $!"
cd ..
# Wait for botmodels
for i in $(seq 1 20); do
if curl -s http://localhost:8085/api/health > /dev/null 2>&1; then
echo " botmodels ready"
break
fi
sleep 1
done
# Start botserver (keep botui running if already up)
if ! pgrep -f "botui" > /dev/null; then
echo "Starting botui..."
cargo build -p botui
cd botui
BOTSERVER_URL="http://localhost:8080" ./target/debug/botui > ../botui.log 2>&1 &
echo " botui PID: $!"
cd ..
fi
# Start botserver
BOTMODELS_HOST="http://localhost:8085" BOTMODELS_API_KEY="starter" RUST_LOG=info ./target/debug/botserver --noconsole > botserver.log 2>&1 &
echo " botserver PID: $!"
# Quick health check
sleep 2
curl -s http://localhost:8080/health > /dev/null 2>&1 && echo "✅ botserver ready" || echo "❌ botserver failed"
echo "Done. botserver $(pgrep -f 'botserver --noconsole') botui $(pgrep -f botui) botmodels $(pgrep -f botmodels)"

74
restart.sh
View file

@ -1,55 +1,49 @@
#!/bin/bash
set -e
echo "=== Fast Restart: botserver only (keeps infra running) ==="
echo "=== Fast Restart: botserver + botmodels only ==="
# Only kill the app services, keep infra (postgres, valkey, minio, vault, zitadel) running
# Kill only the app services, keep infra running
pkill -f "botserver --noconsole" || true
pkill -f "botmodels" || true
# Clean app logs only
# Clean logs
rm -f botserver.log botmodels.log
# Build botserver (incremental, should be fast)
# Build only botserver (botui likely already built)
cargo build -p botserver
# Start botmodels if not running
if ! pgrep -f "botmodels" > /dev/null; then
echo "Starting botmodels..."
cd botmodels
source venv/bin/activate
uvicorn src.main:app --host 0.0.0.0 --port 8085 > ../botmodels.log 2>&1 &
echo " botmodels PID: $!"
cd ..
# Wait for botmodels
for i in $(seq 1 15); do
if curl -s http://localhost:8085/api/health > /dev/null 2>&1; then
echo " botmodels ready"
break
fi
sleep 1
done
else
echo " botmodels already running"
fi
# Start botmodels
cd botmodels
source venv/bin/activate
uvicorn src.main:app --host 0.0.0.0 --port 8085 > ../botmodels.log 2>&1 &
echo " botmodels PID: $!"
cd ..
# Start botserver
echo "Starting botserver..."
BOTMODELS_HOST="http://localhost:8085" BOTMODELS_API_KEY="starter" RUST_LOG=info \
./target/debug/botserver --noconsole > botserver.log 2>&1 &
echo " botserver PID: $!"
# Wait for botserver health with timeout
echo "Waiting for botserver..."
for i in $(seq 1 10); do
if curl -sf http://localhost:8080/health > /dev/null 2>&1; then
echo "✅ botserver ready"
exit 0
# Wait for botmodels
for i in $(seq 1 20); do
if curl -s http://localhost:8085/api/health > /dev/null 2>&1; then
echo " botmodels ready"
break
fi
sleep 1
done
echo "❌ botserver failed to start - check botserver.log"
tail -20 botserver.log
exit 1
# Start botserver (keep botui running if already up)
if ! pgrep -f "botui" > /dev/null; then
echo "Starting botui..."
cargo build -p botui
cd botui
BOTSERVER_URL="http://localhost:8080" ./target/debug/botui > ../botui.log 2>&1 &
echo " botui PID: $!"
cd ..
fi
# Start botserver
BOTMODELS_HOST="http://localhost:8085" BOTMODELS_API_KEY="starter" RUST_LOG=info ./target/debug/botserver --noconsole > botserver.log 2>&1 &
echo " botserver PID: $!"
# Quick health check
sleep 2
curl -s http://localhost:8080/health > /dev/null 2>&1 && echo "✅ botserver ready" || echo "❌ botserver failed"
echo "Done. botserver $(pgrep -f 'botserver --noconsole') botui $(pgrep -f botui) botmodels $(pgrep -f botmodels)"

4
stop.sh
View file

@ -1,4 +0,0 @@
pkill botui
pkill botserver -9