479 lines
16 KiB
Markdown
479 lines
16 KiB
Markdown
# General Bots AI Agent Guidelines
|
||
|
||
- Use apenas a língua culta ao falar.
|
||
- Never save files on root! Use `/tmp` for temp files.
|
||
- Never push to ALM without asking first — it is production!
|
||
- If in trouble with a tool, go to the official website for install instructions.
|
||
- See `botserver/src/drive/local_file_monitor.rs` to load bots from `/opt/gbo/data`.
|
||
|
||
---
|
||
|
||
## 📁 Workspace Structure
|
||
|
||
| Crate | Purpose | Port | Tech Stack |
|
||
|-------|---------|------|------------|
|
||
| **botserver** | Main API server, business logic | 8080 | Axum, Diesel, Rhai BASIC |
|
||
| **botui** | Web UI server (dev) + proxy | 3000 | Axum, HTML/HTMX/CSS |
|
||
| **botapp** | Desktop app wrapper | - | Tauri 2 |
|
||
| **botlib** | Shared library | - | Core types, errors |
|
||
| **botbook** | Documentation | - | mdBook |
|
||
| **bottest** | Integration tests | - | tokio-test |
|
||
| **botdevice** | IoT/Device support | - | Rust |
|
||
| **botplugin** | Browser extension | - | JS |
|
||
|
||
### Key Paths
|
||
- **Binary:** `target/debug/botserver`
|
||
- **Run from:** `botserver/` directory
|
||
- **Env file:** `botserver/.env`
|
||
- **UI Files:** `botui/ui/suite/`
|
||
- **Bot data:** `/opt/gbo/data` (primary)
|
||
- **Test web:** `http://localhost:3000` — Login: `http://localhost:3000/suite/auth/login.html`
|
||
|
||
---
|
||
|
||
## 🧭 LLM Navigation Guide
|
||
|
||
1. Start with **[Component Dependency Graph](../README.md#-component-dependency-graph)**
|
||
2. Review **[Module Responsibility Matrix](../README.md#-module-responsibility-matrix)**
|
||
3. Study **[Data Flow Patterns](../README.md#-data-flow-patterns)**
|
||
4. Reference **[Common Architectural Patterns](../README.md#-common-architectural-patterns)**
|
||
5. Check [Security Rules](#-security-directives---mandatory) — violations are blocking
|
||
6. Follow [Code Patterns](#-mandatory-code-patterns) — consistency is mandatory
|
||
|
||
---
|
||
|
||
## ❌ Absolute Prohibitions
|
||
|
||
### Build & Deploy
|
||
- ❌ **NEVER** search `/target` folder
|
||
- ❌ **NEVER** build in release mode or use `--release`
|
||
- ❌ **NEVER** run `cargo build` — use `cargo check` for verification
|
||
- ❌ **NEVER** run `cargo clean` — causes 30min rebuilds; use `./reset.sh` for DB issues
|
||
- ❌ **NEVER** deploy manually — ALWAYS use CI/CD pipeline (push → ALM → alm-ci builds → deploys)
|
||
- ❌ **NEVER** use `scp`, direct SSH binary copy, or manual deployment
|
||
- ❌ **NEVER** run the binary directly — use `systemctl` or `./restart.sh`
|
||
|
||
### Code Quality
|
||
- ❌ **NEVER** use `panic!()`, `todo!()`, `unimplemented!()`, `unwrap()`, `expect()`
|
||
- ❌ **NEVER** use `Command::new()` directly — use `SafeCommand`
|
||
- ❌ **NEVER** return raw error strings to HTTP clients — use `ErrorSanitizer`
|
||
- ❌ **NEVER** use `#[allow()]` or lint exceptions in `Cargo.toml` — FIX the code
|
||
- ❌ **NEVER** use `_` prefix for unused vars — DELETE or USE them
|
||
- ❌ **NEVER** leave unused imports, dead code, or commented-out code
|
||
- ❌ **NEVER** use CDN links — all assets must be local
|
||
- ❌ **NEVER** create `.md` docs without checking `botbook/` first
|
||
- ❌ **NEVER** hardcode credentials — use `generate_random_string()` or env vars
|
||
|
||
### Security
|
||
- ❌ **NEVER** include sensitive data (IPs, tokens, keys) in docs or code
|
||
- ❌ **NEVER** write internal IPs to logs — mask them (e.g., "10.x.x.x")
|
||
- ❌ **NEVER** create files with secrets in repo root
|
||
|
||
> **Secret files MUST be placed in `/tmp/` only** (ephemeral, not tracked by git).
|
||
|
||
---
|
||
|
||
## 🔐 Security Directives — MANDATORY
|
||
|
||
### 1. Error Handling — No Panics
|
||
```rust
|
||
// ❌ FORBIDDEN: unwrap(), expect(), panic!(), todo!()
|
||
// ✅ REQUIRED:
|
||
value?
|
||
value.ok_or_else(|| Error::NotFound)?
|
||
value.unwrap_or_default()
|
||
if let Some(v) = value { ... }
|
||
```
|
||
|
||
### 2. Command Execution — SafeCommand
|
||
```rust
|
||
// ❌ FORBIDDEN: Command::new("cmd").arg(user_input).output()
|
||
// ✅ REQUIRED:
|
||
use crate::security::command_guard::SafeCommand;
|
||
SafeCommand::new("allowed_command")?.arg("safe_arg")?.execute()
|
||
```
|
||
|
||
### 3. Error Responses — ErrorSanitizer
|
||
```rust
|
||
// ❌ FORBIDDEN: Json(json!({ "error": e.to_string() }))
|
||
// ✅ REQUIRED:
|
||
use crate::security::error_sanitizer::log_and_sanitize;
|
||
let sanitized = log_and_sanitize(&e, "context", None);
|
||
(StatusCode::INTERNAL_SERVER_ERROR, sanitized)
|
||
```
|
||
|
||
### 4. SQL — sql_guard
|
||
```rust
|
||
// ❌ FORBIDDEN: format!("SELECT * FROM {}", user_table)
|
||
// ✅ REQUIRED:
|
||
use crate::security::sql_guard::{sanitize_identifier, validate_table_name};
|
||
let safe_table = sanitize_identifier(&user_table);
|
||
validate_table_name(&safe_table)?;
|
||
```
|
||
|
||
### 5. Rate Limiting
|
||
- General: 100 req/s, Auth: 10 req/s, API: 50 req/s per token, WebSocket: 10 msgs/s
|
||
- Use `governor` crate with per-IP and per-User tracking
|
||
|
||
### 6. CSRF Protection
|
||
- ALL state-changing endpoints (POST/PUT/DELETE/PATCH) MUST require CSRF token
|
||
- Use `tower_csrf`, bound to user session. Exempt: Bearer Token endpoints
|
||
|
||
### 7. Security Headers (ALL responses)
|
||
`Content-Security-Policy`, `Strict-Transport-Security`, `X-Frame-Options: DENY`, `X-Content-Type-Options: nosniff`, `Referrer-Policy: strict-origin-when-cross-origin`, `Permissions-Policy: geolocation=(), microphone=(), camera=()`
|
||
|
||
### 8. Dependency Management
|
||
- App crates track `Cargo.lock`; lib crates don't
|
||
- Critical deps: exact versions (`=1.0.1`); regular: caret (`1.0`)
|
||
- Run `cargo audit` weekly; update only via PR with testing
|
||
|
||
---
|
||
|
||
## ✅ Mandatory Code Patterns
|
||
|
||
```rust
|
||
impl MyStruct { fn new() -> Self { Self { } } } // Use Self, not type name
|
||
#[derive(PartialEq, Eq)] // Always derive both
|
||
format!("Hello {name}") // Inline format args
|
||
match x { A | B => do_thing(), C => other() } // Combine identical arms
|
||
```
|
||
|
||
---
|
||
|
||
## 📏 File Size Limits
|
||
|
||
- **Max 450 lines per file** — split proactively at 350 lines
|
||
- Split by: `types.rs`, `handlers.rs`, `operations.rs`, `utils.rs`, `mod.rs`
|
||
- Re-export all public items in `mod.rs`
|
||
|
||
---
|
||
|
||
## 🔥 Error Fixing Workflow
|
||
|
||
### Preferred: Offline Batch Fix
|
||
1. Read ENTIRE error list first
|
||
2. Group errors by file
|
||
3. For each file: view → fix ALL errors → write once
|
||
4. Verify with build/diagnostics only AFTER all fixes
|
||
|
||
### ⚡ Streaming Build Rule
|
||
Don't wait for `cargo` to finish — cancel at first errors, fix, re-run.
|
||
|
||
### 🧠 Memory Issues (process "Killed")
|
||
```bash
|
||
pkill -9 cargo; pkill -9 rustc; pkill -9 botserver
|
||
CARGO_BUILD_JOBS=1 cargo check -p botserver 2>&1 | tail -200
|
||
```
|
||
|
||
---
|
||
|
||
## 🔄 Reset & Service Management
|
||
|
||
### reset.sh
|
||
- Cleans and restarts dev env (3-5 min bootstrap: Vault, PostgreSQL, Valkey, MinIO, Zitadel, LLM)
|
||
- May timeout waiting for Zitadel — check `botserver.log` for "Bootstrap process completed!"
|
||
|
||
### Verify After Reset
|
||
✅ PostgreSQL (5432), ✅ Valkey (6379), ✅ BotServer (8080), ✅ BotUI (3000), ✅ No errors in logs
|
||
|
||
### Service Commands
|
||
```bash
|
||
ps aux | grep -E "(botserver|botui)" | grep -v grep
|
||
curl http://localhost:8080/health
|
||
./restart.sh # Restart services
|
||
systemctl status|start|stop|restart botserver # systemd management
|
||
journalctl -u botserver -f # Follow logs
|
||
```
|
||
|
||
---
|
||
|
||
## 🎭 Playwright Browser Testing
|
||
|
||
### Browser Setup
|
||
If browser fails: `pkill -9 -f brave; pkill -9 -f chrome; pkill -9 -f chromium` → wait 3s → navigate again.
|
||
|
||
### Bot Testing Flow
|
||
1. Navigate to `http://localhost:3000/<botname>`
|
||
2. Snapshot → verify welcome message + suggestion buttons + Portuguese accents
|
||
3. Click suggestion → wait 3-5s → snapshot → fill data → submit
|
||
4. Verify DB records and backend logs
|
||
|
||
### Desktop UI Note
|
||
Chat window may cover other apps — click **middle button** (restore) to minimize, or navigate directly via URL.
|
||
|
||
### WhatsApp Testing
|
||
- Webhook is **global** — bot routing by typing bot name as first message
|
||
- Single WhatsApp number serves ALL bots; routing via `whatsapp-id` in `config.csv`
|
||
|
||
---
|
||
|
||
## ➕ Adding New Features
|
||
|
||
### Checklist
|
||
- [ ] Which module owns this? (Check Module Responsibility Matrix)
|
||
- [ ] Database migrations needed?
|
||
- [ ] New API endpoints?
|
||
- [ ] Security: input validation, auth, rate limiting, error sanitization?
|
||
- [ ] Screens in botui?
|
||
- [ ] No `unwrap()`/`expect()`?
|
||
|
||
### Pattern: types → schema → Diesel model → business logic → API endpoint → BASIC keyword (if applicable) → tests → docs in `botbook/`
|
||
|
||
### Commit & Deploy
|
||
```bash
|
||
cd botserver && git push alm main && git push origin main
|
||
cd .. && git add botserver && git commit -m "Update botserver: <desc>" && git push alm main && git push origin main
|
||
```
|
||
|
||
---
|
||
|
||
## 🎨 Frontend Standards
|
||
|
||
- **HTMX-first** — server returns HTML fragments, not JSON
|
||
- **Local assets only** — NO CDN links
|
||
- Use `hx-get`, `hx-post`, `hx-target`, `hx-swap`; WebSocket via htmx-ws
|
||
|
||
---
|
||
|
||
## 🚀 Performance & Quality
|
||
|
||
- `cargo clippy --workspace` must pass with **0 warnings**
|
||
- `cargo tree --duplicates` / `cargo machete` / `cargo audit` weekly
|
||
- Release profile: `opt-level = "z"`, `lto = true`, `codegen-units = 1`, `strip = true`, `panic = "abort"`
|
||
- Use `default-features = false` and opt-in to needed features
|
||
|
||
---
|
||
|
||
## 🐛 Debugging
|
||
|
||
### Critical Rule
|
||
**STOP on ANY error** — identify → fix root cause → verify → then continue. Never restart to "fix" errors.
|
||
|
||
### Log Locations
|
||
| Component | Log | Prefix |
|
||
|-----------|-----|--------|
|
||
| botserver | `botserver.log` | — |
|
||
| botui | `botui.log` | — |
|
||
| drive_monitor | botserver logs | `[drive_monitor]` |
|
||
| client errors | botserver logs | `CLIENT:` |
|
||
|
||
### Bug Fix Flow
|
||
1. Reproduce: `grep -E " E | W " botserver.log | tail -20`
|
||
2. Trace data flow backwards through call chain
|
||
3. Fix minimal change, search for similar occurrences
|
||
4. `cargo check -p botserver` → `./restart.sh` → test → check logs
|
||
5. Commit with clear root cause description
|
||
|
||
---
|
||
|
||
## 🧪 Testing
|
||
|
||
- **Unit:** per-crate `tests/` or `#[cfg(test)]` modules — `cargo test -p <crate>`
|
||
- **Integration:** `bottest/` crate — `cargo test -p bottest`
|
||
- **Coverage:** 80%+ on critical paths; ALL error paths and security guards tested
|
||
|
||
---
|
||
|
||
## 🚢 Deploy Workflow (CI/CD Only)
|
||
|
||
1. Push to ALM (triggers CI automatically)
|
||
2. CI builds on alm-ci → deploys to system container via SSH
|
||
3. Service auto-restarts on binary update
|
||
4. Verify: check service status + logs after ~10 min
|
||
|
||
### Container Architecture
|
||
|
||
| Container | Service | Port |
|
||
|-----------|---------|------|
|
||
| system | BotServer + Valkey | 8080/6379 |
|
||
| tables | PostgreSQL | 5432 |
|
||
| vault | Vault | 8200 |
|
||
| directory | Zitadel | 9000 |
|
||
| drive | MinIO | 9100 |
|
||
| cache | Valkey | 6379 |
|
||
| llm | llama.cpp | 8081 |
|
||
| vectordb | Qdrant | 6333 |
|
||
| meet | LiveKit | 7880 |
|
||
| email | Stalwart | 25/587 |
|
||
| alm | Forgejo | **4747** (NOT 3000!) |
|
||
| alm-ci | Forgejo Runner | — |
|
||
| proxy | Caddy | 80/443 |
|
||
|
||
### Container Management (Incus)
|
||
```bash
|
||
sudo incus list # List all
|
||
sudo incus start|stop|restart <container> # Lifecycle
|
||
sudo incus exec <container> -- bash # Shell access
|
||
sudo incus exec <container> -- systemctl restart <service>
|
||
sudo incus snapshot create <container> pre-change-$(date +%Y%m%d%H%M%S)
|
||
```
|
||
|
||
---
|
||
|
||
## 🔑 Core Directives Summary
|
||
|
||
- **OFFLINE FIRST** — fix all errors from list before compiling
|
||
- **BATCH BY FILE** — fix ALL errors in a file at once, write once
|
||
- **VERIFY LAST** — only compile after ALL fixes applied
|
||
- **DELETE DEAD CODE** — never keep unused code
|
||
- **GIT WORKFLOW** — always push to ALL repositories
|
||
- **0 warnings, 0 errors** — loop until clean
|
||
|
||
### 🚨 FUNDAMENTAL: Submodule Push Rule (MANDATORY)
|
||
|
||
**Every time you push the main repo, you MUST also push ALL submodules!**
|
||
|
||
```bash
|
||
# After ANY main repo push, ALWAYS run:
|
||
cd botserver && git push origin main && git push alm main
|
||
cd ../botui && git push origin main && git push alm main
|
||
cd ../botlib && git push origin main && git push alm main
|
||
# ... repeat for ALL submodules
|
||
```
|
||
|
||
**Why:** CI builds based on submodule commits. If submodule isn't pushed, CI deploys old code.
|
||
|
||
**Checklist before pushing:**
|
||
- [ ] botserver pushed?
|
||
- [ ] botui pushed?
|
||
- [ ] botlib pushed?
|
||
- [ ] All other submodules pushed?
|
||
- [ ] Main repo points to new submodule commits?
|
||
|
||
---
|
||
|
||
## 🔐 Zitadel Setup (Directory Service)
|
||
|
||
### Container Architecture
|
||
- **directory container**: Zitadel running on port **8080** internally
|
||
- **tables container**: PostgreSQL database on port 5432
|
||
- Use database **PROD-DIRECTORY** for Zitadel data
|
||
|
||
### Network Access (Container Mode)
|
||
- **Internal API**: `http://<directory-ip>:8080`
|
||
- **External port 9000** redirected via iptables NAT to directory:8080
|
||
- **Health check**: `curl -sf http://localhost:8080/debug/healthz`
|
||
|
||
### Zitadel Installation Steps
|
||
|
||
1. **Reset database** (on tables container):
|
||
```bash
|
||
psql -h localhost -U postgres -d postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'PROD-DIRECTORY' AND pid <> pg_backend_pid();"
|
||
psql -h localhost -U postgres -d postgres -c "DROP DATABASE IF EXISTS \"PROD-DIRECTORY\";"
|
||
psql -h localhost -U postgres -d postgres -c "CREATE DATABASE \"PROD-DIRECTORY\";"
|
||
```
|
||
|
||
2. **Create init config** (on directory container):
|
||
```bash
|
||
cat > /opt/gbo/conf/directory/zitadel-init-steps.yaml << "EOF"
|
||
FirstInstance:
|
||
InstanceName: "BotServer"
|
||
DefaultLanguage: "en"
|
||
PatPath: "/opt/gbo/conf/directory/admin-pat.txt"
|
||
Org:
|
||
Name: "BotServer"
|
||
Machine:
|
||
Machine:
|
||
Username: "admin-sa"
|
||
Name: "Admin Service Account"
|
||
Pat:
|
||
ExpirationDate: "2099-01-01T00:00:00Z"
|
||
Human:
|
||
UserName: "admin"
|
||
FirstName: "Admin"
|
||
LastName: "User"
|
||
Email:
|
||
Address: "admin@localhost"
|
||
Verified: true
|
||
Password: "Admin123!"
|
||
PasswordChangeRequired: false
|
||
EOF
|
||
```
|
||
|
||
3. **Start Zitadel** (on directory container):
|
||
```bash
|
||
pkill -9 zitadel || true
|
||
nohup env \
|
||
ZITADEL_DATABASE_POSTGRES_HOST=<tables-ip> \
|
||
ZITADEL_DATABASE_POSTGRES_PORT=5432 \
|
||
ZITADEL_DATABASE_POSTGRES_DATABASE=PROD-DIRECTORY \
|
||
ZITADEL_DATABASE_POSTGRES_USER_USERNAME=postgres \
|
||
ZITADEL_DATABASE_POSTGRES_USER_PASSWORD=postgres \
|
||
ZITADEL_DATABASE_POSTGRES_USER_SSL_MODE=disable \
|
||
ZITADEL_DATABASE_POSTGRES_ADMIN_USERNAME=postgres \
|
||
ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD=postgres \
|
||
ZITADEL_DATABASE_POSTGRES_ADMIN_SSL_MODE=disable \
|
||
ZITADEL_EXTERNALSECURE=false \
|
||
ZITADEL_EXTERNALDOMAIN=<directory-ip> \
|
||
ZITADEL_EXTERNALPORT=9000 \
|
||
ZITADEL_TLS_ENABLED=false \
|
||
/opt/gbo/bin/zitadel start-from-init \
|
||
--masterkey MasterkeyNeedsToHave32Characters \
|
||
--tlsMode disabled \
|
||
--externalDomain <directory-ip> \
|
||
--externalPort 9000 \
|
||
--steps /opt/gbo/conf/directory/zitadel-init-steps.yaml \
|
||
> /opt/gbo/logs/zitadel.log 2>&1 &
|
||
```
|
||
|
||
4. **Wait for bootstrap** (~90 seconds), then verify:
|
||
```bash
|
||
curl -sf http://localhost:8080/debug/healthz
|
||
cat /opt/gbo/conf/directory/admin-pat.txt
|
||
```
|
||
|
||
5. **Configure iptables** (on system container):
|
||
```bash
|
||
iptables -t nat -A PREROUTING -p tcp --dport 9000 -j DNAT --to-destination <directory-ip>:8080
|
||
iptables -t nat -A OUTPUT -p tcp -d <external-ip> --dport 9000 -j DNAT --to-destination <directory-ip>:8080
|
||
```
|
||
|
||
### Zitadel API Usage
|
||
|
||
**PAT file location**: `/opt/gbo/conf/directory/admin-pat.txt` (on directory container)
|
||
|
||
#### Get IAM Info (internal)
|
||
```bash
|
||
curl -s -H "Authorization: Bearer $PAT" http://<directory-ip>:8080/management/v1/iam
|
||
```
|
||
|
||
#### Get IAM Info (external via port 9000)
|
||
```bash
|
||
curl -s -H "Authorization: Bearer $PAT" -H "Host: <directory-ip>" http://<external-ip>:9000/management/v1/iam
|
||
```
|
||
|
||
#### Create Human User
|
||
```bash
|
||
curl -s -X POST \
|
||
-H "Authorization: Bearer $PAT" \
|
||
-H "Host: <directory-ip>" \
|
||
-H "Content-Type: application/json" \
|
||
http://<external-ip>:9000/management/v1/users/human \
|
||
-d '{
|
||
"userName": "janedoe",
|
||
"name": "Jane Doe",
|
||
"profile": {"firstName": "Jane", "lastName": "Doe"},
|
||
"email": {"email": "jane@example.com"}
|
||
}'
|
||
```
|
||
|
||
### Zitadel API Endpoints Reference
|
||
|
||
| Endpoint | Method | Description |
|
||
|----------|--------|-------------|
|
||
| `/management/v1/iam` | GET | Get IAM info |
|
||
| `/management/v1/orgs/me` | GET | Get current org |
|
||
| `/management/v1/users/human` | POST | Create human user |
|
||
| `/management/v1/users/machine` | POST | Create machine user |
|
||
| `/oauth/v2/token` | POST | Get access token |
|
||
| `/debug/healthz` | GET | Health check |
|
||
|
||
### Important Notes
|
||
|
||
- **Zitadel listens on port 8080 internally**
|
||
- **External port 9000** is forwarded via iptables NAT
|
||
- **Use Host header** with directory IP for external API calls
|
||
- **PAT file**: `/opt/gbo/conf/directory/admin-pat.txt`
|
||
- **Admin credentials**: `admin` / `Admin123!` (human user)
|
||
- **Database**: `PROD-DIRECTORY` on tables container
|
||
- **Zitadel v4.13.1** is the current version
|