gb/AGENTS.md

623 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# General Bots AI Agent Guidelines
- Use apenas a língua culta ao falar.
- Never save files on root! Use `/tmp` for temp files.
- Never push to ALM without asking first — it is production!
- If in trouble with a tool, go to the official website for install instructions.
- See `botserver/src/drive/local_file_monitor.rs` to load bots from `/opt/gbo/data`.
---
## 📁 Workspace Structure
| Crate | Purpose | Port | Tech Stack |
|-------|---------|------|------------|
| **botserver** | Main API server, business logic | 8080 | Axum, Diesel, Rhai BASIC |
| **botui** | Web UI server (dev) + proxy | 3000 | Axum, HTML/HTMX/CSS |
| **botapp** | Desktop app wrapper | - | Tauri 2 |
| **botlib** | Shared library | - | Core types, errors |
| **botbook** | Documentation | - | mdBook |
| **bottest** | Integration tests | - | tokio-test |
| **botdevice** | IoT/Device support | - | Rust |
| **botplugin** | Browser extension | - | JS |
### Key Paths
- **Binary:** `target/debug/botserver`
- **Run from:** `botserver/` directory
- **Env file:** `botserver/.env`
- **UI Files:** `botui/ui/suite/`
- **Bot data:** `/opt/gbo/data` (primary)
- **Test web:** `http://localhost:3000` — Login: `http://localhost:3000/suite/auth/login.html`
### 📦 Data Directory Structure
```
# DEV LOCAL (quando botserver-stack existe)
├── ./botserver-stack/data/system/work/{bot}.gbai/{bot}.gbdialog/
│ ├── *.bas # Scripts compilados (gerado automático)
│ └── *.ast # Cache compilado (deletar para forçar recompilação)
# PRODUCTION (com container Incus)
├── /opt/gbo/data/ # FONTE dos bots
└── (compilação fica em memória ou /opt/gbo/work/ se existir)
```
**IMPORTANTE:**
- **FONTE**: `/opt/gbo/data/{bot}.gbai/{bot}.gbdialog/{tool}.bas`
- **DEV LOCAL**: `./botserver-stack/data/system/work/{bot}.gbai/{bot}.gbdialog/`
- O botserver compila `.bas``.ast` automaticamente
- Se cache, deletar `.ast` para forçar recompilação
---
## 🧪 Debugging & Testing Tools
### 🔍 Ver Erros de Execução
```bash
tail -f botserver.log | grep -i "error\|tool"
```
### 🧪 Testar Ferramenta Específica
1. **Identificar o erro no log:**
```bash
grep -A5 "Tool error" botserver.log
```
2. **Corrigir o arquivo `.bas` na fonte:**
- **Dev local:** `./botserver-stack/data/system/work/{bot}.gbai/{bot}.gbdialog/{tool}.bas`
- **Production:** `/opt/gbo/data/{bot}.gbai/{bot}.gbdialog/{tool}.bas`
3. **Forçar recompilação (se necessário):**
```bash
rm ./botserver-stack/data/system/work/{bot}.gbai/{bot}.gbdialog/{tool}.ast
```
- Em dev local o AST fica em `./botserver-stack/...`
- Em production pode ficar em `/opt/gbo/work/...` se existir
4. **Testar novamente no browser:**
```
http://localhost:3000/{botname}
```
### ⚠️ Erros Comuns em Scripts BASIC
| Erro | Causa | Solução |
|------|-------|---------|
| `=== is not a valid operator` | BASIC usa `==`, não `===` | Substituir `===` por `--` em strings |
| `Syntax error` | Erro de sintaxe BASIC | Verificar parênteses, vírgulas |
| `Tool execution failed` | Erro no script | Ver logs para stack trace |
### 📝 Exemplo: Corrigir Operador Inválido
```bas
# ERRADO (JavaScript syntax):
PRINT "=== RESULTADO ==="
# CORRETO (BASIC syntax):
PRINT "-- RESULTADO --"
```
---
## 🧭 LLM Navigation Guide
1. Start with **[Component Dependency Graph](../README.md#-component-dependency-graph)**
2. Review **[Module Responsibility Matrix](../README.md#-module-responsibility-matrix)**
3. Study **[Data Flow Patterns](../README.md#-data-flow-patterns)**
4. Reference **[Common Architectural Patterns](../README.md#-common-architectural-patterns)**
5. Check [Security Rules](#-security-directives---mandatory) — violations are blocking
6. Follow [Code Patterns](#-mandatory-code-patterns) — consistency is mandatory
---
## ❌ Absolute Prohibitions
### Build & Deploy
- ❌ **NEVER** search `/target` folder
- ❌ **NEVER** build in release mode or use `--release`
- ❌ **NEVER** run `cargo build` — use `cargo check` for verification
- ❌ **NEVER** run `cargo clean` — causes 30min rebuilds; use `./reset.sh` for DB issues
- ❌ **NEVER** deploy manually — ALWAYS use CI/CD pipeline (push → ALM → alm-ci builds → deploys)
- ❌ **NEVER** use `scp`, direct SSH binary copy, or manual deployment
- ❌ **NEVER** run the binary directly — use `systemctl` or `./restart.sh`
### Code Quality
- ❌ **NEVER** use `panic!()`, `todo!()`, `unimplemented!()`, `unwrap()`, `expect()`
- ❌ **NEVER** use `Command::new()` directly — use `SafeCommand`
- ❌ **NEVER** return raw error strings to HTTP clients — use `ErrorSanitizer`
- ❌ **NEVER** use `#[allow()]` or lint exceptions in `Cargo.toml` — FIX the code
- ❌ **NEVER** use `_` prefix for unused vars — DELETE or USE them
- ❌ **NEVER** leave unused imports, dead code, or commented-out code
- ❌ **NEVER** use CDN links — all assets must be local
- ❌ **NEVER** create `.md` docs without checking `botbook/` first
- ❌ **NEVER** hardcode credentials — use `generate_random_string()` or env vars
### Security
- ❌ **NEVER** include sensitive data (IPs, tokens, keys) in docs or code
- ❌ **NEVER** write internal IPs to logs — mask them (e.g., "10.x.x.x")
- ❌ **NEVER** create files with secrets in repo root
> **Secret files MUST be placed in `/tmp/` only** (ephemeral, not tracked by git).
---
## 🔐 Security Directives — MANDATORY
### 1. Error Handling — No Panics
```rust
// ❌ FORBIDDEN: unwrap(), expect(), panic!(), todo!()
// ✅ REQUIRED:
value?
value.ok_or_else(|| Error::NotFound)?
value.unwrap_or_default()
if let Some(v) = value { ... }
```
### 2. Command Execution — SafeCommand
```rust
// ❌ FORBIDDEN: Command::new("cmd").arg(user_input).output()
// ✅ REQUIRED:
use crate::security::command_guard::SafeCommand;
SafeCommand::new("allowed_command")?.arg("safe_arg")?.execute()
```
### 3. Error Responses — ErrorSanitizer
```rust
// ❌ FORBIDDEN: Json(json!({ "error": e.to_string() }))
// ✅ REQUIRED:
use crate::security::error_sanitizer::log_and_sanitize;
let sanitized = log_and_sanitize(&e, "context", None);
(StatusCode::INTERNAL_SERVER_ERROR, sanitized)
```
### 4. SQL — sql_guard
```rust
// ❌ FORBIDDEN: format!("SELECT * FROM {}", user_table)
// ✅ REQUIRED:
use crate::security::sql_guard::{sanitize_identifier, validate_table_name};
let safe_table = sanitize_identifier(&user_table);
validate_table_name(&safe_table)?;
```
### 5. Rate Limiting
- General: 100 req/s, Auth: 10 req/s, API: 50 req/s per token, WebSocket: 10 msgs/s
- Use `governor` crate with per-IP and per-User tracking
### 6. CSRF Protection
- ALL state-changing endpoints (POST/PUT/DELETE/PATCH) MUST require CSRF token
- Use `tower_csrf`, bound to user session. Exempt: Bearer Token endpoints
### 7. Security Headers (ALL responses)
`Content-Security-Policy`, `Strict-Transport-Security`, `X-Frame-Options: DENY`, `X-Content-Type-Options: nosniff`, `Referrer-Policy: strict-origin-when-cross-origin`, `Permissions-Policy: geolocation=(), microphone=(), camera=()`
### 8. Dependency Management
- App crates track `Cargo.lock`; lib crates don't
- Critical deps: exact versions (`=1.0.1`); regular: caret (`1.0`)
- Run `cargo audit` weekly; update only via PR with testing
---
## ✅ Mandatory Code Patterns
```rust
impl MyStruct { fn new() -> Self { Self { } } } // Use Self, not type name
#[derive(PartialEq, Eq)] // Always derive both
format!("Hello {name}") // Inline format args
match x { A | B => do_thing(), C => other() } // Combine identical arms
```
---
## 📏 File Size Limits
- **Max 450 lines per file** — split proactively at 350 lines
- Split by: `types.rs`, `handlers.rs`, `operations.rs`, `utils.rs`, `mod.rs`
- Re-export all public items in `mod.rs`
---
## 🔥 Error Fixing Workflow
### Preferred: Offline Batch Fix
1. Read ENTIRE error list first
2. Group errors by file
3. For each file: view → fix ALL errors → write once
4. Verify with build/diagnostics only AFTER all fixes
### ⚡ Streaming Build Rule
Don't wait for `cargo` to finish — cancel at first errors, fix, re-run.
### 🧠 Memory Issues (process "Killed")
```bash
pkill -9 cargo; pkill -9 rustc; pkill -9 botserver
CARGO_BUILD_JOBS=1 cargo check -p botserver 2>&1 | tail -200
```
---
## 🔄 Modos de Execução
O botserver suporta **dois modos** de execução:
### Modo 1: Local Standalone (sem Docker/Incus)
O botserver sobe **tudo localmente** (PostgreSQL, Valkey, MinIO, Vault, LLM).
```bash
cd /home/rodriguez/src/gb/botserver
cargo run -- --install # Instala dependências (PostgreSQL, Valkey, MinIO, etc.)
cargo run # Sobe tudo e inicia o servidor
```
**O que acontece:**
- `PackageManager` baixa e extrai binários para `botserver-stack/bin/`
- Cria `botserver-stack/data/pgdata/` com PostgreSQL
- Inicia PostgreSQL na porta 5432
- Inicia Valkey na porta 6379
- Inicia MinIO na porta 9100
- Configura Vault para secrets
- Baixa modelo LLM (llama.cpp) para detecção de anomalias
- Ao final: `http://localhost:8080`
**Verificar se está rodando:**
```bash
curl http://localhost:8080/health
curl http://localhost:5432 # PostgreSQL
curl http://localhost:6379 # Valkey
```
**Testar com Playwright:**
```bash
# Navegar para bot de teste
npx playwright open http://localhost:3000/salesianos
# Ou diretamente
npx playwright open http://localhost:3000/detecta
```
### Modo 2: Container (Incus) — Produção
Os serviços rodam em containers Incus separados.
```bash
# Subir todos os containers
sudo incus start system tables vault directory drive cache llm vector_db
# Verificar status
sudo incus list
# Acessar container system (onde roda botserver)
sudo incus exec system -- bash
# Ver logs do botserver
sudo incus exec system -- journalctl -u botserver -f
```
**Arquitetura de Containers:**
| Container | Services | Portas |
|-----------|----------|--------|
| system | BotServer, Valkey | 8080, 6379 |
| tables | PostgreSQL | 5432 |
| vault | Vault | 8200 |
| directory | Zitadel | 9000 |
| drive | MinIO | 9100 |
| cache | Valkey (backup) | 6379 |
| llm | llama.cpp | 8081 |
| vector_db | Qdrant | 6333 |
### reset.sh (Ambiente Local)
```bash
./reset.sh # Limpa e reinicia tudo localmente
```
### Service Commands
```bash
ps aux | grep -E "(botserver|botui)" | grep -v grep
curl http://localhost:8080/health
./restart.sh # Restart services
```
---
## 🎭 Playwright Browser Testing
### Browser Setup
If browser fails: `pkill -9 -f brave; pkill -9 -f chrome; pkill -9 -f chromium` → wait 3s → navigate again.
### Bot Testing Flow
1. Navigate to `http://localhost:3000/<botname>`
2. Snapshot → verify welcome message + suggestion buttons + Portuguese accents
3. Click suggestion → wait 3-5s → snapshot → fill data → submit
4. Verify DB records and backend logs
### Desktop UI Note
Chat window may cover other apps — click **middle button** (restore) to minimize, or navigate directly via URL.
### WhatsApp Testing
- Webhook is **global** — bot routing by typing bot name as first message
- Single WhatsApp number serves ALL bots; routing via `whatsapp-id` in `config.csv`
---
## Adding New Features
### Checklist
- [ ] Which module owns this? (Check Module Responsibility Matrix)
- [ ] Database migrations needed?
- [ ] New API endpoints?
- [ ] Security: input validation, auth, rate limiting, error sanitization?
- [ ] Screens in botui?
- [ ] No `unwrap()`/`expect()`?
### Pattern: types → schema → Diesel model → business logic → API endpoint → BASIC keyword (if applicable) → tests → docs in `botbook/`
### Commit & Deploy
```bash
cd botserver && git push alm main && git push origin main
cd .. && git add botserver && git commit -m "Update botserver: <desc>" && git push alm main && git push origin main
```
---
## 🎨 Frontend Standards
- **HTMX-first** — server returns HTML fragments, not JSON
- **Local assets only** — NO CDN links
- Use `hx-get`, `hx-post`, `hx-target`, `hx-swap`; WebSocket via htmx-ws
---
## 🚀 Performance & Quality
- `cargo clippy --workspace` must pass with **0 warnings**
- `cargo tree --duplicates` / `cargo machete` / `cargo audit` weekly
- Release profile: `opt-level = "z"`, `lto = true`, `codegen-units = 1`, `strip = true`, `panic = "abort"`
- Use `default-features = false` and opt-in to needed features
---
## 🧪 Testing
- **Unit:** per-crate `tests/` or `#[cfg(test)]` modules — `cargo test -p <crate>`
- **Integration:** `bottest/` crate — `cargo test -p bottest`
- **Coverage:** 80%+ on critical paths; ALL error paths and security guards tested
---
## 🚢 Deploy Workflow (CI/CD Only)
1. Push to ALM (triggers CI automatically)
2. CI builds on alm-ci → deploys to system container via SSH
3. Service auto-restarts on binary update
4. Verify: check service status + logs after ~10 min
### Container Architecture
| Container | Service | Port |
|-----------|---------|------|
| system | BotServer + Valkey | 8080/6379 |
| tables | PostgreSQL | 5432 |
| vault | Vault | 8200 |
---
## 🔑 Core Directives Summary
- **OFFLINE FIRST** — fix all errors from list before compiling
- **BATCH BY FILE** — fix ALL errors in a file at once, write once
- **VERIFY LAST** — only compile after ALL fixes applied
- **DELETE DEAD CODE** — never keep unused code
- **GIT WORKFLOW** — always push to ALL repositories
- **0 warnings, 0 errors** — loop until clean
---
## 🔧 Bot Scripts Architecture
### File Types
| File | Purpose |
|------|---------|
| `start.bas` | Entry point, executed on session start |
| `{tool}.bas` | Tool implementation (e.g., `detecta.bas`) |
| `tables.bas` | **SPECIAL** - Defines database tables, auto-creates on compile |
| `init_folha.bas` | Initialization script for specific features |
### tables.bas — SPECIAL FILE
- **DO NOT call via CALL keyword** - it's processed automatically
- Parsed at compile time by `process_table_definitions()`
- Tables are created/updated in database via `sync_bot_tables()`
- Location: `/opt/gbo/data/{bot}.gbai/{bot}.gbdialog/tables.bas`
### Tool Button Execution (TOOL_EXEC)
- Frontend sends `message_type: 6` via WebSocket
- Backend handles in `stream_response()` when `message_type == MessageType::TOOL_EXEC`
- Tool executes directly, skips KB injection and LLM
- Result appears in chat (tool output), no "/tool" text shown
### CALL Keyword
- Can call in-memory procedures OR .bas scripts
- Syntax: `CALL "script_name"` or `CALL "procedure_name"`
- If not in memory, looks for `{name}.bas` in bot's gbdialog folder
### DETECT Keyword
- Analyzes database table for anomalies
- Requires table to exist (defined in tables.bas)
- Example: `result = DETECT "folha_salarios"`
- Calls BotModels API at `/api/anomaly/detect`
### start.bas Execution
- Executed on WebSocket connect (for web clients)
- Also on first user message (blocking, once per session)
- Loads suggestions via `ADD_SUGGESTION_TOOL`
- Marks session with Redis key to prevent re-run
### MessageType Enum (botlib/src/message_types.rs)
| ID | Name | Purpose |
|----|------|---------|
| 0 | EXTERNAL | External message |
| 1 | USER | User message |
| 2 | BOT_RESPONSE | Bot response |
| 3 | CONTINUE | Continue processing |
| 4 | SUGGESTION | Suggestion button |
| 5 | CONTEXT_CHANGE | Context change |
| 6 | TOOL_EXEC | Direct tool execution (skips KB/LLM) |
**Usage:** When frontend sends `message_type: 6`, backend executes tool directly without going through LLM.
### 🚨 FUNDAMENTAL: Submodule Push Rule (MANDATORY)
**Every time you push the main repo, you MUST also push ALL submodules!**
```bash
# After ANY main repo push, ALWAYS run:
cd botserver && git push origin main && git push alm main
cd ../botui && git push origin main && git push alm main
cd ../botlib && git push origin main && git push alm main
# ... repeat for ALL submodules
```
**Why:** CI builds based on submodule commits. If submodule isn't pushed, CI deploys old code.
**Checklist before pushing:**
- [ ] botserver pushed?
- [ ] botui pushed?
- [ ] botlib pushed?
- [ ] All other submodules pushed?
- [ ] Main repo points to new submodule commits?
---
## 🔐 Zitadel Setup (Directory Service)
### Container Architecture
- **directory container**: Zitadel running on port **8080** internally
- **tables container**: PostgreSQL database on port 5432
- Use database **PROD-DIRECTORY** for Zitadel data
### Network Access (Container Mode)
- **Internal API**: `http://<directory-ip>:8080`
- **External port 9000** redirected via iptables NAT to directory:8080
- **Health check**: `curl -sf http://localhost:8080/debug/healthz`
### Zitadel Installation Steps
1. **Reset database** (on tables container):
```bash
psql -h localhost -U postgres -d postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'PROD-DIRECTORY' AND pid <> pg_backend_pid();"
psql -h localhost -U postgres -d postgres -c "DROP DATABASE IF EXISTS \"PROD-DIRECTORY\";"
psql -h localhost -U postgres -d postgres -c "CREATE DATABASE \"PROD-DIRECTORY\";"
```
2. **Create init config** (on directory container):
```bash
cat > /opt/gbo/conf/directory/zitadel-init-steps.yaml << "EOF"
FirstInstance:
InstanceName: "BotServer"
DefaultLanguage: "en"
PatPath: "/opt/gbo/conf/directory/admin-pat.txt"
Org:
Name: "BotServer"
Machine:
Machine:
Username: "admin-sa"
Name: "Admin Service Account"
Pat:
ExpirationDate: "2099-01-01T00:00:00Z"
Human:
UserName: "admin"
FirstName: "Admin"
LastName: "User"
Email:
Address: "admin@localhost"
Verified: true
Password: "Admin123!"
PasswordChangeRequired: false
EOF
```
3. **Start Zitadel** (on directory container):
```bash
pkill -9 zitadel || true
nohup env \
ZITADEL_DATABASE_POSTGRES_HOST=<tables-ip> \
ZITADEL_DATABASE_POSTGRES_PORT=5432 \
ZITADEL_DATABASE_POSTGRES_DATABASE=PROD-DIRECTORY \
ZITADEL_DATABASE_POSTGRES_USER_USERNAME=postgres \
ZITADEL_DATABASE_POSTGRES_USER_PASSWORD=postgres \
ZITADEL_DATABASE_POSTGRES_USER_SSL_MODE=disable \
ZITADEL_DATABASE_POSTGRES_ADMIN_USERNAME=postgres \
ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD=postgres \
ZITADEL_DATABASE_POSTGRES_ADMIN_SSL_MODE=disable \
ZITADEL_EXTERNALSECURE=false \
ZITADEL_EXTERNALDOMAIN=<directory-ip> \
ZITADEL_EXTERNALPORT=9000 \
ZITADEL_TLS_ENABLED=false \
/opt/gbo/bin/zitadel start-from-init \
--masterkey MasterkeyNeedsToHave32Characters \
--tlsMode disabled \
--externalDomain <directory-ip> \
--externalPort 9000 \
--steps /opt/gbo/conf/directory/zitadel-init-steps.yaml \
> /opt/gbo/logs/zitadel.log 2>&1 &
```
4. **Wait for bootstrap** (~90 seconds), then verify:
```bash
curl -sf http://localhost:8080/debug/healthz
cat /opt/gbo/conf/directory/admin-pat.txt
```
5. **Configure iptables** (on system container):
```bash
iptables -t nat -A PREROUTING -p tcp --dport 9000 -j DNAT --to-destination <directory-ip>:8080
iptables -t nat -A OUTPUT -p tcp -d <external-ip> --dport 9000 -j DNAT --to-destination <directory-ip>:8080
```
### Zitadel API Usage
**PAT file location**: `/opt/gbo/conf/directory/admin-pat.txt` (on directory container)
#### Get IAM Info (internal)
```bash
curl -s -H "Authorization: Bearer $PAT" http://<directory-ip>:8080/management/v1/iam
```
#### Get IAM Info (external via port 9000)
```bash
curl -s -H "Authorization: Bearer $PAT" -H "Host: <directory-ip>" http://<external-ip>:9000/management/v1/iam
```
#### Create Human User
```bash
curl -s -X POST \
-H "Authorization: Bearer $PAT" \
-H "Host: <directory-ip>" \
-H "Content-Type: application/json" \
http://<external-ip>:9000/management/v1/users/human \
-d '{
"userName": "janedoe",
"name": "Jane Doe",
"profile": {"firstName": "Jane", "lastName": "Doe"},
"email": {"email": "jane@example.com"}
}'
```
### Zitadel API Endpoints Reference
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/management/v1/iam` | GET | Get IAM info |
| `/management/v1/orgs/me` | GET | Get current org |
| `/management/v1/users/human` | POST | Create human user |
| `/management/v1/users/machine` | POST | Create machine user |
| `/oauth/v2/token` | POST | Get access token |
| `/debug/healthz` | GET | Health check |
### Important Notes
- **Zitadel listens on port 8080 internally**
- **External port 9000** is forwarded via iptables NAT
- **Use Host header** with directory IP for external API calls
- **PAT file**: `/opt/gbo/conf/directory/admin-pat.txt`
- **Admin credentials**: `admin` / `Admin123!` (human user)
- **Database**: `PROD-DIRECTORY` on tables container
- **Zitadel v4.13.1** is the current version