# General Bots AI Agent Guidelines

- Use apenas a língua culta ao falar.
- Never save files on root! Use `/tmp` for temp files.
- Never push to ALM without asking first — it is production!
- If in trouble with a tool, go to the official website for install instructions.
- See `botserver/src/drive/local_file_monitor.rs` to load bots from `/opt/gbo/data`.

---

## 📁 Workspace Structure

| Crate | Purpose | Port | Tech Stack |
|-------|---------|------|------------|
| **botserver** | Main API server, business logic | 8080 | Axum, Diesel, Rhai BASIC |
| **botui** | Web UI server (dev) + proxy | 3000 | Axum, HTML/HTMX/CSS |
| **botapp** | Desktop app wrapper | - | Tauri 2 |
| **botlib** | Shared library | - | Core types, errors |
| **botbook** | Documentation | - | mdBook |
| **bottest** | Integration tests | - | tokio-test |
| **botdevice** | IoT/Device support | - | Rust |
| **botplugin** | Browser extension | - | JS |

### Key Paths
- **Binary:** `target/debug/botserver`
- **Run from:** `botserver/` directory
- **Env file:** `botserver/.env`
- **UI Files:** `botui/ui/suite/`
- **Bot data:** `/opt/gbo/data` (primary)
- **Test web:** `http://localhost:3000` — Login: `http://localhost:3000/suite/auth/login.html`

### 📦 Data Directory Structure

```
# DEV LOCAL (quando botserver-stack existe)
├── ./botserver-stack/data/system/work/{bot}.gbai/{bot}.gbdialog/
│   ├── *.bas                     # Scripts compilados (gerado automático)
│   └── *.ast                     # Cache compilado (deletar para forçar recompilação)

# PRODUCTION (com container Incus)
├── /opt/gbo/data/               # FONTE dos bots
└── (compilação fica em memória ou /opt/gbo/work/ se existir)
```

**IMPORTANTE:** 
- **FONTE**: `/opt/gbo/data/{bot}.gbai/{bot}.gbdialog/{tool}.bas`
- **DEV LOCAL**: `./botserver-stack/data/system/work/{bot}.gbai/{bot}.gbdialog/`
- O botserver compila `.bas` → `.ast` automaticamente
- Se cache, deletar `.ast` para forçar recompilação

---

## 🧪 Debugging & Testing Tools

### 🔍 Ver Erros de Execução
```bash
tail -f botserver.log | grep -i "error\|tool"
```

### 🧪 Testar Ferramenta Específica

1. **Identificar o erro no log:**
   ```bash
   grep -A5 "Tool error" botserver.log
   ```

2. **Corrigir o arquivo `.bas` na fonte:**
   - **Dev local:** `./botserver-stack/data/system/work/{bot}.gbai/{bot}.gbdialog/{tool}.bas`
   - **Production:** `/opt/gbo/data/{bot}.gbai/{bot}.gbdialog/{tool}.bas`

3. **Forçar recompilação (se necessário):**
   ```bash
   rm ./botserver-stack/data/system/work/{bot}.gbai/{bot}.gbdialog/{tool}.ast
   ```
   - Em dev local o AST fica em `./botserver-stack/...`
   - Em production pode ficar em `/opt/gbo/work/...` se existir

4. **Testar novamente no browser:**
   ```
   http://localhost:3000/{botname}
   ```

### ⚠️ Erros Comuns em Scripts BASIC

| Erro | Causa | Solução |
|------|-------|---------|
| `=== is not a valid operator` | BASIC usa `==`, não `===` | Substituir `===` por `--` em strings |
| `Syntax error` | Erro de sintaxe BASIC | Verificar parênteses, vírgulas |
| `Tool execution failed` | Erro no script | Ver logs para stack trace |

### 📝 Exemplo: Corrigir Operador Inválido
```bas
# ERRADO (JavaScript syntax):
PRINT "=== RESULTADO ==="

# CORRETO (BASIC syntax):
PRINT "-- RESULTADO --"
```

---

## 🧭 LLM Navigation Guide

1. Start with **[Component Dependency Graph](../README.md#-component-dependency-graph)**
2. Review **[Module Responsibility Matrix](../README.md#-module-responsibility-matrix)**
3. Study **[Data Flow Patterns](../README.md#-data-flow-patterns)**
4. Reference **[Common Architectural Patterns](../README.md#-common-architectural-patterns)**
5. Check [Security Rules](#-security-directives---mandatory) — violations are blocking
6. Follow [Code Patterns](#-mandatory-code-patterns) — consistency is mandatory

---

## ❌ Absolute Prohibitions

### Build & Deploy
- ❌ **NEVER** search `/target` folder
- ❌ **NEVER** build in release mode or use `--release`
- ❌ **NEVER** run `cargo build` — use `cargo check` for verification
- ❌ **NEVER** run `cargo clean` — causes 30min rebuilds; use `./reset.sh` for DB issues
- ❌ **NEVER** deploy manually — ALWAYS use CI/CD pipeline (push → ALM → alm-ci builds → deploys)
- ❌ **NEVER** use `scp`, direct SSH binary copy, or manual deployment
- ❌ **NEVER** run the binary directly — use `systemctl` or `./restart.sh`

### Code Quality
- ❌ **NEVER** use `panic!()`, `todo!()`, `unimplemented!()`, `unwrap()`, `expect()`
- ❌ **NEVER** use `Command::new()` directly — use `SafeCommand`
- ❌ **NEVER** return raw error strings to HTTP clients — use `ErrorSanitizer`
- ❌ **NEVER** use `#[allow()]` or lint exceptions in `Cargo.toml` — FIX the code
- ❌ **NEVER** use `_` prefix for unused vars — DELETE or USE them
- ❌ **NEVER** leave unused imports, dead code, or commented-out code
- ❌ **NEVER** use CDN links — all assets must be local
- ❌ **NEVER** create `.md` docs without checking `botbook/` first
- ❌ **NEVER** hardcode credentials — use `generate_random_string()` or env vars

### Security
- ❌ **NEVER** include sensitive data (IPs, tokens, keys) in docs or code
- ❌ **NEVER** write internal IPs to logs — mask them (e.g., "10.x.x.x")
- ❌ **NEVER** create files with secrets in repo root

> **Secret files MUST be placed in `/tmp/` only** (ephemeral, not tracked by git).

---

## 🔐 Security Directives — MANDATORY

### 1. Error Handling — No Panics
```rust
// ❌ FORBIDDEN: unwrap(), expect(), panic!(), todo!()
// ✅ REQUIRED:
value?
value.ok_or_else(|| Error::NotFound)?
value.unwrap_or_default()
if let Some(v) = value { ... }
```

### 2. Command Execution — SafeCommand
```rust
// ❌ FORBIDDEN: Command::new("cmd").arg(user_input).output()
// ✅ REQUIRED:
use crate::security::command_guard::SafeCommand;
SafeCommand::new("allowed_command")?.arg("safe_arg")?.execute()
```

### 3. Error Responses — ErrorSanitizer
```rust
// ❌ FORBIDDEN: Json(json!({ "error": e.to_string() }))
// ✅ REQUIRED:
use crate::security::error_sanitizer::log_and_sanitize;
let sanitized = log_and_sanitize(&e, "context", None);
(StatusCode::INTERNAL_SERVER_ERROR, sanitized)
```

### 4. SQL — sql_guard
```rust
// ❌ FORBIDDEN: format!("SELECT * FROM {}", user_table)
// ✅ REQUIRED:
use crate::security::sql_guard::{sanitize_identifier, validate_table_name};
let safe_table = sanitize_identifier(&user_table);
validate_table_name(&safe_table)?;
```

### 5. Rate Limiting
- General: 100 req/s, Auth: 10 req/s, API: 50 req/s per token, WebSocket: 10 msgs/s
- Use `governor` crate with per-IP and per-User tracking

### 6. CSRF Protection
- ALL state-changing endpoints (POST/PUT/DELETE/PATCH) MUST require CSRF token
- Use `tower_csrf`, bound to user session. Exempt: Bearer Token endpoints

### 7. Security Headers (ALL responses)
`Content-Security-Policy`, `Strict-Transport-Security`, `X-Frame-Options: DENY`, `X-Content-Type-Options: nosniff`, `Referrer-Policy: strict-origin-when-cross-origin`, `Permissions-Policy: geolocation=(), microphone=(), camera=()`

### 8. Dependency Management
- App crates track `Cargo.lock`; lib crates don't
- Critical deps: exact versions (`=1.0.1`); regular: caret (`1.0`)
- Run `cargo audit` weekly; update only via PR with testing

---

## ✅ Mandatory Code Patterns

```rust
impl MyStruct { fn new() -> Self { Self { } } }       // Use Self, not type name
#[derive(PartialEq, Eq)]                               // Always derive both
format!("Hello {name}")                                 // Inline format args
match x { A | B => do_thing(), C => other() }           // Combine identical arms
```

---

## 📏 File Size Limits

- **Max 450 lines per file** — split proactively at 350 lines
- Split by: `types.rs`, `handlers.rs`, `operations.rs`, `utils.rs`, `mod.rs`
- Re-export all public items in `mod.rs`

---

## 🔥 Error Fixing Workflow

### Preferred: Offline Batch Fix
1. Read ENTIRE error list first
2. Group errors by file
3. For each file: view → fix ALL errors → write once
4. Verify with build/diagnostics only AFTER all fixes

### ⚡ Streaming Build Rule
Don't wait for `cargo` to finish — cancel at first errors, fix, re-run.

### 🧠 Memory Issues (process "Killed")
```bash
pkill -9 cargo; pkill -9 rustc; pkill -9 botserver
CARGO_BUILD_JOBS=1 cargo check -p botserver 2>&1 | tail -200
```

---

## 🔄 Modos de Execução

O botserver suporta **dois modos** de execução:

### Modo 1: Local Standalone (sem Docker/Incus)

O botserver sobe **tudo localmente** (PostgreSQL, Valkey, MinIO, Vault, LLM).

```bash
cd /home/rodriguez/src/gb/botserver
cargo run -- --install    # Instala dependências (PostgreSQL, Valkey, MinIO, etc.)
cargo run                 # Sobe tudo e inicia o servidor
```

**O que acontece:**
- `PackageManager` baixa e extrai binários para `botserver-stack/bin/`
- Cria `botserver-stack/data/pgdata/` com PostgreSQL
- Inicia PostgreSQL na porta 5432
- Inicia Valkey na porta 6379
- Inicia MinIO na porta 9100
- Configura Vault para secrets
- Baixa modelo LLM (llama.cpp) para detecção de anomalias
- Ao final: `http://localhost:8080`

**Verificar se está rodando:**
```bash
curl http://localhost:8080/health
curl http://localhost:5432  # PostgreSQL
curl http://localhost:6379  # Valkey
```

**Testar com Playwright:**
```bash
# Navegar para bot de teste
npx playwright open http://localhost:3000/salesianos
# Ou diretamente
npx playwright open http://localhost:3000/detecta
```

### Modo 2: Container (Incus) — Produção

Os serviços rodam em containers Incus separados.

```bash
# Subir todos os containers
sudo incus start system tables vault directory drive cache llm vector_db

# Verificar status
sudo incus list

# Acessar container system (onde roda botserver)
sudo incus exec system -- bash

# Ver logs do botserver
sudo incus exec system -- journalctl -u botserver -f
```

**Arquitetura de Containers:**

| Container | Services | Portas |
|-----------|----------|--------|
| system | BotServer, Valkey | 8080, 6379 |
| tables | PostgreSQL | 5432 |
| vault | Vault | 8200 |
| directory | Zitadel | 9000 |
| drive | MinIO | 9100 |
| cache | Valkey (backup) | 6379 |
| llm | llama.cpp | 8081 |
| vector_db | Qdrant | 6333 |

### reset.sh (Ambiente Local)
```bash
./reset.sh  # Limpa e reinicia tudo localmente
```

### Service Commands
```bash
ps aux | grep -E "(botserver|botui)" | grep -v grep
curl http://localhost:8080/health
./restart.sh                        # Restart services
```

---

## 🎭 Playwright Browser Testing

### Browser Setup
If browser fails: `pkill -9 -f brave; pkill -9 -f chrome; pkill -9 -f chromium` → wait 3s → navigate again.

### Bot Testing Flow
1. Navigate to `http://localhost:3000/<botname>`
2. Snapshot → verify welcome message + suggestion buttons + Portuguese accents
3. Click suggestion → wait 3-5s → snapshot → fill data → submit
4. Verify DB records and backend logs

### Desktop UI Note
Chat window may cover other apps — click **middle button** (restore) to minimize, or navigate directly via URL.

### WhatsApp Testing
- Webhook is **global** — bot routing by typing bot name as first message
- Single WhatsApp number serves ALL bots; routing via `whatsapp-id` in `config.csv`

---

## ➕ Adding New Features

### Checklist
- [ ] Which module owns this? (Check Module Responsibility Matrix)
- [ ] Database migrations needed?
- [ ] New API endpoints?
- [ ] Security: input validation, auth, rate limiting, error sanitization?
- [ ] Screens in botui?
- [ ] No `unwrap()`/`expect()`?

### Pattern: types → schema → Diesel model → business logic → API endpoint → BASIC keyword (if applicable) → tests → docs in `botbook/`

### Commit & Deploy
```bash
cd botserver && git push alm main && git push origin main
cd .. && git add botserver && git commit -m "Update botserver: <desc>" && git push alm main && git push origin main
```

---

## 🎨 Frontend Standards

- **HTMX-first** — server returns HTML fragments, not JSON
- **Local assets only** — NO CDN links
- Use `hx-get`, `hx-post`, `hx-target`, `hx-swap`; WebSocket via htmx-ws

---

## 🚀 Performance & Quality

- `cargo clippy --workspace` must pass with **0 warnings**
- `cargo tree --duplicates` / `cargo machete` / `cargo audit` weekly
- Release profile: `opt-level = "z"`, `lto = true`, `codegen-units = 1`, `strip = true`, `panic = "abort"`
- Use `default-features = false` and opt-in to needed features

---

## 🧪 Testing

- **Unit:** per-crate `tests/` or `#[cfg(test)]` modules — `cargo test -p <crate>`
- **Integration:** `bottest/` crate — `cargo test -p bottest`
- **Coverage:** 80%+ on critical paths; ALL error paths and security guards tested

---

## 🚢 Deploy Workflow (CI/CD Only)

1. Push to ALM (triggers CI automatically)
2. CI builds on alm-ci → deploys to system container via SSH
3. Service auto-restarts on binary update
4. Verify: check service status + logs after ~10 min

### Container Architecture

| Container | Service | Port |
|-----------|---------|------|
| system | BotServer + Valkey | 8080/6379 |
| tables | PostgreSQL | 5432 |
| vault | Vault | 8200 |
---

## 🔑 Core Directives Summary

- **OFFLINE FIRST** — fix all errors from list before compiling
- **BATCH BY FILE** — fix ALL errors in a file at once, write once
- **VERIFY LAST** — only compile after ALL fixes applied
- **DELETE DEAD CODE** — never keep unused code
- **GIT WORKFLOW** — always push to ALL repositories
- **0 warnings, 0 errors** — loop until clean

---

## 🔧 Bot Scripts Architecture

### File Types
| File | Purpose |
|------|---------|
| `start.bas` | Entry point, executed on session start |
| `{tool}.bas` | Tool implementation (e.g., `detecta.bas`) |
| `tables.bas` | **SPECIAL** - Defines database tables, auto-creates on compile |
| `init_folha.bas` | Initialization script for specific features |

### tables.bas — SPECIAL FILE
- **DO NOT call via CALL keyword** - it's processed automatically
- Parsed at compile time by `process_table_definitions()`
- Tables are created/updated in database via `sync_bot_tables()`
- Location: `/opt/gbo/data/{bot}.gbai/{bot}.gbdialog/tables.bas`

### Tool Button Execution (TOOL_EXEC)
- Frontend sends `message_type: 6` via WebSocket
- Backend handles in `stream_response()` when `message_type == MessageType::TOOL_EXEC`
- Tool executes directly, skips KB injection and LLM
- Result appears in chat (tool output), no "/tool" text shown

### CALL Keyword
- Can call in-memory procedures OR .bas scripts
- Syntax: `CALL "script_name"` or `CALL "procedure_name"`
- If not in memory, looks for `{name}.bas` in bot's gbdialog folder

### DETECT Keyword
- Analyzes database table for anomalies
- Requires table to exist (defined in tables.bas)
- Example: `result = DETECT "folha_salarios"`
- Calls BotModels API at `/api/anomaly/detect`

### start.bas Execution
- Executed on WebSocket connect (for web clients)
- Also on first user message (blocking, once per session)
- Loads suggestions via `ADD_SUGGESTION_TOOL`
- Marks session with Redis key to prevent re-run

### MessageType Enum (botlib/src/message_types.rs)
| ID | Name | Purpose |
|----|------|---------|
| 0 | EXTERNAL | External message |
| 1 | USER | User message |
| 2 | BOT_RESPONSE | Bot response |
| 3 | CONTINUE | Continue processing |
| 4 | SUGGESTION | Suggestion button |
| 5 | CONTEXT_CHANGE | Context change |
| 6 | TOOL_EXEC | Direct tool execution (skips KB/LLM) |

**Usage:** When frontend sends `message_type: 6`, backend executes tool directly without going through LLM.

### 🚨 FUNDAMENTAL: Submodule Push Rule (MANDATORY)

**Every time you push the main repo, you MUST also push ALL submodules!**

```bash
# After ANY main repo push, ALWAYS run:
cd botserver && git push origin main && git push alm main
cd ../botui && git push origin main && git push alm main
cd ../botlib && git push origin main && git push alm main
# ... repeat for ALL submodules
```

**Why:** CI builds based on submodule commits. If submodule isn't pushed, CI deploys old code.

**Checklist before pushing:**
- [ ] botserver pushed?
- [ ] botui pushed?
- [ ] botlib pushed?
- [ ] All other submodules pushed?
- [ ] Main repo points to new submodule commits?

---

## 🔐 Zitadel Setup (Directory Service)

### Container Architecture
- **directory container**: Zitadel running on port **8080** internally
- **tables container**: PostgreSQL database on port 5432
- Use database **PROD-DIRECTORY** for Zitadel data

### Network Access (Container Mode)
- **Internal API**: `http://<directory-ip>:8080`
- **External port 9000** redirected via iptables NAT to directory:8080
- **Health check**: `curl -sf http://localhost:8080/debug/healthz`

### Zitadel Installation Steps

1. **Reset database** (on tables container):
```bash
psql -h localhost -U postgres -d postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'PROD-DIRECTORY' AND pid <> pg_backend_pid();"
psql -h localhost -U postgres -d postgres -c "DROP DATABASE IF EXISTS \"PROD-DIRECTORY\";"
psql -h localhost -U postgres -d postgres -c "CREATE DATABASE \"PROD-DIRECTORY\";"
```

2. **Create init config** (on directory container):
```bash
cat > /opt/gbo/conf/directory/zitadel-init-steps.yaml << "EOF"
FirstInstance:
  InstanceName: "BotServer"
  DefaultLanguage: "en"
  PatPath: "/opt/gbo/conf/directory/admin-pat.txt"
  Org:
    Name: "BotServer"
    Machine:
      Machine:
        Username: "admin-sa"
        Name: "Admin Service Account"
      Pat:
        ExpirationDate: "2099-01-01T00:00:00Z"
    Human:
      UserName: "admin"
      FirstName: "Admin"
      LastName: "User"
      Email:
        Address: "admin@localhost"
        Verified: true
      Password: "Admin123!"
      PasswordChangeRequired: false
EOF
```

3. **Start Zitadel** (on directory container):
```bash
pkill -9 zitadel || true
nohup env \
  ZITADEL_DATABASE_POSTGRES_HOST=<tables-ip> \
  ZITADEL_DATABASE_POSTGRES_PORT=5432 \
  ZITADEL_DATABASE_POSTGRES_DATABASE=PROD-DIRECTORY \
  ZITADEL_DATABASE_POSTGRES_USER_USERNAME=postgres \
  ZITADEL_DATABASE_POSTGRES_USER_PASSWORD=postgres \
  ZITADEL_DATABASE_POSTGRES_USER_SSL_MODE=disable \
  ZITADEL_DATABASE_POSTGRES_ADMIN_USERNAME=postgres \
  ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD=postgres \
  ZITADEL_DATABASE_POSTGRES_ADMIN_SSL_MODE=disable \
  ZITADEL_EXTERNALSECURE=false \
  ZITADEL_EXTERNALDOMAIN=<directory-ip> \
  ZITADEL_EXTERNALPORT=9000 \
  ZITADEL_TLS_ENABLED=false \
  /opt/gbo/bin/zitadel start-from-init \
    --masterkey MasterkeyNeedsToHave32Characters \
    --tlsMode disabled \
    --externalDomain <directory-ip> \
    --externalPort 9000 \
    --steps /opt/gbo/conf/directory/zitadel-init-steps.yaml \
  > /opt/gbo/logs/zitadel.log 2>&1 &
```

4. **Wait for bootstrap** (~90 seconds), then verify:
```bash
curl -sf http://localhost:8080/debug/healthz
cat /opt/gbo/conf/directory/admin-pat.txt
```

5. **Configure iptables** (on system container):
```bash
iptables -t nat -A PREROUTING -p tcp --dport 9000 -j DNAT --to-destination <directory-ip>:8080
iptables -t nat -A OUTPUT -p tcp -d <external-ip> --dport 9000 -j DNAT --to-destination <directory-ip>:8080
```

### Zitadel API Usage

**PAT file location**: `/opt/gbo/conf/directory/admin-pat.txt` (on directory container)

#### Get IAM Info (internal)
```bash
curl -s -H "Authorization: Bearer $PAT" http://<directory-ip>:8080/management/v1/iam
```

#### Get IAM Info (external via port 9000)
```bash
curl -s -H "Authorization: Bearer $PAT" -H "Host: <directory-ip>" http://<external-ip>:9000/management/v1/iam
```

#### Create Human User
```bash
curl -s -X POST \
  -H "Authorization: Bearer $PAT" \
  -H "Host: <directory-ip>" \
  -H "Content-Type: application/json" \
  http://<external-ip>:9000/management/v1/users/human \
  -d '{
    "userName": "janedoe",
    "name": "Jane Doe",
    "profile": {"firstName": "Jane", "lastName": "Doe"},
    "email": {"email": "jane@example.com"}
  }'
```

### Zitadel API Endpoints Reference

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/management/v1/iam` | GET | Get IAM info |
| `/management/v1/orgs/me` | GET | Get current org |
| `/management/v1/users/human` | POST | Create human user |
| `/management/v1/users/machine` | POST | Create machine user |
| `/oauth/v2/token` | POST | Get access token |
| `/debug/healthz` | GET | Health check |

### Important Notes

- **Zitadel listens on port 8080 internally**
- **External port 9000** is forwarded via iptables NAT
- **Use Host header** with directory IP for external API calls
- **PAT file**: `/opt/gbo/conf/directory/admin-pat.txt`
- **Admin credentials**: `admin` / `Admin123!` (human user)
- **Database**: `PROD-DIRECTORY` on tables container
- **Zitadel v4.13.1** is the current version