diff --git a/AGENTS.md b/AGENTS.md index 82c2f9c..1682a76 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,7 @@ # General Bots AI Agent Guidelines + +NEVER INCLUDE HERE CREDENTIALS OR COMPANY INFORMATION, THIS IS COMPANY AGNOSTIC. Use apenas a língua culta ao falar. Never save files to root — use `/tmp` for temp files. Never push to ALM without asking first (it is production). If a tool fails to install, check the official website for instructions. Local file support (`/opt/gbo/data`) has been removed; bots are loaded only from Drive (MinIO/S3). --- @@ -108,23 +110,12 @@ The deploy workflow is: push to ALM → CI triggers on alm-ci → builds inside ## Zitadel Setup (Directory Service) -Zitadel runs in the `directory` container on port 8080 internally. External port 9000 is forwarded to it via iptables NAT on the system container. The database is `PROD-DIRECTORY` on the `tables` container. The PAT file is at `/opt/gbo/conf/directory/admin-pat.txt` on the directory container. Admin credentials are username `admin`, password `Admin123!`. Current version is Zitadel v4.13.1. +Zitadel runs in the `directory` container on port 8080 internally. External port 9000 is forwarded to it via iptables NAT on the system container. The database is `PROD-DIRECTORY` on the `tables` container. The PAT file is at `/opt/gbo/conf/directory/admin-pat.txt` on the directory container. Admin credentials are username `admin`, password `Admin123!`. Current version is Zitadel v4.13.1. **Known bug**: Web console UI will return 404 for environment.json when accessed via reverse proxy public domain. Use http://:9000/ui/console for administrative interface instead. To reinstall: drop and recreate `PROD-DIRECTORY` on the tables container, write the init YAML to `/opt/gbo/conf/directory/zitadel-init-steps.yaml` (defining org name, admin user, and PAT expiry), then start Zitadel with env vars for the PostgreSQL host/port/database/credentials, `ZITADEL_EXTERNALSECURE=false`, `ZITADEL_EXTERNALDOMAIN=`, `ZITADEL_EXTERNALPORT=9000`, and `ZITADEL_TLS_ENABLED=false`. Pass `--masterkey MasterkeyNeedsToHave32Characters`, `--tlsMode disabled`, and `--steps `. Bootstrap takes about 90 seconds; verify with `curl -sf http://localhost:8080/debug/healthz`. -Key API endpoints: `GET /management/v1/iam` for IAM info, `GET /management/v1/orgs/me` for current org, `POST /management/v1/users/human` to create a human user, `POST /oauth/v2/token` for access tokens, `GET /debug/healthz` for health. When calling externally via port 9000, include `Host: ` header. +Key API endpoints: Use **v2 API endpoints** for all operations: `POST /v2/organizations/{org_id}/domains` to add domains, `POST /v2/users/new` to create users, `POST /oauth/v2/token` for access tokens, `GET /debug/healthz` for health. When calling externally via port 9000, include `Host: ` header. The v1 Management API is deprecated and not functional in this version. ---- - -## SEPLAGSE Bot Configuration - -SEPLAGSE bot files are at `{botname}.gbai/{botname}.gbdialog/` in MinIO. Key files: `start.bas` is the entry point with suggestion buttons; `detecta.bas` implements anomaly detection on `folha_salarios`; `init_folha.bas` initializes test data (note: the INSERT keyword has parsing issues in multi-line scripts — workaround by inserting data manually or via external SQL); `tables.bas` defines database tables auto-processed on compile. - -The detection flow is: user clicks the "Detectar Desvios" tool button → frontend sends `message_type: 6` (TOOL_EXEC) → backend executes `detecta.bas` directly → `DETECT "folha_salarios"` queries the bot-specific database → data is sent to BotModels API at `/api/anomaly/detect` → results appear in chat. - -MinIO in production runs on port 9100 (not 9000). Default credentials are `gbadmin` / `Pesquisa@1000`. The mc config is at `/root/.mc/config.json` inside the drive container. To set up a new alias: `mc alias set local2 http://127.0.0.1:9100 gbadmin "Pesquisa@1000"`. Use `--recursive` for copying or moving buckets. To rename a bucket, copy all contents to the new name then delete the old one with `mc rb`. Bot buckets should be named `{botname}.gbai` without a `gbo-` prefix. - ---- ## Frontend Standards & Performance diff --git a/PROD.md b/PROD.md index 26ecd59..c11592c 100644 --- a/PROD.md +++ b/PROD.md @@ -2,6 +2,7 @@ ## CRITICAL RULES — READ FIRST +NEVER INCLUDE HERE CREDENTIALS OR COMPANY INFORMATION, THIS IS COMPANY AGNOSTIC. Always manage services with `systemctl` inside the `system` Incus container. Never run `/opt/gbo/bin/botserver` or `/opt/gbo/bin/botui` directly — they will fail because they won't load the `.env` file containing Vault credentials and paths. The correct commands are `sudo incus exec system -- systemctl start|stop|restart|status botserver` and the same for `ui`. Systemctl handles environment loading, auto-restart, logging, and dependencies. Never push secrets (API keys, passwords, tokens) to git. Never commit `init.json` (it contains Vault unseal keys). All secrets must come from Vault — only `VAULT_*` variables are allowed in `.env`. Never deploy manually via scp or ssh; always use CI/CD. Always push all submodules (botserver, botui, botlib) before or alongside the main repo. Always ask before pushing to ALM. @@ -10,7 +11,7 @@ Never push secrets (API keys, passwords, tokens) to git. Never commit `init.json ## Infrastructure Overview -The host machine is `PROD-GBO1`, accessed via `ssh administrator@`, running Incus (an LXD fork) as hypervisor. All services run inside named Incus containers. You enter containers with `sudo incus exec -- ` and list them with `sudo incus list`. +The host machine is `PROD-GBO1`, accessed via `ssh user@`, running Incus (an LXD fork) as hypervisor. All services run inside named Incus containers. You enter containers with `sudo incus exec -- ` and list them with `sudo incus list`. The containers and their roles are: `system` runs botserver on port 5858 and botui on port 5859; `alm-ci` runs the Forgejo Actions CI runner; `alm` hosts the Forgejo git server; `tables` runs PostgreSQL on port 5432; `cache` runs Valkey/Redis on port 6379; `drive` runs MinIO object storage on port 9100; `vault` runs HashiCorp Vault on port 8200; `vector` runs Qdrant on port 6333. @@ -22,7 +23,16 @@ Externally, botserver is reachable at `https://system.example.com` and botui at Botserver runs as user `gbuser`, binary at `/opt/gbo/bin/botserver`, logs at `/opt/gbo/logs/out.log` and `/opt/gbo/logs/err.log`, systemd unit at `/etc/systemd/system/botserver.service`, env loaded from `/opt/gbo/bin/.env`. Bot BASIC scripts live under `/opt/gbo/data/.gbai/.gbdialog/*.bas`; compiled AST cache goes to `/opt/gbo/work/`. +The directory service runs Zitadel as user `root`, binary at `/opt/gbo/bin/zitadel`, logs at `/opt/gbo/logs/zitadel.log`, systemd unit at `/etc/systemd/system/directory.service`, and loads environment from the service configuration. Zitadel provides identity management and OAuth2 services for the platform. + +Internally, Zitadel listens on port 8080 within the directory container. For external access: +- Via public domain (HTTPS): `https://login.example.com` (configured through proxy container) +- Via host IP (HTTP): `http://:9000` (direct container port forwarding) +- Via container IP (HTTP): `http://:9000` (direct container access) +Access the Zitadel console at `https://login.example.com/ui/console` with admin credentials. Zitadel implements v1 Management API (deprecated) and v2 Organization/User services. Always use the v2 endpoints under `/v2/organizations` and `/v2/users` for all operations. + The botserver bootstrap also manages: Vault (secrets), PostgreSQL (database), Valkey (cache, password auth), MinIO (object storage), Zitadel (identity provider), and llama.cpp (LLM). +To obtain a PAT for Zitadel API access, check /opt/gbo/conf/directory/admin-pat.txt in the directory container. Use it with curl by setting the Authorization header: `Authorization: Bearer $(cat /opt/gbo/conf/directory/admin-pat.txt)` and include `-H \"Host: \"` for correct host resolution (replace with your directory container IP). --- @@ -112,4 +122,6 @@ HashiCorp Vault is the single source of truth for all secrets. Botserver reads ` **IPv6 DNS timeouts on external APIs (Groq, Cloudflare):** The container's DNS may return AAAA records without IPv6 connectivity. The container should have `IPV6=no` in its network config and `gai.conf` set appropriately. Check for `RES_OPTIONS=inet4` in `botserver.service` if issues persist. -**Logs show development paths instead of `/opt/gbo/data/`:** Botserver is using hardcoded dev paths. Check `.env` has `DATA_DIR=/opt/gbo/data/` and `WORK_DIR=/opt/gbo/work/`, verify the systemd unit has `EnvironmentFile=/opt/gbo/bin/.env`, and confirm Vault is reachable so service discovery works. Expected startup log lines include `info watcher:Watching data directory /opt/gbo/data` and `info botserver:BotServer started successfully on port 5858`. \ No newline at end of file +**Logs show development paths instead of `/opt/gbo/data/`:** Botserver is using hardcoded dev paths. Check `.env` has `DATA_DIR=/opt/gbo/data/` and `WORK_DIR=/opt/gbo/work/`, verify the systemd unit has `EnvironmentFile=/opt/gbo/bin/.env`, and confirm Vault is reachable so service discovery works. Expected startup log lines include `info watcher:Watching data directory /opt/gbo/data` and `info botserver:BotServer started successfully on port 5858`. + +**Migrations not running after push:** If `stat /opt/gbo/bin/botserver` shows old timestamp and `__diesel_schema_migrations` table has no new entries, CI did not rebuild. Make a trivial code change (e.g., add a comment) in botserver and push again to force rebuild. \ No newline at end of file diff --git a/botserver b/botserver index 504e6e1..74ac734 160000 --- a/botserver +++ b/botserver @@ -1 +1 @@ -Subproject commit 504e6e12ad60fc710f75385c41c2f1e6d0330ff1 +Subproject commit 74ac7342530990871aa2ef5e9d3e417e795385e3 diff --git a/setup_zitadel.sh b/setup_zitadel.sh deleted file mode 100755 index 3a9f35e..0000000 --- a/setup_zitadel.sh +++ /dev/null @@ -1,194 +0,0 @@ -#!/bin/bash - -# Script para configurar domínios, organizações e usuários no Zitadel via API -# Uso: ./setup_zitadel.sh - -PAT_TOKEN=$1 -INSTANCE_ID="367250249682552560" -BASE_URL="http://10.157.134.240:8080" - -if [ -z "$PAT_TOKEN" ]; then - echo "ERRO: É necessário fornecer um PAT token válido" - echo "Uso: $0 " - echo "" - echo "Para obter um PAT token, acesse:" - echo "https://login.pragmatismo.com.br/ui/console" - echo "Login: admin / Admin123!" - echo "Depois vá em Profile -> Personal Access Tokens -> New" - exit 1 -fi - -echo "=== Configuração do Zitadel via API ===" -echo "" - -# 1. Adicionar domínios à instância -echo "1. Adicionando domínios à instância..." - -echo " a) Adicionando domínio pragmatismo.com.br..." -curl -s -X PUT "$BASE_URL/management/v1/instances/$INSTANCE_ID/domains/pragmatismo.com.br" \ - -H "Authorization: Bearer $PAT_TOKEN" \ - -H "Host: 10.157.134.240" \ - -H "Content-Type: application/json" \ - -d '{ - "isVerified": true, - "isPrimary": true, - "generator": false - }' | python3 -m json.tool - -echo "" -echo " b) Adicionando domínio salesianos.br..." -curl -s -X PUT "$BASE_URL/management/v1/instances/$INSTANCE_ID/domains/salesianos.br" \ - -H "Authorization: Bearer $PAT_TOKEN" \ - -H "Host: 10.157.134.240" \ - -H "Content-Type: application/json" \ - -d '{ - "isVerified": true, - "isPrimary": false, - "generator": false - }' | python3 -m json.tool - -echo "" - -# 2. Criar organização Pragmatismo -echo "2. Criando organização Pragmatismo..." -PRAGMATISMO_ORG_RESPONSE=$(curl -s -X POST "$BASE_URL/management/v1/orgs" \ - -H "Authorization: Bearer $PAT_TOKEN" \ - -H "Host: 10.157.134.240" \ - -H "Content-Type: application/json" \ - -d '{ - "name": "Pragmatismo" - }') - -echo "$PRAGMATISMO_ORG_RESPONSE" | python3 -m json.tool -PRAGMATISMO_ORG_ID=$(echo "$PRAGMATISMO_ORG_RESPONSE" | python3 -c "import sys, json; print(json.load(sys.stdin)['id'])") -echo " Org ID: $PRAGMATISMO_ORG_ID" - -echo "" - -# 3. Adicionar domínio pragmatismo.com.br à organização Pragmatismo -echo "3. Adicionando domínio pragmatismo.com.br à organização Pragmatismo..." -curl -s -X POST "$BASE_URL/management/v1/orgs/$PRAGMATISMO_ORG_ID/domains" \ - -H "Authorization: Bearer $PAT_TOKEN" \ - -H "Host: 10.157.134.240" \ - -H "Content-Type: application/json" \ - -d '{ - "domain": "pragmatismo.com.br", - "isVerified": true, - "isPrimary": true - }' | python3 -m json.tool - -echo "" - -# 4. Criar organização Salesianos -echo "4. Criando organização Salesianos..." -SALESIANOS_ORG_RESPONSE=$(curl -s -X POST "$BASE_URL/management/v1/orgs" \ - -H "Authorization: Bearer $PAT_TOKEN" \ - -H "Host: 10.157.134.240" \ - -H "Content-Type: application/json" \ - -d '{ - "name": "Salesianos" - }') - -echo "$SALESIANOS_ORG_RESPONSE" | python3 -m json.tool -SALESIANOS_ORG_ID=$(echo "$SALESIANOS_ORG_RESPONSE" | python3 -c "import sys, json; print(json.load(sys.stdin)['id'])") -echo " Org ID: $SALESIANOS_ORG_ID" - -echo "" - -# 5. Adicionar domínio salesianos.br à organização Salesianos -echo "5. Adicionando domínio salesianos.br à organização Salesianos..." -curl -s -X POST "$BASE_URL/management/v1/orgs/$SALESIANOS_ORG_ID/domains" \ - -H "Authorization: Bearer $PAT_TOKEN" \ - -H "Host: 10.157.134.240" \ - -H "Content-Type: application/json" \ - -d '{ - "domain": "salesianos.br", - "isVerified": true, - "isPrimary": true - }' | python3 -m json.tool - -echo "" - -# 6. Criar usuário rodriguez@pragmatismo.com.br -echo "6. Criando usuário rodriguez@pragmatismo.com.br..." -RODRIGUEZ_USER_RESPONSE=$(curl -s -X POST "$BASE_URL/management/v1/users" \ - -H "Authorization: Bearer $PAT_TOKEN" \ - -H "Host: 10.157.134.240" \ - -H "Content-Type: application/json" \ - -d '{ - "userName": "rodriguez", - "email": "rodriguez@pragmatismo.com.br", - "profile": { - "firstName": "Rodriguez", - "lastName": "Pragmatismo" - }, - "isEmailVerified": true, - "password": "imago10$", - "passwordChangeRequired": false - }') - -echo "$RODRIGUEZ_USER_RESPONSE" | python3 -m json.tool -RODRIGUEZ_USER_ID=$(echo "$RODRIGUEZ_USER_RESPONSE" | python3 -c "import sys, json; print(json.load(sys.stdin)['userId'])") -echo " User ID: $RODRIGUEZ_USER_ID" - -echo "" - -# 7. Adicionar rodriguez à organização Pragmatismo com roles -echo "7. Adicionando rodriguez à organização Pragmatismo com roles..." -curl -s -X POST "$BASE_URL/management/v1/orgs/$PRAGMATISMO_ORG_ID/members/$RODRIGUEZ_USER_ID" \ - -H "Authorization: Bearer $PAT_TOKEN" \ - -H "Host: 10.157.134.240" \ - -H "Content-Type: application/json" \ - -d '{ - "roles": ["ORG_OWNER"] - }' | python3 -m json.tool - -echo "" - -# 8. Criar usuário marcelo.alves@salesianos.br -echo "8. Criando usuário marcelo.alves@salesianos.br..." -MARCELO_USER_RESPONSE=$(curl -s -X POST "$BASE_URL/management/v1/users" \ - -H "Authorization: Bearer $PAT_TOKEN" \ - -H "Host: 10.157.134.240" \ - -H "Content-Type: application/json" \ - -d '{ - "userName": "marcelo.alves", - "email": "marcelo.alves@salesianos.br", - "profile": { - "firstName": "Marcelo", - "lastName": "Alves" - }, - "isEmailVerified": true, - "password": "imago10$", - "passwordChangeRequired": false - }') - -echo "$MARCELO_USER_RESPONSE" | python3 -m json.tool -MARCELO_USER_ID=$(echo "$MARCELO_USER_RESPONSE" | python3 -c "import sys, json; print(json.load(sys.stdin)['userId'])") -echo " User ID: $MARCELO_USER_ID" - -echo "" - -# 9. Adicionar marcelo à organização Salesianos com roles -echo "9. Adicionando marcelo à organização Salesianos com roles..." -curl -s -X POST "$BASE_URL/management/v1/orgs/$SALESIANOS_ORG_ID/members/$MARCELO_USER_ID" \ - -H "Authorization: Bearer $PAT_TOKEN" \ - -H "Host: 10.157.134.240" \ - -H "Content-Type: application/json" \ - -d '{ - "roles": ["ORG_OWNER"] - }' | python3 -m json.tool - -echo "" -echo "=== Configuração concluída! ===" -echo "" -echo "Resumo:" -echo "- Domínios adicionados: pragmatismo.com.br, salesianos.br" -echo "- Organização Pragmatismo criada (ID: $PRAGMATISMO_ORG_ID)" -echo "- Organização Salesianos criada (ID: $SALESIANOS_ORG_ID)" -echo "- Usuário rodriguez@pragmatismo.com.br criado (ID: $RODRIGUEZ_USER_ID)" -echo "- Usuário marcelo.alves@salesianos.br criado (ID: $MARCELO_USER_ID)" -echo "" -echo "Senha para ambos os usuários: imago10$" -echo "" -echo "Para fazer login, acesse: https://login.pragmatismo.com.br/ui/console"