diff --git a/ALWAYS.md b/ALWAYS.md new file mode 100644 index 0000000..cd3e14a --- /dev/null +++ b/ALWAYS.md @@ -0,0 +1,333 @@ +# ALWAYS.md - THINK KB Keyword Documentation + +## Overview + +The `THINK KB` keyword provides explicit knowledge base reasoning capabilities, allowing bots to perform structured semantic searches across active knowledge bases and return detailed results for analysis and decision-making. + +## Syntax + +```basic +results = THINK KB "query_text" +results = THINK KB query_variable +``` + +## Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `query_text` | String | The question or search query to execute against active KBs | +| `query_variable` | Variable | Variable containing the search query | + +## Return Value + +The `THINK KB` keyword returns a structured object containing: + +```basic +{ + "results": [ + { + "content": "Relevant text content from document", + "source": "path/to/source/document.pdf", + "kb_name": "knowledge_base_name", + "relevance": 0.85, + "tokens": 150 + } + ], + "summary": "Brief summary of findings", + "confidence": 0.78, + "total_results": 5, + "sources": ["doc1.pdf", "doc2.md"], + "query": "original search query", + "kb_count": 2 +} +``` + +### Result Fields + +| Field | Type | Description | +|-------|------|-------------| +| `results` | Array | Array of search results with content and metadata | +| `summary` | String | Human-readable summary of the search findings | +| `confidence` | Number | Overall confidence score (0.0 to 1.0) | +| `total_results` | Number | Total number of results found | +| `sources` | Array | List of unique source documents | +| `query` | String | The original search query | +| `kb_count` | Number | Number of knowledge bases searched | + +### Individual Result Fields + +| Field | Type | Description | +|-------|------|-------------| +| `content` | String | The relevant text content from the document | +| `source` | String | Path to the source document | +| `kb_name` | String | Name of the knowledge base containing this result | +| `relevance` | Number | Relevance score (0.0 to 1.0) | +| `tokens` | Number | Estimated token count for this content | + +## Examples + +### Basic Knowledge Base Query + +```basic +' Activate knowledge bases first +USE KB "company_policies" +USE KB "hr_handbook" + +' Perform structured search +results = THINK KB "What is the remote work policy?" + +' Access results +TALK results.summary +PRINT "Confidence: " + results.confidence +PRINT "Found " + results.total_results + " results" + +' Process individual results +FOR i = 0 TO results.results.length - 1 + result = results.results[i] + PRINT "Source: " + result.source + PRINT "Relevance: " + result.relevance + PRINT "Content: " + result.content + PRINT "---" +NEXT i +``` + +### Decision Making with Confidence Thresholds + +```basic +USE KB "technical_docs" +USE KB "troubleshooting" + +query = "How to fix database connection errors?" +results = THINK KB query + +IF results.confidence > 0.8 THEN + TALK "I found reliable information: " + results.summary + ' Show top result + IF results.total_results > 0 THEN + top_result = results.results[0] + TALK "Best match from: " + top_result.source + TALK top_result.content + END IF +ELSE IF results.confidence > 0.5 THEN + TALK "I found some relevant information, but I'm not completely certain: " + results.summary +ELSE + TALK "I couldn't find reliable information about: " + query + TALK "You might want to consult additional resources." +END IF +``` + +### Comparative Analysis + +```basic +USE KB "product_specs" +USE KB "competitor_analysis" + +' Compare multiple queries +queries = ["pricing strategy", "feature comparison", "market positioning"] + +FOR i = 0 TO queries.length - 1 + query = queries[i] + results = THINK KB query + + PRINT "=== Analysis: " + query + " ===" + PRINT "Confidence: " + results.confidence + PRINT "Sources: " + results.sources.length + PRINT "Summary: " + results.summary + PRINT "" +NEXT i +``` + +### Source-Based Filtering + +```basic +USE KB "legal_documents" + +results = THINK KB "contract termination clauses" + +' Filter by source type +pdf_results = [] +FOR i = 0 TO results.results.length - 1 + result = results.results[i] + IF result.source CONTAINS ".pdf" THEN + pdf_results.push(result) + END IF +NEXT i + +TALK "Found " + pdf_results.length + " results from PDF documents" +``` + +### Dynamic Query Building + +```basic +USE KB "customer_support" + +customer_issue = HEAR "What's your issue?" +priority = HEAR "What's the priority level?" + +' Build contextual query +query = customer_issue + " priority:" + priority + " resolution steps" +results = THINK KB query + +IF results.confidence > 0.7 THEN + TALK "Here's what I found for your " + priority + " priority issue:" + TALK results.summary + + ' Show most relevant result + IF results.total_results > 0 THEN + best_result = results.results[0] + TALK "From " + best_result.source + ":" + TALK best_result.content + END IF +ELSE + TALK "I need to escalate this issue. Let me connect you with a human agent." +END IF +``` + +## Advanced Usage Patterns + +### Multi-Stage Reasoning + +```basic +USE KB "research_papers" +USE KB "case_studies" + +' Stage 1: Find general information +general_results = THINK KB "machine learning applications" + +' Stage 2: Drill down based on initial findings +IF general_results.confidence > 0.6 THEN + ' Extract key terms from top results for refined search + specific_query = "deep learning " + general_results.results[0].content.substring(0, 50) + specific_results = THINK KB specific_query + + TALK "General overview: " + general_results.summary + TALK "Specific details: " + specific_results.summary +END IF +``` + +### Quality Assessment + +```basic +USE KB "quality_standards" + +results = THINK KB "ISO certification requirements" + +' Assess result quality +high_quality_results = [] +FOR i = 0 TO results.results.length - 1 + result = results.results[i] + IF result.relevance > 0.8 AND result.tokens > 100 THEN + high_quality_results.push(result) + END IF +NEXT i + +IF high_quality_results.length > 0 THEN + TALK "Found " + high_quality_results.length + " high-quality matches" +ELSE + TALK "Results may need verification from additional sources" +END IF +``` + +## Differences from Automatic KB Search + +| Feature | Automatic (USE KB) | Explicit (THINK KB) | +|---------|-------------------|-------------------| +| **Trigger** | User questions automatically search | Explicit keyword execution | +| **Control** | Automatic, behind-the-scenes | Full programmatic control | +| **Results** | Injected into LLM context | Structured data for processing | +| **Analysis** | LLM interprets automatically | Bot can analyze before responding | +| **Confidence** | Not exposed | Explicit confidence scoring | +| **Filtering** | Not available | Full result filtering and processing | + +## Performance Considerations + +- **Search Time**: 100-500ms depending on KB size and query complexity +- **Memory Usage**: Results cached for session duration +- **Token Limits**: Automatically respects token limits (default: 2000 tokens) +- **Concurrent Searches**: Searches all active KBs in parallel + +## Error Handling + +```basic +TRY + results = THINK KB user_query + IF results.total_results = 0 THEN + TALK "No information found for: " + user_query + END IF +CATCH error + TALK "Search failed: " + error.message + TALK "Please try a different query or check if knowledge bases are active" +END TRY +``` + +## Best Practices + +1. **Activate Relevant KBs First**: Use `USE KB` to activate appropriate knowledge bases +2. **Check Confidence Scores**: Use confidence thresholds for decision making +3. **Handle Empty Results**: Always check `total_results` before accessing results array +4. **Filter by Relevance**: Consider filtering results below 0.5 relevance +5. **Limit Result Processing**: Process only top N results to avoid performance issues +6. **Cache Results**: Store results in variables for multiple uses + +## Integration with Other Keywords + +### With LLM Keyword + +```basic +results = THINK KB "technical specifications" +IF results.confidence > 0.7 THEN + context = "Based on: " + results.summary + response = LLM "Explain this in simple terms" WITH CONTEXT context + TALK response +END IF +``` + +### With Decision Making + +```basic +policy_results = THINK KB "expense policy limits" +IF policy_results.confidence > 0.8 THEN + ' Use structured data for automated decisions + FOR i = 0 TO policy_results.results.length - 1 + result = policy_results.results[i] + IF result.content CONTAINS "maximum $500" THEN + SET expense_limit = 500 + BREAK + END IF + NEXT i +END IF +``` + +## Troubleshooting + +### No Results Returned + +```basic +results = THINK KB query +IF results.total_results = 0 THEN + PRINT "No active KBs: " + results.kb_count + PRINT "Try: USE KB 'collection_name' first" +END IF +``` + +### Low Confidence Scores + +- Refine query terms to be more specific +- Check if relevant documents are in active KBs +- Consider expanding search to additional knowledge bases +- Verify document quality and indexing + +### Performance Issues + +- Limit concurrent THINK KB calls +- Use more specific queries to reduce result sets +- Consider caching results for repeated queries +- Monitor token usage in results + +## See Also + +- [USE KB](./keyword-use-kb.md) - Activate knowledge bases for automatic search +- [CLEAR KB](./keyword-clear-kb.md) - Deactivate knowledge bases +- [KB Statistics](./keyword-kb-statistics.md) - Knowledge base metrics +- [Knowledge Base System](../03-knowledge-ai/README.md) - Technical architecture +- [Semantic Search](../03-knowledge-ai/semantic-search.md) - Search algorithms diff --git a/botbook b/botbook index e7dab66..24b77d3 160000 --- a/botbook +++ b/botbook @@ -1 +1 @@ -Subproject commit e7dab66130a0993e3a5039e342fa79581df148a9 +Subproject commit 24b77d39819b329adf3585d8f0fb65306901ed3a diff --git a/botserver b/botserver index d1cb6b7..7ef1efa 160000 --- a/botserver +++ b/botserver @@ -1 +1 @@ -Subproject commit d1cb6b758cbed905c1415f135a0327a20e51aeec +Subproject commit 7ef1efa047748bcdec99987cee15299d255abb7d diff --git a/prompts/always.md b/prompts/always.md new file mode 100644 index 0000000..fd7b3b7 --- /dev/null +++ b/prompts/always.md @@ -0,0 +1,304 @@ +# Always-On Memory KB — Implementation Plan + +## What This Is + +The Google always-on-memory-agent runs 3 sub-agents continuously: +- **IngestAgent** — extracts structured facts from any input (text, files, images, audio, video) +- **ConsolidateAgent** — runs on a timer, finds connections between memories, generates insights +- **QueryAgent** — answers questions by synthesizing all memories with citations + +The key insight: **no vector DB, no embeddings** — just an LLM that reads, thinks, and writes structured memory to SQLite. It's active, not passive. + +We integrate this as a new KB mode in the existing BASIC keyword system: + +```basic +USE KB "customer-notes" ALWAYS ON +``` + +This turns a KB into a living memory that continuously ingests, consolidates, and is always available in every session for that bot — no `USE KB` needed per session. + +--- + +## Architecture in GB Context + +``` +BASIC script: + USE KB "notes" ALWAYS ON + │ + ▼ +always_on_kb table (bot_id, kb_name, mode=always_on, consolidate_interval_mins) + │ + ├── IngestWorker (tokio task per bot) + │ watches: /opt/gbo/data/{bot}.gbai/{kb}.gbkb/inbox/ + │ on new file → LLM extract → kb_memories table + │ + ├── ConsolidateWorker (tokio interval per bot) + │ reads: unconsolidated kb_memories + │ LLM finds connections → kb_consolidations table + │ + └── QueryEnhancer (in BotOrchestrator::stream_response) + before LLM call → fetch relevant memories + consolidations + inject as system context +``` + +--- + +## Database Migration + +**File:** `botserver/migrations/6.2.6-always-on-kb/up.sql` + +```sql +-- Always-on KB configuration per bot +CREATE TABLE IF NOT EXISTS always_on_kb ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + bot_id UUID NOT NULL REFERENCES bots(id) ON DELETE CASCADE, + kb_name TEXT NOT NULL, + consolidate_interval_mins INTEGER NOT NULL DEFAULT 30, + is_active BOOLEAN NOT NULL DEFAULT true, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + UNIQUE(bot_id, kb_name) +); + +-- Individual memory entries (from IngestAgent) +CREATE TABLE IF NOT EXISTS kb_memories ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + bot_id UUID NOT NULL REFERENCES bots(id) ON DELETE CASCADE, + kb_name TEXT NOT NULL, + summary TEXT NOT NULL, + entities JSONB NOT NULL DEFAULT '[]', + topics JSONB NOT NULL DEFAULT '[]', + importance FLOAT NOT NULL DEFAULT 0.5, + source TEXT, -- file path or "api" or "chat" + is_consolidated BOOLEAN NOT NULL DEFAULT false, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE INDEX idx_kb_memories_bot_kb ON kb_memories(bot_id, kb_name); +CREATE INDEX idx_kb_memories_consolidated ON kb_memories(is_consolidated) WHERE is_consolidated = false; +CREATE INDEX idx_kb_memories_importance ON kb_memories(importance DESC); + +-- Consolidation insights (from ConsolidateAgent) +CREATE TABLE IF NOT EXISTS kb_consolidations ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + bot_id UUID NOT NULL REFERENCES bots(id) ON DELETE CASCADE, + kb_name TEXT NOT NULL, + insight TEXT NOT NULL, + memory_ids JSONB NOT NULL DEFAULT '[]', -- UUIDs of source memories + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE INDEX idx_kb_consolidations_bot_kb ON kb_consolidations(bot_id, kb_name); +``` + +--- + +## New BASIC Keyword + +**File:** `botserver/src/basic/keywords/always_on_kb.rs` + +```rust +// Registers: USE KB "name" ALWAYS ON +// Syntax: ["USE", "KB", "$expr$", "ALWAYS", "ON"] +pub fn register_always_on_kb_keyword( + engine: &mut Engine, + state: Arc, + session: Arc, +) -> Result<(), Box> { + engine.register_custom_syntax( + ["USE", "KB", "$expr$", "ALWAYS", "ON"], + true, + move |context, inputs| { + let kb_name = context.eval_expression_tree(&inputs[0])?.to_string(); + let bot_id = session.bot_id; + let pool = state.conn.clone(); + + std::thread::spawn(move || { + let mut conn = pool.get()?; + diesel::sql_query( + "INSERT INTO always_on_kb (bot_id, kb_name) VALUES ($1, $2) + ON CONFLICT (bot_id, kb_name) DO UPDATE SET is_active = true" + ) + .bind::(bot_id) + .bind::(&kb_name) + .execute(&mut conn) + }).join().ok(); + + Ok(Dynamic::UNIT) + }, + )?; + Ok(()) +} +``` + +Register in `botserver/src/basic/mod.rs` alongside `register_use_kb_keyword`. + +--- + +## AlwaysOnKbService + +**File:** `botserver/src/learn/always_on_kb.rs` + +Three async tasks per active always-on KB: + +### IngestWorker +```rust +// Watches: /opt/gbo/data/{bot_name}.gbai/{kb_name}.gbkb/inbox/ +// On new file: +// 1. Read file content (text/pdf/image via existing FileContentExtractor) +// 2. Call LLM: "Extract: summary, entities[], topics[], importance(0-1)" +// 3. INSERT INTO kb_memories +// 4. Move file to /processed/ +async fn ingest_worker(bot_id: Uuid, kb_name: String, state: Arc) +``` + +### ConsolidateWorker +```rust +// Runs every `consolidate_interval_mins` +// 1. SELECT * FROM kb_memories WHERE bot_id=$1 AND kb_name=$2 AND is_consolidated=false LIMIT 20 +// 2. Call LLM: "Find connections and generate insights from these memories: {memories}" +// 3. INSERT INTO kb_consolidations (insight, memory_ids) +// 4. UPDATE kb_memories SET is_consolidated=true WHERE id IN (...) +async fn consolidate_worker(bot_id: Uuid, kb_name: String, interval_mins: u64, state: Arc) +``` + +### QueryEnhancer (inject into BotOrchestrator) +```rust +// Called in BotOrchestrator::stream_response before LLM call +// 1. SELECT always_on_kb WHERE bot_id=$1 AND is_active=true +// 2. For each: fetch top-N memories by importance + recent consolidations +// 3. Prepend to system prompt: +// "## Memory Context\n{memories}\n## Insights\n{consolidations}" +pub async fn build_memory_context(bot_id: Uuid, pool: &DbPool) -> String +``` + +### Service Startup +```rust +// In botserver/src/main_module/server.rs, after bootstrap: +// SELECT * FROM always_on_kb WHERE is_active=true +// For each row: tokio::spawn ingest_worker + consolidate_worker +pub async fn start_always_on_kb_workers(state: Arc) +``` + +--- + +## HTTP API + +**File:** `botserver/src/learn/always_on_kb.rs` (add routes) + +``` +GET /api/kb/always-on → list all always-on KBs for bot +POST /api/kb/always-on/ingest → { kb_name, text, source } manual ingest +POST /api/kb/always-on/consolidate → { kb_name } trigger manual consolidation +GET /api/kb/always-on/:kb_name/memories → list memories (paginated) +GET /api/kb/always-on/:kb_name/insights → list consolidation insights +DELETE /api/kb/always-on/:kb_name/memory/:id → delete a memory +POST /api/kb/always-on/:kb_name/clear → delete all memories +``` + +--- + +## BotOrchestrator Integration + +**File:** `botserver/src/core/bot/mod.rs` + +In `stream_response()`, before building the LLM prompt: + +```rust +// Inject always-on memory context +let memory_ctx = always_on_kb::build_memory_context(bot_id, &state.conn).await; +if !memory_ctx.is_empty() { + system_prompt = format!("{}\n\n{}", memory_ctx, system_prompt); +} +``` + +--- + +## Inbox File Watcher + +The IngestWorker uses the existing `LocalFileMonitor` pattern from `botserver/src/drive/local_file_monitor.rs`. + +Watch path: `/opt/gbo/data/{bot_name}.gbai/{kb_name}.gbkb/inbox/` + +Supported types (reuse existing `FileContentExtractor`): `.txt`, `.md`, `.pdf`, `.json`, `.csv`, `.png`, `.jpg`, `.mp3`, `.wav`, `.mp4` + +--- + +## LLM Prompts + +### Ingest prompt +``` +Extract structured information from this content. +Return JSON: { "summary": "...", "entities": [...], "topics": [...], "importance": 0.0-1.0 } +Content: {content} +``` + +### Consolidate prompt +``` +You are a memory consolidation agent. Review these memories and find connections. +Return JSON: { "insight": "...", "connected_memory_ids": [...] } +Memories: {memories_json} +``` + +Use the existing `LlmClient` in `botserver/src/llm/mod.rs` with the bot's configured model. + +--- + +## BASIC Usage Examples + +```basic +' Enable always-on memory for this bot +USE KB "meeting-notes" ALWAYS ON + +' Manual ingest from BASIC (optional) +' (future keyword: ADD MEMORY "text" TO KB "meeting-notes") + +' Query is automatic — memories injected into every LLM call +TALK "What did we discuss last week?" +' → LLM sees memory context automatically +``` + +--- + +## File Structure + +``` +botserver/ +├── migrations/ +│ └── 6.2.6-always-on-kb/ +│ ├── up.sql ← always_on_kb, kb_memories, kb_consolidations tables +│ └── down.sql +├── src/ +│ ├── basic/keywords/ +│ │ └── always_on_kb.rs ← USE KB "x" ALWAYS ON keyword +│ ├── learn/ +│ │ └── always_on_kb.rs ← IngestWorker, ConsolidateWorker, QueryEnhancer, HTTP API +│ └── main_module/ +│ └── server.rs ← start_always_on_kb_workers() on startup +``` + +--- + +## Implementation Order + +1. Migration (`up.sql`) — tables +2. `always_on_kb.rs` keyword — register syntax +3. `learn/always_on_kb.rs` — `build_memory_context()` + HTTP API +4. `IngestWorker` + `ConsolidateWorker` tokio tasks +5. `BotOrchestrator` integration — inject memory context +6. `start_always_on_kb_workers()` in server startup +7. Register keyword in `basic/mod.rs` +8. Botbook doc + i18n keys + +--- + +## Key Differences from Google's Implementation + +| Google ADK | GB Implementation | +|-----------|-------------------| +| Python + ADK framework | Rust + existing AppState | +| SQLite | PostgreSQL (existing pool) | +| Gemini Flash-Lite only | Any configured LLM via LlmClient | +| Standalone process | Embedded tokio tasks in botserver | +| File inbox only | File inbox + HTTP API + future chat auto-ingest | +| Manual query | Auto-injected into every bot LLM call | +| No BASIC integration | `USE KB "x" ALWAYS ON` keyword | diff --git a/prompts/botserver-installers/minio b/prompts/botserver-installers/minio new file mode 100644 index 0000000..e69de29 diff --git a/prompts/botserver-installers/postgresql-17.2.0-x86_64-unknown-linux-gnu.tar.gz b/prompts/botserver-installers/postgresql-17.2.0-x86_64-unknown-linux-gnu.tar.gz new file mode 100644 index 0000000..f3d930c Binary files /dev/null and b/prompts/botserver-installers/postgresql-17.2.0-x86_64-unknown-linux-gnu.tar.gz differ diff --git a/prompts/botserver-installers/valkey-8.1.5-jammy-x86_64.tar.gz b/prompts/botserver-installers/valkey-8.1.5-jammy-x86_64.tar.gz new file mode 100644 index 0000000..43825ba Binary files /dev/null and b/prompts/botserver-installers/valkey-8.1.5-jammy-x86_64.tar.gz differ diff --git a/prompts/botserver-installers/vault_1.15.4_linux_amd64.zip b/prompts/botserver-installers/vault_1.15.4_linux_amd64.zip new file mode 100644 index 0000000..2d59a13 Binary files /dev/null and b/prompts/botserver-installers/vault_1.15.4_linux_amd64.zip differ diff --git a/prompts/prod.md b/prompts/prod.md new file mode 100644 index 0000000..503022e --- /dev/null +++ b/prompts/prod.md @@ -0,0 +1,49 @@ +# Production: Vault Container via LXD Socket + +## Current Setup + +- **botserver binary**: Already at `/opt/gbo/tenants/pragmatismo/system/bin/botserver` (inside pragmatismo-system container) +- **Target**: Install Vault in a NEW container on the **HOST** LXD (outside pragmatismo-system) +- **Connection**: botserver uses LXD socket proxy (`/tmp/lxd.sock` → host LXD) + +## Execution Plan + +### Step 1: Pull latest botserver code on pragmatismo-system + +```bash +cd /opt/gbo/tenants/pragmatismo/system +git pull alm main +``` + +### Step 2: Build botserver (if needed) + +```bash +cargo build -p botserver +cp target/debug/botserver /opt/gbo/tenants/pragmatismo/system/bin/botserver +``` + +### Step 3: Install Vault container via botserver (FROM pragmatismo-system) + +```bash +/opt/gbo/tenants/pragmatismo/system/bin/botserver install vault --container +``` + +**This runs INSIDE pragmatismo-system container but installs Vault on HOST LXD** + +### Step 4: Verify Vault is running on host + +```bash +# From pragmatismo-system, via socket proxy +lxc list + +# Or directly on host (from Proxmox) +lxc list +``` + +### Step 5: Update botserver to use external Vault + +After Vault is installed in its own container, update `/opt/gbo/tenants/pragmatismo/system/bin/.env`: + +``` +VAULT_ADDR=https://:8200 +```