Commit graph

225 commits

Author SHA1 Message Date
4d7297243e Fix clippy warnings: reduce 17 warnings to 0
All checks were successful
BotServer CI/CD / build (push) Successful in 6m58s
- Fix double_ended_iterator_last: use next_back() instead of last()
- Fix manual_clamp: use .clamp() instead of min().max()
- Fix too_many_arguments: create KbInjectionContext struct
- Fix needless_borrow: remove unnecessary & reference
- Fix let_and_return: return value directly
- Fix await_holding_lock: drop guard before await
- Fix collapsible_else_if: collapse nested if-else

All changes verified with cargo clippy (0 warnings, 0 errors)
Note: Local botserver crashes with existing panic during LocalFileMonitor initialization
This panic exists in original code too, not caused by these changes
2026-04-03 22:34:43 -03:00
eb98574c8a fix(runtime): use TransferResult instead of Result in transfer_to_human
All checks were successful
BotServer CI/CD / build (push) Successful in 5m19s
2026-04-03 20:49:17 -03:00
3f94d23e1f fix(runtime): replace Handle::current().block_on() with std:🧵:spawn in transfer_to_human
Some checks failed
BotServer CI/CD / build (push) Failing after 1m18s
- Handle::current().block_on() panics when called from within a runtime
- replaced all 5 occurrences with std:🧵:spawn + mpsc::channel
- matches the pattern already used across other keyword files
2026-04-03 20:43:48 -03:00
8019107ebf fix: remove last remaining block_in_place in TALK TO keyword
Some checks failed
BotServer CI/CD / build (push) Failing after 19m14s
This was the only block_in_place left causing the production panic during
bot compilation. Replaced with std:🧵:spawn + mpsc channel pattern.
2026-04-03 18:35:27 -03:00
6f183c63d2 feat: dual-mode service configs - Vault first, fallback to DB/localhost
Some checks failed
BotServer CI/CD / build (push) Has been cancelled
All services now try Vault first (remote/distributed mode), then fall back
to database config, then localhost defaults (local/dev mode).

Services fixed:
- Qdrant/VectorDB: kb_indexer.rs, kb_statistics.rs, bootstrap_utils.rs, kb_context.rs
- LLM/Embedding: email/vectordb.rs (was hardcoded localhost:8082)
- All services: security/integration.rs (postgres, cache, drive, directory, qdrant, llm)

Pattern: SecretsManager::get_X_config_sync() → DB config → localhost default
2026-04-03 15:01:37 -03:00
61642343a8 fix: replace all block_in_place with std:🧵:spawn to fix nested runtime panic
Some checks are pending
BotServer CI/CD / build (push) Waiting to run
Root cause: block_in_place + new_current_thread().block_on() panics when
called from within tokio runtime (including spawn_blocking). Tokio doesn't
allow nested block_on() calls.

Fix: Replace ALL block_in_place patterns with std:🧵:spawn + mpsc channel.
This creates a completely separate OS thread with its own runtime, avoiding
any nesting issues. Works from any context: async, spawn_blocking, or sync.

Files: 14 files across secrets, utils, state, calendar, analytics, email,
and all keyword handlers (universal_messaging, search, book, create_draft,
create_site, hearing/syntax, use_tool, find, admin_email, goals)
2026-04-03 12:54:36 -03:00
f6a864aa67 fix: replace nested runtime block_on with new_current_thread to prevent panic
All checks were successful
BotServer CI/CD / build (push) Successful in 5m32s
Root cause: Handle::current().block_on() panics inside multi_thread runtime
with 'Cannot start a runtime from within a runtime' error.

Fix: All sync-to-async bridges now use tokio::runtime::Builder::new_current_thread()
instead of Handle::current().block_on(). Also changed SECRETS_MANAGER from
tokio::sync::RwLock to std::sync::RwLock to eliminate unnecessary async overhead.

Files: 14 files across keywords, secrets, utils, state, calendar, analytics, email
Impact: Fixes production crash during bot loading phase
2026-04-03 09:17:23 -03:00
e992ed3b39 Enforce Vault-only secrets: remove env var fallbacks, all secrets from Vault
Some checks are pending
BotServer CI/CD / build (push) Waiting to run
- Remove all std::env::var calls except VAULT_* and PORT
- get_from_env returns hardcoded defaults only (no env var reading)
- Auth config, rate limits, email, analytics, calendar all use Vault
- WORK_PATH replaced with get_work_path() helper reading from Vault
- .env on production cleaned to only VAULT_ADDR, VAULT_TOKEN, VAULT_CACERT, PORT
- All service IPs/credentials stored in Vault secret/gbo/*
2026-04-03 07:11:40 -03:00
fb2e5242da fix: Vault seeding, service health checks, and restart idempotency
All checks were successful
BotServer CI/CD / build (push) Successful in 55m52s
- Replace hardcoded passwords with generate_random_string() for all Vault-seeded services
- Add valkey-cli, nc to SafeCommand allowlist; fix PATH in all 4 execution methods
- Fix empty Vault KV values ('none' placeholder) preventing 'Failed to parse K=V' errors
- Fix special chars in generated passwords triggering shell injection false positives
- Add ALM app.ini creation with absolute paths for Forgejo CLI
- Increase Qdrant timeout 15s→45s, ALM wait 5s→20s
- Persist file_states and kb_states to disk for .bas/KB idempotency across restarts
- Add duplicate check to use_website registration (debug log for existing)
- Remove dead code (SERVER_START_EPOCH, server_epoch)
- Add generate_random_string() to shared mod.rs, remove duplicates
2026-04-01 12:22:57 -03:00
adb26330d2 fix: Simple 50ms timeout for Redis connection 2026-03-21 10:48:47 -03:00
9d6c2686f1 fix: Remove connection caching (no Clone) 2026-03-21 10:37:49 -03:00
b3ce293487 fix: Clean up duplicate Redis code and fix WebSocket log level 2026-03-21 10:30:19 -03:00
cfe6453d1e perf: Add shared Redis connection pool with 50ms timeout 2026-03-21 10:14:10 -03:00
43fd40aed9 fix: Add timeout to Redis get_connection to prevent blocking
- Added get_redis_connection() helper with 2s timeout
- All cache operations now fail fast if Valkey is not ready
- Prevents start.bas from blocking for minutes waiting for cache
- Changes: add_suggestion.rs
2026-03-21 09:34:41 -03:00
e5f3380469 perf: Fix USE TOOL thread contention by removing runtime creation
- Replace thread spawn + tokio runtime creation with block_in_place
- Eliminates 10+ runtime creations per start.bas execution
- Reduces USE TOOL execution from ~2min to milliseconds
- Fixes suggestions not appearing due to start.bas timeout
2026-03-20 22:54:19 -03:00
d19984fa07 feat: Improve KB keywords and package manager installer 2026-03-20 17:38:47 -03:00
3bb115266b feat: Add GUID prefix to Qdrant collection names for KB security isolation 2026-03-19 19:51:28 -03:00
d6ebd0cf6e fix: send suggestions separately from TALK, clear Redis keys for refresh
- Remove suggestions fetching from TALK function
- WebSocket handler now fetches and sends suggestions after start.bas executes
- Clear suggestions and start_bas_executed keys to allow re-run on refresh
- Decouple TALK from suggestions handling
2026-03-19 09:53:39 -03:00
2fcfb05fd6 fix: USE_WEBSITE non-blocking - timeout 3s, never blocks start.bas 2026-03-18 19:41:23 -03:00
1a9208b88e Fix: Use bot_id instead of user_id in TALK suggestions Redis key 2026-03-18 11:05:56 -03:00
ec4fcc094a Fix: Use bot_id instead of user_id in suggestion Redis keys
- Changed all suggestion key formats from suggestions:user_id:session_id to suggestions:bot_id:session_id
- Fixes bug where suggestions were stored under wrong key, preventing frontend from retrieving them
- Affects: CLEAR SUGGESTIONS, ADD SUGGESTION, ADD SUGGESTION TEXT, ADD_SUGGESTION_TOOL
- Impact: Suggestions now correctly associated with bot, not user
2026-03-18 10:39:27 -03:00
af7441ebcb fix: generate mcp.json for tools without PARAM declarations
Tools using only USE KB or other keywords without PARAM were not
getting .mcp.json generated, causing USE TOOL to silently skip them.
2026-03-17 12:20:47 -03:00
ab1f2df476 Read Drive config from Vault at runtime with fallback defaults
Some checks failed
BotServer CI / build (push) Failing after 7m26s
2026-03-17 00:00:36 -03:00
b57c53e2ff Remove WORKFLOW_PLAN.md (moved to gb/prompts)
Some checks failed
BotServer CI / build (push) Failing after 7m22s
2026-03-16 23:40:56 -03:00
7849031ffe Move WORKFLOW_PLAN.md to src/basic/
Some checks failed
BotServer CI / build (push) Has been cancelled
2026-03-16 23:38:35 -03:00
ec1e203859 HEAR: add configurable timeout (hear-timeout-secs, default 1h)
Some checks failed
BotServer CI / build (push) Has been cancelled
2026-03-16 23:12:45 -03:00
dae78479d2 Fix HEAR: block thread on channel instead of throw+re-run from top
Some checks failed
BotServer CI / build (push) Failing after 6m54s
- Add hear_channels: HashMap<Uuid, SyncSender<String>> to AppState
- HEAR now blocks the spawn_blocking thread via sync_channel recv()
- deliver_hear_input() called at top of stream_response() to unblock
- Script continues from exact HEAR position, no side-effect re-execution
- All three HEAR variants (basic, AS TYPE, AS MENU) use same mechanism
2026-03-16 23:00:17 -03:00
6c139fdf63 fix: suggestion and talk keyword improvements
All checks were successful
BotServer CI / build (push) Successful in 13m59s
2026-03-16 13:29:14 -03:00
f062cc4a3e Add THINK KB keyword for explicit knowledge base reasoning
- Implement think_kb.rs with structured search results
- Add keyword registration in mod.rs files
- Include confidence scoring and multi-KB support
2026-03-16 08:15:21 -03:00
7fb73e683f feat: add campaigns, attendance SLA, and marketing modules 2026-03-14 16:35:42 -03:00
13892b3157 Fix tenant-org-bot relationship and CRM lead form 2026-03-12 18:19:18 -03:00
ad4aca21ff feat: implement WhatsApp bursting rate limit and consume activation tokens
All checks were successful
BotServer CI / build (push) Successful in 11m4s
2026-03-10 21:18:14 -03:00
1053c86a73 fix: whatsapp dynamic routing and openai tool call accumulation
All checks were successful
BotServer CI / build (push) Successful in 13m40s
2026-03-10 17:19:17 -03:00
260a13e77d refactor: apply various fixes across botserver
Some checks failed
BotServer CI / build (push) Has been cancelled
2026-03-10 15:15:21 -03:00
5404e3e7ba feat: Enhance KB context, embedding generator, and website crawler
- Improved kb_context with better context management
- Enhanced embedding_generator with extended functionality (+231 lines)
- Updated kb_indexer with improved indexing logic
- Expanded website_crawler_service capabilities (+230 lines)
- Updated use_website keyword implementation
- Refined bootstrap_manager and utils
- Improved drive monitoring and local file monitor
- Added server enhancements
2026-03-04 15:43:37 -03:00
c326581a9e fix(zitadel): resolve OAuth client initialization timing issue
- Fix PAT extraction timing with retry loop (waits up to 60s for PAT in logs)
- Add sync command to flush filesystem buffers before extraction
- Improve logging with progress messages and PAT verification
- Refactor setup code into consolidated setup.rs module
- Fix YAML indentation for PatPath and MachineKeyPath
- Change Zitadel init parameter from --config to --steps

The timing issue occurred because:
1. Zitadel writes PAT to logs at startup (~18:08:59)
2. Post-install extraction ran too early (~18:09:35)
3. PAT file wasn't created until ~18:10:38 (63s after installation)
4. OAuth client creation failed because PAT file didn't exist yet

With the retry loop:
- Waits for PAT to appear in logs with sync+grep check
- Extracts PAT immediately when found
- OAuth client creation succeeds
- directory_config.json saved with valid credentials
- Login flow works end-to-end

Tested: Full reset.sh and login verification successful
2026-03-01 19:06:09 -03:00
2c92a81302 merge: Unify master into main - all commits unified
Some checks failed
BotServer CI / build (push) Failing after 6m9s
2026-03-01 07:43:07 -03:00
8f495c75ec WIP: Local changes before merging master into main 2026-03-01 07:40:11 -03:00
0b1b17406d chore(core): various email and security updates 2026-02-24 19:02:48 -03:00
1856215d05 chore: update dependencies and formatting
All checks were successful
BotServer CI / build (push) Successful in 7m30s
2026-02-22 15:55:39 -03:00
affb702e90 fix(tools): auto-create work directory if missing in production
All checks were successful
BotServer CI / build (push) Successful in 8m12s
Added logic to create botserver-stack/data/system/work directory
if it doesn't exist. This ensures production deployments work
without manual directory setup.

Changes:
- Added fs::create_dir_all() in use_tool.rs
- Added fs::create_dir_all() in tool_context.rs
- Logs when directory is created
- Fixes production deployment where /system/work may not exist

This ensures the tool loading works in fresh production environments
where the work folder hasn't been populated yet.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-02-22 12:58:58 -03:00
febfa2e708 fix(tools): use relative path for tool .mcp.json files
All checks were successful
BotServer CI / build (push) Successful in 8m37s
Changed tool loading to use relative path from current directory instead
of hardcoded HOME/gb path. This makes the code portable across different
deployment environments.

- Updated use_tool.rs to use std::env::current_dir()
- Updated tool_context.rs to use std::env::current_dir()
- Added PathBuf import to both files
- Tools now load from botserver-stack/data/system/work/

Fixes issue where tools weren't being loaded because .mcp.json files
were in a different location than expected.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-02-22 11:21:07 -03:00
de017241f2 fix: Complete security remediation - RCE and SSRF fixes
All checks were successful
BotServer CI / build (push) Successful in 7m34s
- Fixed RCE vulnerability in trusted_shell_script_arg execution
- Fixed SSRF vulnerability in GET command with internal IP blocking
- Updated SafeCommand to use explicit positional arguments

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-20 01:14:14 +00:00
e143968179 feat: Add JWT secret rotation and health verification
SEC-02: Implement credential rotation security improvements

- Add JWT secret rotation to rotate-secret command
- Generate 64-character HS512-compatible secrets
- Automatic .env backup with timestamp
- Atomic file updates via temp+rename pattern
- Add health verification for rotated credentials
- Route rotate-secret, rotate-secrets, vault commands in CLI
- Add verification attempts for database and JWT endpoints

Security improvements:
- JWT_SECRET now rotatable (previously impossible)
- Automatic rollback via backup files
- Health checks catch configuration errors
- Clear warnings about token invalidation

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-19 19:42:41 +00:00
98813fbdc8 chore: Fix warnings and clean TODO refs 2026-02-19 12:18:39 +00:00
d7211a6c19 fix: Resolve unused import and variable warnings 2026-02-19 11:48:17 +00:00
3b21ab5ef9 fix: prevent duplicate message in chat when tool is executed
- Track tool_was_executed flag in stream_response
- Send empty content in final is_complete message when tool already sent results
- Prevents the LLM's pre-tool text from appearing twice in the chat UI
- DB message saving is unaffected (uses full_response_clone before the check)
2026-02-18 20:31:34 +00:00
b1118f977d fix: Correct parameter names in tool .bas files to match database schema
- Tool 06: Change tipoExibicao to tipoDescricao (matches pedidos_uso_imagem table)
- Tool 07: Change tipoExibicao to categoriaDescricao (matches licenciamentos table)
- Both tools now compile and execute successfully with database inserts

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-18 17:51:47 +00:00
f7c60362e3 fix: Add SAVE statement conversion for tool compilation
- Add convert_save_for_tools() to convert SAVE to INSERT syntax
- Generate map-based INSERT: let __data__ = #{field: value, ...}; INSERT "table", __data__
- Fix parameter names to match database schema (tipoExibicao -> tipoDescricao)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-18 17:50:04 +00:00
9b86b204f2 fix: Add # comment support and remove hardcoded column lists
- Support # as comment marker like ' in BASIC preprocessor
- Remove hardcoded column lists from get_table_field_names()
- Let runtime use database schema dynamically via get_table_columns()
- Fix SELECT/CASE conversion to add semicolons to body statements

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-18 17:19:30 +00:00