Commit graph

82 commits

Author SHA1 Message Date
e992ed3b39 Enforce Vault-only secrets: remove env var fallbacks, all secrets from Vault
Some checks are pending
BotServer CI/CD / build (push) Waiting to run
- Remove all std::env::var calls except VAULT_* and PORT
- get_from_env returns hardcoded defaults only (no env var reading)
- Auth config, rate limits, email, analytics, calendar all use Vault
- WORK_PATH replaced with get_work_path() helper reading from Vault
- .env on production cleaned to only VAULT_ADDR, VAULT_TOKEN, VAULT_CACERT, PORT
- All service IPs/credentials stored in Vault secret/gbo/*
2026-04-03 07:11:40 -03:00
ab1f2df476 Read Drive config from Vault at runtime with fallback defaults
Some checks failed
BotServer CI / build (push) Failing after 7m26s
2026-03-17 00:00:36 -03:00
dae78479d2 Fix HEAR: block thread on channel instead of throw+re-run from top
Some checks failed
BotServer CI / build (push) Failing after 6m54s
- Add hear_channels: HashMap<Uuid, SyncSender<String>> to AppState
- HEAR now blocks the spawn_blocking thread via sync_channel recv()
- deliver_hear_input() called at top of stream_response() to unblock
- Script continues from exact HEAR position, no side-effect re-execution
- All three HEAR variants (basic, AS TYPE, AS MENU) use same mechanism
2026-03-16 23:00:17 -03:00
d1cb6b758c Fix LXD container mode: PATH, socket proxy, exec
All checks were successful
BotServer CI / build (push) Successful in 10m54s
2026-03-15 20:00:06 -03:00
7fb73e683f feat: add campaigns, attendance SLA, and marketing modules 2026-03-14 16:35:42 -03:00
13892b3157 Fix tenant-org-bot relationship and CRM lead form 2026-03-12 18:19:18 -03:00
c072fb936e fix(llm): load system-prompt from config.csv correctly
All checks were successful
BotServer CI / build (push) Successful in 17m27s
- Move system_prompt retrieval inside spawn_blocking closure
- Include system_prompt in the return tuple to fix scope issue
- Add trace logging for debugging system-prompt loading
- GLM-5 and other LLM providers now correctly receive custom system prompts

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-03-09 11:55:05 -03:00
5404e3e7ba feat: Enhance KB context, embedding generator, and website crawler
- Improved kb_context with better context management
- Enhanced embedding_generator with extended functionality (+231 lines)
- Updated kb_indexer with improved indexing logic
- Expanded website_crawler_service capabilities (+230 lines)
- Updated use_website keyword implementation
- Refined bootstrap_manager and utils
- Improved drive monitoring and local file monitor
- Added server enhancements
2026-03-04 15:43:37 -03:00
0b1b17406d chore(core): various email and security updates 2026-02-24 19:02:48 -03:00
1856215d05 chore: update dependencies and formatting
All checks were successful
BotServer CI / build (push) Successful in 7m30s
2026-02-22 15:55:39 -03:00
e143968179 feat: Add JWT secret rotation and health verification
SEC-02: Implement credential rotation security improvements

- Add JWT secret rotation to rotate-secret command
- Generate 64-character HS512-compatible secrets
- Automatic .env backup with timestamp
- Atomic file updates via temp+rename pattern
- Add health verification for rotated credentials
- Route rotate-secret, rotate-secrets, vault commands in CLI
- Add verification attempts for database and JWT endpoints

Security improvements:
- JWT_SECRET now rotatable (previously impossible)
- Automatic rollback via backup files
- Health checks catch configuration errors
- Clear warnings about token invalidation

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-19 19:42:41 +00:00
ac5b814536 fix(security): Fix unsafe code, CORS logic, and expect usage 2026-02-19 12:06:05 +00:00
b1118f977d fix: Correct parameter names in tool .bas files to match database schema
- Tool 06: Change tipoExibicao to tipoDescricao (matches pedidos_uso_imagem table)
- Tool 07: Change tipoExibicao to categoriaDescricao (matches licenciamentos table)
- Both tools now compile and execute successfully with database inserts

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-18 17:51:47 +00:00
4b44602d39 Update bootstrap and core modules
All checks were successful
BotServer CI / build (push) Successful in 10m27s
2026-02-14 09:54:14 +00:00
17cb4ef147 Fix: Update source files and fix notify dependency reference
All checks were successful
BotServer CI / build (push) Successful in 11m35s
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-13 22:30:57 +00:00
5ea171d126 Refactor: Split large files into modular subdirectories
Some checks failed
BotServer CI / build (push) Failing after 1m34s
Split 20+ files over 1000 lines into focused subdirectories for better
maintainability and code organization. All changes maintain backward
compatibility through re-export wrappers.

Major splits:
- attendance/llm_assist.rs (2074→7 modules)
- basic/keywords/face_api.rs → face_api/ (7 modules)
- basic/keywords/file_operations.rs → file_ops/ (8 modules)
- basic/keywords/hear_talk.rs → hearing/ (6 modules)
- channels/wechat.rs → wechat/ (10 modules)
- channels/youtube.rs → youtube/ (5 modules)
- contacts/mod.rs → contacts_api/ (6 modules)
- core/bootstrap/mod.rs → bootstrap/ (5 modules)
- core/shared/admin.rs → admin_*.rs (5 modules)
- designer/canvas.rs → canvas_api/ (6 modules)
- designer/mod.rs → designer_api/ (6 modules)
- docs/handlers.rs → handlers_api/ (11 modules)
- drive/mod.rs → drive_handlers.rs, drive_types.rs
- learn/mod.rs → types.rs
- main.rs → main_module/ (7 modules)
- meet/webinar.rs → webinar_api/ (8 modules)
- paper/mod.rs → (10 modules)
- security/auth.rs → auth_api/ (7 modules)
- security/passkey.rs → (4 modules)
- sources/mod.rs → sources_api/ (5 modules)
- tasks/mod.rs → task_api/ (5 modules)

Stats: 38,040 deletions, 1,315 additions across 318 files

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-12 21:09:30 +00:00
355215c2a2 Update: refactor migrations, update source files, and add new features 2026-02-04 13:29:29 -03:00
b103c07248 Fix migration errors and reorganize migration files
- Fixed 'relation session_kb_associations does not exist' error in core consolidated migration.
- Renamed migration directories from timestamp-based to version-based (6.0.x, 6.1.x, 6.2.x).
- Reorganized migrations into dedicated feature folders (products, dashboards, learn, video).
- Updated migration execution order in core/shared/utils.rs.
- Moves legacy migrations to 6.0.x/6.1.x and workflow to 6.2.0.
2026-01-27 13:45:54 -03:00
9087bb17cd feat: complete General Bots 7.0 (v6.2.0) 2026-01-25 10:29:54 -03:00
0a24cd4b50 Fix build errors and unused imports in core, security and package_manager modules 2026-01-24 22:04:47 -03:00
6fa52e1dd8 feat: implement feature bundling architecture and fix conditional compilation
- Restructured Cargo.toml with Bundle Pattern for easy feature selection
- Added feature bundles: tasks → automation + drive + monitoring
- Applied conditional compilation guards throughout codebase:
  * AppState fields (drive, cache, task_engine, task_scheduler)
  * main.rs initialization (S3, Redis, Tasks)
  * SessionManager Redis usage
  * bootstrap S3/Drive operations
  * compiler task scheduling
  * shared module Task/NewTask exports
- Eliminated all botserver compilation warnings
- Minimal build now compiles successfully
- Accepted core dependencies: automation (Rhai), drive (S3), cache (Redis)
- Created DEPENDENCY_FIX_PLAN.md with complete documentation

Minimal feature set: chat + automation + drive + cache
Verified: cargo check -p botserver --no-default-features --features minimal 
2026-01-23 13:14:20 -03:00
66abce913f Feature gating refactor: modular compilation with minimal feature set 2026-01-22 19:45:18 -03:00
3db87c029d Modularize learn schema and clean up channels/project schema stubs 2026-01-22 14:01:50 -03:00
fc9c07d204 Refactor schema.rs into modular files 2026-01-22 13:57:40 -03:00
18b8afd54c O 2026-01-19 21:19:10 -03:00
4f63065da2 Update botserver submodule 2026-01-19 15:43:45 -03:00
5126c648ff Auto-commit: 20260118_195334 2026-01-18 19:53:34 -03:00
033bb504b9 Various updates: dependencies, features, and bug fixes 2026-01-16 11:29:22 -03:00
446dee26b6 Add extended product fields: dimensions, weight, tax codes, brand, inventory, pricing, SEO 2026-01-14 10:02:55 -03:00
95a7591c39 Add Stone Pagamentos fields: SKU, dimensões, peso, NCM, GTIN, tributos, etc 2026-01-14 09:50:39 -03:00
a2783f9b32 Fix 5 errors and 32 warnings: calendar, compliance, billing_alert_broadcast, unused vars 2026-01-13 22:21:25 -03:00
31777432b4 Implement TODO items: session auth, face API, task logs, intent storage
Learn Module:
- All 9 handlers now use AuthenticatedUser extractor

Security:
- validate_session_sync reads roles from SESSION_CACHE

AutoTask:
- get_task_logs reads from manifest with status logs
- store_compiled_intent saves to cache and database

Face API:
- AWS Rekognition, OpenCV, InsightFace implementations
- Detection, verification, analysis methods

Other fixes:
- Calendar/task integration database queries
- Recording database methods
- Analytics insights trends
- Email/folder monitoring mock data
2026-01-13 14:48:49 -03:00
a886478548 Implement database persistence for dashboards, legal, and compliance modules
- Add PostgreSQL persistence for dashboards module (was returning empty vec![])
  - Tables: dashboards, dashboard_widgets, dashboard_data_sources, dashboard_filters,
    dashboard_widget_data_sources, conversational_queries
  - Full CRUD operations with spawn_blocking pattern

- Add PostgreSQL persistence for legal module (was using in-memory HashMap)
  - Tables: legal_documents, legal_document_versions, cookie_consents, consent_history,
    legal_acceptances, data_deletion_requests, data_export_requests
  - GDPR-compliant consent tracking and document management

- Add PostgreSQL persistence for compliance module (was returning empty results)
  - Tables: compliance_checks, compliance_issues, compliance_audit_log, compliance_evidence,
    compliance_risk_assessments, compliance_risks, compliance_training_records,
    compliance_access_reviews
  - Support for GDPR, SOC2, ISO27001, HIPAA, PCI-DSS frameworks

- Add migration files for all new tables
- Update schema.rs with new table definitions and joinables
- Register new routes in main.rs
- Add recursion_limit = 512 for macro expansion
2026-01-13 00:07:22 -03:00
faeae250bc Add security protection module with sudo-based privilege escalation
- Create installer.rs for 'botserver install protection' command
- Requires root to install packages and create sudoers config
- Sudoers uses exact commands (no wildcards) for security
- Update all tool files (lynis, rkhunter, chkrootkit, suricata, lmd) to use sudo
- Update manager.rs service management to use sudo
- Add 'sudo' and 'visudo' to command_guard.rs whitelist
- Update CLI with install/remove/status protection commands

Security model:
- Installation requires root (sudo botserver install protection)
- Runtime uses sudoers NOPASSWD for specific commands only
- No wildcards in sudoers - exact command specifications
- Tools run on host system, not in containers
2026-01-10 09:41:12 -03:00
b674d85583 Fix SafeCommand to allow shell scripts with redirects and command chaining
- Add shell_script_arg() method for bash/sh/cmd -c scripts
- Allow > < redirects in shell scripts (blocked in regular args)
- Allow && || command chaining in shell scripts
- Update safe_sh_command functions to use shell_script_arg
- Update run_commands, start, and LLM server commands
- Block dangerous patterns: backticks, path traversal
- Fix struct field mismatches and type errors
2026-01-08 23:50:38 -03:00
5919aa6bf0 Add video module, RBAC, security features, billing, contacts, dashboards, learn, social, and multiple new modules
Major additions:
- Video editing engine with AI features (transcription, captions, TTS, scene detection)
- RBAC middleware and organization management
- Security enhancements (MFA, passkey, DLP, encryption, audit)
- Billing and subscription management
- Contacts management
- Dashboards module
- Learn/LMS module
- Social features
- Compliance (SOC2, SOP middleware, vulnerability scanner)
- New migrations for RBAC, learn, and video tables
2026-01-08 13:16:17 -03:00
479950945b feat(auth): Add OTP password display on bootstrap and fix Zitadel login flow
- Add generate_secure_password() for OTP generation during admin bootstrap
- Display admin credentials (username/password) in console on first run
- Save credentials to ~/.gb-setup-credentials file
- Fix Zitadel client to support PAT token authentication
- Replace OAuth2 password grant with Zitadel Session API for login
- Fix get_current_user to fetch user data from Zitadel session
- Return session_id as access_token for proper authentication
- Set email as verified on user creation to skip verification
- Add password grant type to OAuth application config
- Update directory_setup to include proper redirect URIs
2026-01-06 22:56:35 -03:00
29b80f597c Fix email_accounts -> user_email_accounts table name typo in list_emails_htmx 2026-01-04 08:48:27 -03:00
c743754c6c Add per-bot database creation
- Added database_name field to bots schema
- Bot creation now creates a dedicated PostgreSQL database (bot_{name})
- Updated add_bot.rs to create database and store database_name
- Added create_bot_database() function with safe name validation
- Added dynamic table check to all db_api handlers
2026-01-02 19:34:59 -03:00
0385047c5c Fix task progress: real-time updates, MIME types, WebSocket event types
- Fix MIME type for app files by preserving directory structure in sanitize_file_path()
- Add with_event_type() to TaskProgressEvent for correct WebSocket event types
- broadcast_manifest_update() now sends 'manifest_update' type correctly
- update_item_status() broadcasts automatically for real-time file progress
2025-12-31 23:45:29 -03:00
50d58ff59f Fix task UI and MinIO app generation
- Fix MinIO bucket name sanitization (replace spaces with hyphens)
- Write apps to MinIO path: botname.gbapp/appname/files
- Serve apps directly from MinIO via /apps/:app_name route
- Add WebSocket reconnection on HTMX page load
- Remove sync_app_to_site_root (drive monitor handles sync)
2025-12-31 12:38:35 -03:00
061c14b4a2 Fix tasks UI, WebSocket progress, memory monitoring, and app generator
Tasks UI fixes:
- Fix task list to query auto_tasks table instead of tasks table
- Fix task detail endpoint to use UUID binding for auto_tasks query
- Add proper filter handling: complete, active, awaiting, paused, blocked
- Add TaskStats fields: awaiting, paused, blocked, time_saved
- Add /api/tasks/time-saved endpoint
- Add count-all to stats HTML response

App generator improvements:
- Add AgentActivity struct for detailed terminal-style progress
- Add emit_activity method for rich progress events
- Add detailed logging for LLM calls with timing
- Track files_written, tables_synced, bytes_generated

Memory and performance:
- Add memory_monitor module for tracking RSS and thread activity
- Skip 0-byte files in drive monitor and document processor
- Change DRIVE_MONITOR checking logs from info to trace
- Remove unused profile_section macro

WebSocket progress:
- Ensure TaskProgressEvent includes activity field
- Add with_activity builder method
2025-12-30 22:42:32 -03:00
b0baf36b11 Fix TLS configuration for MinIO, Qdrant, and template structure
- Fix MinIO health check to use HTTPS instead of HTTP
- Add Vault connectivity check before fetching credentials
- Add CA cert configuration for S3 client
- Add Qdrant vector_db setup with TLS configuration
- Fix Qdrant default URL to use HTTPS
- Always sync templates to S3 buckets (not just on create)
- Skip .gbkb root files, only index files in subfolders
2025-12-29 18:21:03 -03:00
7634429896 Consolidate all database migrations into single optimized schema
BREAKING CHANGE: Complete database redesign

Migrations:
- Remove all 6.x.x and 7.0.0 migrations
- Create single consolidated migration 20250101000000_consolidated_schema
- Add Diesel initial setup migration

Schema Optimizations:
- All domain values use SMALLINT instead of VARCHAR/TEXT (2 bytes vs 20+ bytes)
- Sharding infrastructure: shard_config, tenant_shard_map tables
- All tables include tenant_id and shard_id for horizontal scaling
- Proper indexing strategies for high-throughput queries
- Foreign key constraints for data integrity
- Updated_at triggers via diesel_manage_updated_at

Tables (35 total):
- Core: tenants, users, bots, bot_configuration, bot_channels
- Sessions: user_sessions, message_history
- Memory: bot_memories
- Auto Tasks: auto_tasks, execution_plans, task_approvals, task_decisions
- Safety: safety_audit_log, intent_classifications
- Apps: generated_apps, designer_changes, designer_pending_changes
- KB: kb_collections, kb_documents, session_kb_associations, kb_sources
- Tools: tools, system_automations, pending_info
- Analytics: usage_analytics, analytics_events
- Tasks: tasks, task_comments
- Accounts: connected_accounts, session_account_associations
- Communication: whatsapp_numbers, clicks
- Access: table_role_access, context_injections
- Organizations: organizations, user_organizations

Rust Schema:
- Updated schema.rs to match new consolidated schema
- All diesel table definitions with proper types
- Foreign key relationships defined

WARNING: This is a DESTRUCTIVE migration - drop existing database before applying
2025-12-29 12:29:23 -03:00
1f150228af Add billion-scale database redesign with enums and sharding
Database Schema v7.0.0:
- Create new 'gb' schema with PostgreSQL ENUMs instead of VARCHAR for all domain values
- Add sharding infrastructure (shard_config, tenant_shard_map tables)
- Implement partitioned tables for sessions, messages, and analytics (monthly partitions)
- Add Snowflake-like ID generation for distributed systems
- Design for billion-user scale with proper indexing strategies

Rust Enums:
- Add comprehensive enum types in core/shared/enums.rs
- Implement ToSql/FromSql for Diesel ORM integration
- Include: ChannelType, MessageRole, MessageType, LlmProvider, ContextProvider
- Include: TaskStatus, TaskPriority, ExecutionMode, RiskLevel, ApprovalStatus, IntentType
- All enums stored as SMALLINT for efficiency

Other fixes:
- Fix hardcoded gpt-4 model in auto_task modules to use bot config
- Add vector_db to required bootstrap components
- Add Qdrant health check before KB indexing
- Change verbose START messages to trace level
- Fix episodic memory role handling in Claude client
- Disable auth for /api routes during development

This is a DESTRUCTIVE migration - only for fresh installations.
2025-12-29 11:27:13 -03:00
a5dee11002 Security audit: Remove all production .unwrap()/.expect(), add SafeCommand, ErrorSanitizer
- Phase 1 Critical: All 115 .unwrap() verified in test code only
- Phase 1 Critical: All runtime .expect() converted to proper error handling
- Phase 2 H1: Antivirus commands now use SafeCommand (added which/where to whitelist)
- Phase 2 H2: db_api.rs error responses use log_and_sanitize()
- Phase 2 H5: Removed duplicate sanitize_identifier (re-exports from sql_guard)

32 files modified for security hardening.
Moon deployment criteria: 10/10 met
2025-12-28 21:26:08 -03:00
c67aaa677a feat(security): Complete security infrastructure implementation
SECURITY MODULES ADDED:
- security/auth.rs: Full RBAC with roles (Anonymous, User, Moderator, Admin, SuperAdmin, Service, Bot, BotOwner, BotOperator, BotViewer) and permissions
- security/cors.rs: Hardened CORS (no wildcard in production, env-based config)
- security/panic_handler.rs: Panic catching middleware with safe 500 responses
- security/path_guard.rs: Path traversal protection, null byte prevention
- security/request_id.rs: UUID request tracking with correlation IDs
- security/error_sanitizer.rs: Sensitive data redaction from responses
- security/zitadel_auth.rs: Zitadel token introspection and role mapping
- security/sql_guard.rs: SQL injection prevention with table whitelist
- security/command_guard.rs: Command injection prevention
- security/secrets.rs: Zeroizing secret management
- security/validation.rs: Input validation utilities
- security/rate_limiter.rs: Rate limiting with governor crate
- security/headers.rs: Security headers (CSP, HSTS, X-Frame-Options)

MAIN.RS UPDATES:
- Replaced tower_http::cors::Any with hardened create_cors_layer()
- Added panic handler middleware
- Added request ID tracking middleware
- Set global panic hook

SECURITY STATUS:
- 0 unwrap() in production code
- 0 panic! in production code
- 0 unsafe blocks
- cargo audit: PASS (no vulnerabilities)
- Estimated completion: ~98%

Remaining: Wire auth middleware to handlers, audit logs for sensitive data
2025-12-28 19:29:18 -03:00
561264521c Add SECURITY_TASKS.md - security audit checklist, consolidate duplicate utils 2025-12-28 15:32:48 -03:00
14b7cf70af feat(autotask): Implement AutoTask system with intent classification and app generation
- Add IntentClassifier with 7 intent types (APP_CREATE, TODO, MONITOR, ACTION, SCHEDULE, GOAL, TOOL)
- Add AppGenerator with LLM-powered app structure analysis
- Add DesignerAI for modifying apps through conversation
- Add app_server for serving generated apps with clean URLs
- Add db_api for CRUD operations on bot database tables
- Add ask_later keyword for pending info collection
- Add migration 6.1.1 with tables: pending_info, auto_tasks, execution_plans, task_approvals, task_decisions, safety_audit_log, generated_apps, intent_classifications, designer_changes
- Write apps to S3 drive and sync to SITE_ROOT for serving
- Clean URL structure: /apps/{app_name}/
- Integrate with DriveMonitor for file sync

Based on Chapter 17 - Autonomous Tasks specification
2025-12-27 21:10:09 -03:00
5da86bbef2 Fix clippy warnings: match arms, async/await, Debug impls, formatting
- Fix match arms with identical bodies by consolidating patterns
- Fix case-insensitive file extension comparisons using eq_ignore_ascii_case
- Fix unnecessary Debug formatting in log/format macros
- Fix clone_from usage instead of clone assignment
- Fix let...else patterns where appropriate
- Fix format! append to String using write! macro
- Fix unwrap_or with function calls to use unwrap_or_else
- Add missing fields to manual Debug implementations
- Fix duplicate code in if blocks
- Add type aliases for complex types
- Rename struct fields to avoid common prefixes
- Various other clippy warning fixes

Note: Some 'unused async' warnings remain for functions that are
called with .await but don't contain await internally - these are
kept async for API compatibility.
2025-12-26 08:59:25 -03:00