diff --git a/src/17-testing/README.md b/src/17-testing/README.md new file mode 100644 index 00000000..e4c58b09 --- /dev/null +++ b/src/17-testing/README.md @@ -0,0 +1,76 @@ +# Testing + +General Bots uses a comprehensive testing framework including unit tests, integration tests, and end-to-end (E2E) tests to ensure platform reliability and quality. + +## Overview + +The testing strategy covers: + +- **Unit Tests** - Individual component testing +- **Integration Tests** - Service interaction testing +- **E2E Tests** - Complete user journey validation + +## Test Structure + +All tests are organized in the `bottest` package: + +``` +bottest/ +├── src/ # Test utilities and harness +├── tests/ +│ ├── unit/ # Unit tests +│ ├── integration/ # Integration tests +│ └── e2e/ # End-to-end tests +├── benches/ # Performance benchmarks +└── Cargo.toml +``` + +## Running Tests + +### All Tests +```bash +cd gb/bottest +cargo test +``` + +### Specific Test Types +```bash +# Unit tests +cargo test --lib + +# Integration tests +cargo test --test integration + +# E2E tests +cargo test --test e2e -- --nocapture +``` + +## Test Harness + +The test harness provides utilities for setting up test environments: + +```rust +use bottest::prelude::*; + +#[tokio::test] +async fn my_test() { + let ctx = TestHarness::full().await.unwrap(); + // Test code here + ctx.cleanup().await.unwrap(); +} +``` + +## Continuous Integration + +Tests run automatically on: +- Pull requests +- Commits to main branch +- Pre-release checks + +See the repository's CI/CD configuration for details. + +## Next Steps + +- [End-to-End Testing](./e2e-testing.md) - Browser automation and user flow testing +- [Performance Testing](./performance.md) - Benchmarking and profiling +- [Test Architecture](./architecture.md) - Design patterns and best practices \ No newline at end of file diff --git a/src/17-testing/architecture.md b/src/17-testing/architecture.md new file mode 100644 index 00000000..3b4b8eb0 --- /dev/null +++ b/src/17-testing/architecture.md @@ -0,0 +1,410 @@ +# Testing Architecture + +## Overview + +The General Bots testing framework is designed with a multi-layered, isolated approach to ensure comprehensive coverage from individual components to complete user workflows. + +## Architecture Diagram + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Test Execution Layer │ +│ (GitHub Actions, CI/CD, Local Development) │ +└────────────────────┬────────────────────────────────────────┘ + │ + ┌────────────┼────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌─────────┐ ┌─────────┐ ┌──────────┐ + │ Unit │ │ Integr. │ │ E2E │ + │ Tests │ │ Tests │ │ Tests │ + └────┬────┘ └────┬────┘ └─────┬────┘ + │ │ │ + └────────────┼────────────┘ + │ + ┌────────────▼────────────┐ + │ Test Harness Layer │ + │ (Context, Utils, Mocks) │ + └────────────┬────────────┘ + │ + ┌────────────┼────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌─────────┐ ┌─────────┐ ┌──────────┐ + │BotServer│ │ Browser│ │ Services │ + │(Testing)│ │ (WebDrv)│ │(Mock/Iso)│ + └─────────┘ └─────────┘ └──────────┘ + │ │ │ + └────────────┼────────────┘ + │ + ┌────────────▼────────────┐ + │ Temporary Stack Layer │ + │ (Isolated Environments) │ + └────────────┬────────────┘ + │ + ┌────────────┴────────────┐ + │ │ + ▼ ▼ + ┌─────────────┐ ┌──────────────┐ + │ PostgreSQL │ │ Redis, MinIO │ + │ (Isolated) │ │ (Isolated) │ + └─────────────┘ └──────────────┘ +``` + +## Test Layers + +### 1. Unit Tests + +**Purpose**: Test individual components in isolation + +**Scope**: +- Single functions or methods +- Mocked external dependencies +- No database or external services + +**Example**: +```rust +#[test] +fn test_message_formatting() { + let msg = format_message("Hello"); + assert_eq!(msg, "Hello!"); +} +``` + +**Location**: `bottest/tests/unit/` + +### 2. Integration Tests + +**Purpose**: Test multiple components working together + +**Scope**: +- Multi-component interactions +- Real database connections +- Service integration +- Error handling across components + +**Example**: +```rust +#[tokio::test] +async fn test_message_storage_and_retrieval() { + let db = setup_test_db().await; + let msg = Message::new("Hello"); + db.save(&msg).await.unwrap(); + let retrieved = db.get(msg.id).await.unwrap(); + assert_eq!(retrieved.text, "Hello"); +} +``` + +**Location**: `bottest/tests/integration/` + +### 3. End-to-End Tests + +**Purpose**: Test complete user workflows + +**Scope**: +- Complete user journeys +- Browser interactions +- Multi-phase workflows +- Real-world scenarios + +**Phases**: +1. Platform Loading +2. BotServer Initialization +3. User Authentication +4. Chat Interaction +5. Logout & Session Management + +**Example**: +```rust +#[tokio::test] +async fn test_complete_platform_flow_login_chat_logout() { + let ctx = E2ETestContext::setup_with_browser().await?; + + verify_platform_loading(&ctx).await?; + verify_botserver_running(&ctx).await?; + test_user_login(browser, &ctx).await?; + test_chat_interaction(browser, &ctx).await?; + test_user_logout(browser, &ctx).await?; + + ctx.close().await; +} +``` + +**Location**: `bottest/tests/e2e/` + +## Test Harness + +The test harness provides utilities for test setup and context management: + +``` +TestHarness +├── Setup utilities +│ ├── Create test database +│ ├── Start mock services +│ ├── Initialize configurations +│ └── Provision test data +├── Context management +│ ├── Resource tracking +│ ├── Cleanup coordination +│ └── Error handling +└── Helper functions + ├── HTTP requests + ├── Browser interactions + └── Service mocking +``` + +### E2ETestContext + +Provides complete environment for E2E testing: + +```rust +pub struct E2ETestContext { + pub ctx: TestContext, + pub server: BotServerInstance, + pub browser: Option, +} + +impl E2ETestContext { + pub async fn setup() -> Result + pub async fn setup_with_browser() -> Result + pub fn base_url(&self) -> &str + pub fn has_browser(&self) -> bool + pub async fn close(self) +} +``` + +## Temporary Stack Architecture + +Isolated test environments for complete system integration: + +``` +/tmp/botserver-test-{timestamp}-{id}/ +├── postgres/ +│ ├── data/ ← PostgreSQL data files +│ ├── postgresql.log ← Database logs +│ └── postgresql.conf ← Configuration +├── redis/ +│ ├── data/ ← Redis persistence +│ └── redis.log +├── minio/ +│ ├── data/ ← S3-compatible storage +│ └── minio.log +├── botserver/ +│ ├── config/ +│ │ ├── config.toml ← Application config +│ │ └── .env ← Environment variables +│ ├── logs/ +│ │ ├── botserver.log ← Main application logs +│ │ ├── api.log ← API logs +│ │ └── debug.log ← Debug logs +│ ├── cache/ ← Local cache +│ └── state.json ← Stack metadata +└── env.stack ← Connection strings for tests +``` + +## Isolation Strategy + +### Service Isolation + +Each test gets dedicated service instances: + +- **Database**: Separate PostgreSQL cluster on port 5433 +- **Cache**: Separate Redis instance on port 6380 +- **Storage**: Separate MinIO instance on port 9001 +- **API**: Separate BotServer on port 8000 + +### Network Isolation + +- All services on localhost (127.0.0.1) +- Non-standard ports to avoid conflicts +- Docker containers for complete OS-level isolation + +### Data Isolation + +- Separate database schemas per test +- Temporary file systems for storage +- No shared configuration between tests +- Automatic cleanup on completion + +## Test Execution Flow + +``` +1. Test Initialization + ├─ Parse environment variables + ├─ Check prerequisites (WebDriver, services) + └─ Create test context + +2. Stack Setup + ├─ Create temporary directory + ├─ Initialize databases + ├─ Start services + └─ Wait for readiness + +3. Test Execution + ├─ Setup phase + ├─ Action phase + ├─ Verification phase + └─ Assertion phase + +4. Cleanup + ├─ Close browser connections + ├─ Shutdown services gracefully + ├─ Remove temporary directories + └─ Report results +``` + +## Browser Automation + +Uses WebDriver (Selenium) protocol for browser testing: + +``` +Test Code + ↓ +Reqwest HTTP Client + ↓ +WebDriver Protocol (JSON-RPC) + ↓ +chromedriver / Selenium Server + ↓ +Chrome/Chromium Browser + ↓ +Test Verification +``` + +### WebDriver Commands + +- Navigate to URL +- Find elements by selector +- Click buttons and links +- Fill form inputs +- Wait for elements +- Execute JavaScript +- Take screenshots +- Get element text + +## Error Handling + +Comprehensive error handling at all levels: + +``` +Test Execution + │ + ├─ Setup Error + │ └─ Fail fast, preserve environment + │ + ├─ Execution Error + │ ├─ Log detailed context + │ ├─ Capture screenshots + │ └─ Optionally preserve stack + │ + └─ Cleanup Error + └─ Log warning, continue cleanup +``` + +## Performance Considerations + +### Test Execution Times + +- **Unit Tests**: ~0.1-1 second +- **Integration Tests**: ~1-10 seconds +- **E2E Tests**: ~30-60 seconds +- **Full Suite**: ~2-3 minutes + +### Optimization Strategies + +1. **Parallel Execution**: Run independent tests simultaneously +2. **Caching**: Reuse expensive resources +3. **Lazy Loading**: Initialize only needed components +4. **Release Mode**: Use `--release` for faster compilation +5. **Selective Testing**: Run only relevant tests during development + +## CI/CD Integration + +### GitHub Actions Workflow + +``` +Trigger (push/PR) + ↓ +Setup Environment + ├─ Install Rust + ├─ Start WebDriver + └─ Setup test infrastructure + ↓ +Run Tests + ├─ Unit tests + ├─ Integration tests + └─ E2E tests + ↓ +Collect Artifacts + ├─ Test results + ├─ Coverage reports + ├─ Screenshots/logs + └─ Performance metrics + ↓ +Report Results + └─ Pass/fail status +``` + +## Best Practices + +### 1. Test Organization + +- Keep tests focused and single-purpose +- Use descriptive names +- Group related tests +- Organize by layer (unit/integration/e2e) + +### 2. Test Design + +- Make tests independent +- Use realistic data +- Test both happy and error paths +- Avoid test interdependencies + +### 3. Test Maintenance + +- Keep tests up to date with code +- Remove obsolete tests +- Refactor test helpers +- Monitor test execution time + +### 4. Test Documentation + +- Document complex test logic +- Explain test prerequisites +- Document setup/teardown +- Include troubleshooting tips + +## Debugging + +### Debug Helpers + +- `RUST_LOG=debug` - Verbose logging +- `HEADED=1` - Show browser UI +- `--nocapture` - Print test output +- `--test-threads=1` - Run sequentially + +### Debug Techniques + +- Check server logs +- Review screenshots +- Inspect HTTP requests +- Step through code +- Use REPL for experimentation + +## Future Enhancements + +1. **Load Testing** - Concurrent user scenarios +2. **Visual Regression** - Screenshot comparison +3. **Accessibility Testing** - WCAG compliance +4. **Security Testing** - Vulnerability scanning +5. **Performance Profiling** - Memory and CPU analysis +6. **Multi-region** - Test across deployments +7. **Snapshot Testing** - Compare outputs over time + +## References + +- [End-to-End Testing Guide](./e2e-testing.md) +- [Test Harness API](./test-harness.md) +- [CI/CD Integration](./ci-cd.md) +- [Performance Benchmarking](./performance.md) \ No newline at end of file diff --git a/src/17-testing/e2e-testing.md b/src/17-testing/e2e-testing.md new file mode 100644 index 00000000..9feb4ad6 --- /dev/null +++ b/src/17-testing/e2e-testing.md @@ -0,0 +1,447 @@ +# End-to-End Testing + +End-to-end (E2E) testing validates complete user workflows from platform loading through authentication, interaction, and logout. + +## Overview + +E2E tests simulate real user interactions: + +1. **Platform Loading** - UI and API infrastructure operational +2. **BotServer Initialization** - Backend service running and ready +3. **User Authentication** - Login workflow functional +4. **Chat Interaction** - Message sending and receiving +5. **Logout** - Session management and access control + +## Complete Platform Flow Test + +The main E2E test validates the entire user journey: + +```rust +#[tokio::test] +async fn test_complete_platform_flow_login_chat_logout() { + // Setup + let ctx = E2ETestContext::setup_with_browser().await?; + let browser = ctx.browser.as_ref().unwrap(); + + // Phase 1: Platform Loading + verify_platform_loading(&ctx).await?; + + // Phase 2: BotServer Running + verify_botserver_running(&ctx).await?; + + // Phase 3: User Login + test_user_login(browser, &ctx).await?; + + // Phase 4: Chat Interaction + test_chat_interaction(browser, &ctx).await?; + + // Phase 5: Logout + test_user_logout(browser, &ctx).await?; + + ctx.close().await; +} +``` + +## Test Phases + +### Phase 1: Platform Loading + +Verifies UI and API infrastructure: + +```rust +verify_platform_loading(&ctx).await? +``` + +Checks: +- Health endpoint responds with 2xx status +- API endpoints are accessible +- Database migrations completed +- Services are initialized + +### Phase 2: BotServer Initialization + +Verifies the backend service is operational: + +```rust +verify_botserver_running(&ctx).await? +``` + +Checks: +- Process is alive and responding +- Configuration properly loaded +- Dependencies connected (DB, cache, storage) +- Health checks pass + +### Phase 3: User Authentication + +Tests the login workflow: + +```rust +test_user_login(browser, &ctx).await? +``` + +Tests: +- Navigate to login page +- Form elements present and functional +- Accept valid test credentials (test@example.com / TestPassword123!) +- Create session and authentication token +- Redirect to dashboard/chat interface + +### Phase 4: Chat Interaction + +Tests messaging functionality: + +```rust +test_chat_interaction(browser, &ctx).await? +``` + +Tests: +- Chat interface loads correctly +- User can type and send messages +- Bot responds with valid output +- Message history persists +- Multiple exchanges work correctly + +### Phase 5: Logout & Session Management + +Tests secure session handling: + +```rust +test_user_logout(browser, &ctx).await? +``` + +Tests: +- Logout button/action works +- Session is invalidated +- User redirected to login page +- Protected routes block unauthenticated access +- Cannot access chat after logout + +## Running E2E Tests + +### HTTP-Only Tests (No Browser Required) + +These tests verify API and infrastructure without browser automation: + +```bash +cd gb/bottest + +# Platform loading verification +cargo test --test e2e test_platform_loading_http_only -- --nocapture + +# BotServer startup verification +cargo test --test e2e test_botserver_startup -- --nocapture +``` + +Execution time: ~2-5 seconds + +### Complete Flow Tests (Requires WebDriver) + +Full browser-based tests with user interactions: + +```bash +# Start WebDriver first +chromedriver --port=4444 & + +# Run complete platform flow +cargo test --test e2e test_complete_platform_flow_login_chat_logout -- --nocapture + +# Run simplified flow +cargo test --test e2e test_login_and_chat_flow -- --nocapture +``` + +Execution time: ~30-60 seconds + +## WebDriver Setup + +### Option 1: Local Installation + +```bash +# Download chromedriver from https://chromedriver.chromium.org/ +# Place in PATH, then start: +chromedriver --port=4444 +``` + +### Option 2: Docker + +```bash +docker run -d -p 4444:4444 selenium/standalone-chrome +``` + +### Option 3: Docker Compose + +```bash +docker-compose up -d webdriver +``` + +## Environment Variables + +Control test behavior: + +| Variable | Default | Purpose | +|----------|---------|---------| +| `HEADED` | unset | Show browser window instead of headless | +| `WEBDRIVER_URL` | `http://localhost:4444` | WebDriver server endpoint | +| `SKIP_E2E_TESTS` | unset | Skip E2E tests if set | +| `RUST_LOG` | info | Logging level: debug, info, warn, error | +| `KEEP_TEMP_STACK_ON_ERROR` | unset | Preserve temp directory on failure | + +### Examples + +```bash +# Show browser UI for debugging +HEADED=1 cargo test --test e2e -- --nocapture + +# Use custom WebDriver +WEBDRIVER_URL=http://localhost:4445 cargo test --test e2e -- --nocapture + +# Verbose logging +RUST_LOG=debug cargo test --test e2e -- --nocapture + +# Run single-threaded with output +cargo test --test e2e -- --nocapture --test-threads=1 +``` + +## Test Helpers + +Reusable helper functions for custom tests: + +```rust +// Verify platform is operational +verify_platform_loading(&ctx) -> Result<()> + +// Verify BotServer is running +verify_botserver_running(&ctx) -> Result<()> + +// Perform login with credentials +test_user_login(browser, &ctx) -> Result<()> + +// Send message and wait for response +test_chat_interaction(browser, &ctx) -> Result<()> + +// Logout and verify session invalidation +test_user_logout(browser, &ctx) -> Result<()> +``` + +## Test Context + +Setup a test context for E2E testing: + +```rust +use bottest::prelude::*; +use bottest::web::{Browser, BrowserConfig}; + +// HTTP-only context +let ctx = E2ETestContext::setup().await?; + +// With browser automation +let ctx = E2ETestContext::setup_with_browser().await?; +let browser = ctx.browser.as_ref().unwrap(); + +// Access base URL +let url = ctx.base_url(); + +// Access running server +let is_running = ctx.server.is_running(); + +// Cleanup +ctx.close().await; +``` + +## Common Issues + +### WebDriver Not Available + +**Problem**: Test fails with "WebDriver not available" + +**Solution**: +```bash +# Start WebDriver +chromedriver --port=4444 +# or +docker run -d -p 4444:4444 selenium/standalone-chrome +``` + +### Port Already in Use + +**Problem**: Services fail to start due to port conflicts + +**Solution**: +```bash +# Kill existing services +pkill -f chromedriver +pkill -f botserver +pkill -f postgres +pkill -f redis-server +``` + +### Test Hangs or Timeout + +**Problem**: Test appears to hang or timeout + +**Solution**: +```bash +# Run with timeout and verbose output +timeout 120s RUST_LOG=debug cargo test --test e2e test_name -- --nocapture --test-threads=1 +``` + +### Browser Connection Issues + +**Problem**: Browser fails to connect to WebDriver + +**Solution**: +```bash +# Use different WebDriver port +WEBDRIVER_URL=http://localhost:4445 cargo test --test e2e -- --nocapture +``` + +## Debugging + +### View Test Output +```bash +# Show all output +cargo test --test e2e test_name -- --nocapture + +# With timestamps +RUST_LOG=debug cargo test --test e2e test_name -- --nocapture + +# Save to file +cargo test --test e2e test_name -- --nocapture 2>&1 | tee test.log +``` + +### Watch Browser in Action +```bash +# Run with visible browser +HEADED=1 cargo test --test e2e test_name -- --nocapture --test-threads=1 +``` + +### Check Server Logs +```bash +# Monitor logs while tests run +tail -f /tmp/bottest-*/botserver.log + +# In another terminal: +cargo test --test e2e test_name -- --nocapture +``` + +## Performance + +Typical execution times: + +| Test | Time | Resources | +|------|------|-----------| +| Platform loading (HTTP-only) | ~2s | Minimal | +| BotServer startup (HTTP-only) | ~5s | Minimal | +| Login and chat flow | ~20s | Browser + Memory | +| Complete flow with all phases | ~45s | Browser + Memory | +| Full E2E test suite | ~2-3 min | High | + +Use release mode for faster execution: +```bash +cargo test --test e2e --release -- --nocapture +``` + +## CI/CD Integration + +### GitHub Actions Example + +```yaml +name: E2E Tests +on: [push, pull_request] + +jobs: + e2e: + runs-on: ubuntu-latest + services: + chromedriver: + image: selenium/standalone-chrome + options: --shm-size=2gb + steps: + - uses: actions/checkout@v3 + - uses: actions-rs/toolchain@v1 + with: + toolchain: stable + - run: cd gb/bottest && cargo test --test e2e -- --nocapture +``` + +## Temporary Stack Architecture (Future) + +When BotServer implements `--temp-stack`, E2E tests will run in isolated environments: + +```bash +botserver --temp-stack +# Creates: /tmp/botserver-test-{timestamp}-{random}/ +# With isolated: PostgreSQL, Redis, MinIO, Mock LLM +# Auto-cleanup after test completion +``` + +Benefits: +- ✓ Isolation - Each test in separate environment +- ✓ Reproducibility - Consistent setup every time +- ✓ Automation - No manual configuration +- ✓ Safety - Won't interfere with development +- ✓ Cleanup - Automatic resource management +- ✓ Parallel - Multiple tests simultaneously +- ✓ CI/CD Ready - Perfect for automated pipelines + +## Writing Custom E2E Tests + +Create new test files in `gb/bottest/tests/e2e/`: + +```rust +#[tokio::test] +async fn test_my_feature() { + // Setup context + let ctx = E2ETestContext::setup_with_browser().await?; + let browser = ctx.browser.as_ref().unwrap(); + + // Navigate to feature + browser.navigate(&format!("{}/my-feature", ctx.base_url())).await?; + + // Interact with UI + browser.click("button.action").await?; + browser.wait_for_element(".result", Duration::from_secs(10)).await?; + + // Verify results + let text = browser.get_text(".result").await?; + assert_eq!(text, "Expected result"); + + // Cleanup + ctx.close().await; +} +``` + +Register in `tests/e2e/mod.rs`: + +```rust +mod my_feature; +``` + +## Best Practices + +1. **Keep tests focused** - Test one user workflow per test +2. **Use meaningful names** - `test_complete_platform_flow` not `test_1` +3. **Explicit waits** - Use `wait_for_element` instead of `sleep` +4. **Test realistic flows** - Use actual test credentials +5. **Verify results explicitly** - Check status codes, UI elements, and state +6. **Clean up properly** - Always call `ctx.close().await` +7. **Handle errors gracefully** - Use `?` operator for error propagation +8. **Make tests independent** - Don't rely on test execution order + +## Test Success Criteria + +✓ Platform fully loads without errors +✓ BotServer starts and becomes ready +✓ User can login with credentials +✓ Chat messages are sent and responses received +✓ User can logout and session is invalidated +✓ Protected routes block unauthenticated access +✓ Tests run consistently multiple times +✓ Tests complete within acceptable time (~60 seconds) + +## See Also + +- [Testing Overview](./README.md) - Testing strategy and structure +- [Performance Testing](./performance.md) - Benchmarks and load tests +- [Test Architecture](./architecture.md) - Design patterns and best practices +- [Integration Testing](./integration.md) - Multi-component testing \ No newline at end of file diff --git a/src/SUMMARY.md b/src/SUMMARY.md index a17e9147..89db9654 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -347,6 +347,15 @@ - [Automation Migration](./14-migration/automation.md) - [Validation and Testing](./14-migration/validation.md) +# Part XIV - Testing + +- [Chapter 17: Testing](./17-testing/README.md) + - [End-to-End Testing](./17-testing/e2e-testing.md) + - [Testing Architecture](./17-testing/architecture.md) + - [Performance Testing](./17-testing/performance.md) + - [Best Practices](./17-testing/best-practices.md) + - [CI/CD Integration](./17-testing/ci-cd.md) + # Appendices - [Appendix A: Database Model](./15-appendix/README.md)