gb/fullsheet.md

FullSheet - Enterprise Grade Online Spreadsheet

Vision

Transform the current sheet.js into a full-featured Excel competitor with:

• Virtual scrolling for million+ row datasets
• xlsx import/export via SheetJS (xlsx library)
• Real-time collaboration with operational transformation
• Enterprise features: audit trail, permissions, version history

---

Phase 1: Virtual Scrolling Engine (Priority: HIGH)

Current Problem

• 26×100 = 2600 DOM elements rendered simultaneously
• No support for large datasets (Excel supports 1M+ rows)
• Poor performance with 10K+ cells

Solution Architecture

// NEW: VirtualGrid class - drop-in replacement for current grid
class VirtualGrid {
  constructor(container, config) {
    this.config = {
      colCount: 16384,  // Excel's max columns (XFD)
      rowCount: 1048576, // Excel's max rows
      colWidth: 100,
      rowHeight: 24,
      bufferSize: 20,    // Render 20 extra rows/cols outside viewport
      ...config
    };
    
    this.visibleStartRow = 0;
    this.visibleEndRow = 0;
    this.visibleStartCol = 0;
    this.visibleEndCol = 0;
    
    this.cellCache = new Map();     // Only stores non-empty cells
    this.renderedCells = new Map();  // Currently in DOM
    
    this.initialize();
  }
  
  initialize() {
    // Use CSS transform for virtual scrolling
    this.viewport = document.createElement('div');
    this.viewport.className = 'virtual-viewport';
    this.viewport.style.overflow = 'auto';
    this.viewport.style.position = 'relative';
    
    this.content = document.createElement('div');
    this.content.className = 'virtual-content';
    this.content.style.position = 'absolute';
    this.content.style.top = '0';
    this.content.style.left = '0';
    
    this.viewport.appendChild(this.content);
    this.container.appendChild(this.viewport);
    
    // Calculate dimensions
    this.content.style.width = `${this.config.colCount * this.config.colWidth}px`;
    this.content.style.height = `${this.config.rowCount * this.config.rowHeight}px`;
    
    // Bind scroll events
    this.viewport.addEventListener('scroll', () => this.onScroll());
    this.resizeObserver = new ResizeObserver(() => this.onResize());
    this.resizeObserver.observe(this.viewport);
    
    // Initial render
    this.onScroll();
  }
  
  onScroll() {
    const scrollTop = this.viewport.scrollTop;
    const scrollLeft = this.viewport.scrollLeft;
    const viewHeight = this.viewport.clientHeight;
    const viewWidth = this.viewport.clientWidth;
    
    // Calculate visible range
    this.visibleStartRow = Math.floor(scrollTop / this.config.rowHeight);
    this.visibleEndRow = Math.min(
      this.config.rowCount - 1,
      Math.ceil((scrollTop + viewHeight) / this.config.rowHeight)
    );
    
    this.visibleStartCol = Math.floor(scrollLeft / this.config.colWidth);
    this.visibleEndCol = Math.min(
      this.config.colCount - 1,
      Math.ceil((scrollLeft + viewWidth) / this.config.colWidth)
    );
    
    // Apply buffer
    const buffer = this.config.bufferSize;
    this.visibleStartRow = Math.max(0, this.visibleStartRow - buffer);
    this.visibleEndRow = Math.min(this.config.rowCount - 1, this.visibleEndRow + buffer);
    this.visibleStartCol = Math.max(0, this.visibleStartCol - buffer);
    this.visibleEndCol = Math.min(this.config.colCount - 1, this.visibleEndCol + buffer);
    
    this.renderVisibleCells();
  }
  
  renderVisibleCells() {
    // Remove cells no longer visible
    for (const [key, el] of this.renderedCells) {
      const [r, c] = key.split(',').map(Number);
      if (r < this.visibleStartRow || r > this.visibleEndRow ||
          c < this.visibleStartCol || c > this.visibleEndCol) {
        el.remove();
        this.renderedCells.delete(key);
      }
    }
    
    // Render visible cells
    for (let row = this.visibleStartRow; row <= this.visibleEndRow; row++) {
      for (let col = this.visibleStartCol; col <= this.visibleEndCol; col++) {
        const key = `${row},${col}`;
        if (!this.renderedCells.has(key)) {
          const cellData = this.cellCache.get(key);
          if (cellData) {
            const cell = this.createCellElement(row, col, cellData);
            cell.style.position = 'absolute';
            cell.style.top = `${row * this.config.rowHeight}px`;
            cell.style.left = `${col * this.config.colWidth}px`;
            cell.style.width = `${this.config.colWidth}px`;
            cell.style.height = `${this.config.rowHeight}px`;
            this.content.appendChild(cell);
            this.renderedCells.set(key, cell);
          }
        }
      }
    }
  }
  
  setCellValue(row, col, value) {
    const key = `${row},${col}`;
    if (value === null || value === undefined || value === '') {
      this.cellCache.delete(key);
    } else {
      this.cellCache.set(key, value);
    }
    
    // Update if in visible range
    if (row >= this.visibleStartRow && row <= this.visibleEndRow &&
        col >= this.visibleStartCol && col <= this.visibleEndCol) {
      const cell = this.renderedCells.get(key);
      if (cell) {
        if (value) {
          cell.textContent = value;
        } else {
          cell.remove();
          this.renderedCells.delete(key);
        }
      } else if (value) {
        const el = this.createCellElement(row, col, value);
        el.style.position = 'absolute';
        el.style.top = `${row * this.config.rowHeight}px`;
        el.style.left = `${col * this.config.colWidth}px`;
        this.content.appendChild(el);
        this.renderedCells.set(key, el);
      }
    }
  }
  
  getCellValue(row, col) {
    return this.cellCache.get(`${row},${col}`);
  }
  
  scrollToCell(row, col) {
    const scrollTop = row * this.config.rowHeight;
    const scrollLeft = col * this.config.colWidth;
    this.viewport.scrollTo(scrollLeft, scrollTop);
  }
  
  getViewportPosition() {
    return {
      top: this.viewport.scrollTop,
      left: this.viewport.scrollLeft,
      height: this.viewport.clientHeight,
      width: this.viewport.clientWidth
    };
  }
}

Integration Steps

1. Add VirtualGrid to sheet.js (keep existing functionality):

   // At top of sheet.js, after CONFIG
   let virtualGrid = null;
   
   function initVirtualGrid() {
     const container = document.getElementById('cellsContainer');
     virtualGrid = new VirtualGrid(container, {
       colCount: CONFIG.COLS,
       rowCount: CONFIG.ROWS,
       colWidth: CONFIG.COL_WIDTH,
       rowHeight: CONFIG.ROW_HEIGHT
     });
   
     // Copy existing data to virtual grid
     const ws = state.worksheets[state.activeWorksheet];
     for (const [key, data] of Object.entries(ws.data)) {
       const [row, col] = key.split(',').map(Number);
       virtualGrid.setCellValue(row, col, data.value || data.formula);
     }
   }
   

2. Modify renderGrid() to use virtual grid:

   function renderGrid() {
     // Keep header rendering (not virtualized)
     renderColumnHeaders();
     renderRowHeaders();
   
     // Initialize virtual grid for cells
     if (!virtualGrid) {
       initVirtualGrid();
     }
   }
   

---

Phase 2: xlsx Import/Export (Priority: HIGH)

Library Integration

Add SheetJS library to botui. Place in: botui/ui/suite/js/vendor/xlsx.full.min.js

File Import

// Add to sheet.js
async function importXlsx(file) {
  return new Promise((resolve, reject) => {
    const reader = new FileReader();
    reader.onload = (e) => {
      try {
        const data = new Uint8Array(e.target.result);
        const workbook = XLSX.read(data, { type: 'array' });
        
        // Convert to sheet format
        const worksheets = workbook.SheetNames.map((name, index) => {
          const sheet = workbook.Sheets[name];
          const jsonData = XLSX.utils.sheet_to_json(sheet, { 
            header: 1,  // Array of arrays
            defval: '' 
          });
          
          // Convert to sparse cell format
          const cellData = {};
          jsonData.forEach((row, rowIndex) => {
            row.forEach((value, colIndex) => {
              if (value !== '' && value !== null && value !== undefined) {
                cellData[`${rowIndex},${colIndex}`] = { value: String(value) };
              }
            });
          });
          
          return { name, data: cellData };
        });
        
        state.worksheets = worksheets;
        state.activeWorksheet = 0;
        renderWorksheetTabs();
        
        // Reinitialize virtual grid with new data
        initVirtualGrid();
        
        resolve();
      } catch (err) {
        reject(err);
      }
    };
    reader.onerror = reject;
    reader.readAsArrayBuffer(file);
  });
}

File Export

async function exportXlsx() {
  // Convert sheet format to array of arrays
  const ws = state.worksheets[state.activeWorksheet];
  const maxRow = findMaxRow(ws.data);
  const maxCol = findMaxCol(ws.data);
  
  const jsonData = [];
  for (let r = 0; r <= maxRow; r++) {
    const row = [];
    for (let c = 0; c <= maxCol; c++) {
      const key = `${r},${c}`;
      const cell = ws.data[key];
      row.push(cell?.value || cell?.formula || '');
    }
    jsonData.push(row);
  }
  
  // Create workbook
  const worksheet = XLSX.utils.aoa_to_sheet(jsonData);
  const workbook = XLSX.utils.book_new();
  XLSX.utils.book_append_sheet(workbook, worksheet, state.sheetName);
  
  // Generate file
  const xlsxData = XLSX.write(workbook, { 
    bookType: 'xlsx', 
    type: 'array' 
  });
  
  // Download
  const blob = new Blob([xlsxData], { 
    type: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' 
  });
  const url = URL.createObjectURL(blob);
  const a = document.createElement('a');
  a.href = url;
  a.download = `${state.sheetName}.xlsx`;
  a.click();
  URL.revokeObjectURL(url);
}

UI Integration

Add import/export buttons to toolbar in sheet.html:

<button class="btn-icon" id="importXlsxBtn" title="Import xlsx">
  <svg>...</svg>
</button>
<button class="btn-icon" id="exportXlsxBtn" title="Export xlsx">
  <svg>...</svg>
</button>

Add hidden file input:

<input type="file" id="xlsxFileInput" accept=".xlsx,.xls" style="display:none" />

Bind events:

document.getElementById('importXlsxBtn')?.addEventListener('click', () => {
  document.getElementById('xlsxFileInput').click();
});

document.getElementById('xlsxFileInput')?.addEventListener('change', async (e) => {
  if (e.target.files[0]) {
    await importXlsx(e.target.files[0]);
  }
});

document.getElementById('exportXlsxBtn')?.addEventListener('click', exportXlsx);

---

Phase 3: Real-time Collaboration (Priority: HIGH)

WebSocket Architecture

Current implementation has basic collaboration. Enhance with:

// Enhanced collaboration state
const collabState = {
  operations: [],        // Operation log
  revision: 0,          // Current revision number
  pendingOps: [],       // Ops not yet acknowledged
  cursorPositions: new Map(),  // Other users' cursors
  userColors: new Map()        // Assign colors to users
};

// Operational Transformation for concurrent edits
function transformOperation(op1, op2) {
  // Transform op1 against op2
  if (op1.type === 'set' && op2.type === 'set') {
    if (op1.row === op2.row && op1.col === op2.col) {
      // Same cell - use op2 (later operation wins)
      return null;
    }
    // Different cells - no transformation needed
    return op1;
  }
  // Handle insert/delete row/column
  // ... (OT logic)
}

function applyOperation(op) {
  const ws = state.worksheets[state.activeWorksheet];
  const key = `${op.row},${op.col}`;
  
  switch (op.type) {
    case 'set':
      if (op.value === null || op.value === '') {
        delete ws.data[key];
      } else {
        ws.data[key] = { value: op.value };
      }
      break;
    case 'delete':
      delete ws.data[key];
      break;
  }
  
  // Update virtual grid
  if (virtualGrid) {
    virtualGrid.setCellValue(op.row, op.col, op.value);
  }
}

Conflict Resolution

function resolveConflict(localOp, remoteOp) {
  // If same cell, remote wins (last-write-wins for simplicity)
  // For enterprise: use CRDT or OT
  if (localOp.row === remoteOp.row && localOp.col === remoteOp.col) {
    return remoteOp; // Remote takes precedence
  }
  
  // Different cells - apply both
  return localOp;
}

---

Phase 4: Enterprise Features (Priority: MEDIUM)

4.1 Audit Trail

class AuditLog {
  constructor() {
    this.entries = [];
    this.maxEntries = 10000;
  }
  
  log(action, details) {
    const entry = {
      timestamp: new Date().toISOString(),
      userId: getUserId(),
      action,
      details,
      sheetId: state.sheetId
    };
    
    this.entries.push(entry);
    if (this.entries.length > this.maxEntries) {
      this.entries.shift();
    }
    
    // Send to server for persistence
    this.persistEntry(entry);
  }
  
  async persistEntry(entry) {
    await fetch('/api/sheet/audit', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(entry)
    });
  }
  
  getHistory(startTime, endTime) {
    return this.entries.filter(e => {
      const time = new Date(e.timestamp);
      return time >= startTime && time <= endTime;
    });
  }
}

const auditLog = new AuditLog();

// Wrap cell operations with audit
const originalSetCellValue = setCellValue;
setCellValue = function(row, col, value) {
  const oldValue = getCellValue(row, col);
  originalSetCellValue(row, col, value);
  auditLog.log('cell_change', { row, col, oldValue, newValue: value });
};

4.2 Version History

class VersionManager {
  constructor() {
    this.versions = [];
    this.currentVersion = 0;
    this.autoSaveInterval = null;
  }
  
  createSnapshot() {
    const snapshot = {
      timestamp: new Date().toISOString(),
      worksheets: JSON.parse(JSON.stringify(state.worksheets)),
      sheetName: state.sheetName
    };
    
    this.versions.push(snapshot);
    this.currentVersion = this.versions.length - 1;
    
    // Persist to server
    this.persistVersion(snapshot);
    
    // Keep last 100 versions in memory
    if (this.versions.length > 100) {
      this.versions.shift();
    }
  }
  
  restoreVersion(versionIndex) {
    if (versionIndex < 0 || versionIndex >= this.versions.length) return;
    
    const version = this.versions[versionIndex];
    state.worksheets = JSON.parse(JSON.stringify(version.worksheets));
    state.sheetName = version.sheetName;
    state.activeWorksheet = 0;
    
    renderAllCells();
    renderWorksheetTabs();
    
    auditLog.log('version_restore', { versionIndex });
  }
  
  getVersionList() {
    return this.versions.map((v, i) => ({
      index: i,
      timestamp: v.timestamp,
      sheetName: v.sheetName
    }));
  }
}

4.3 Permissions System

class PermissionManager {
  constructor() {
    this.permissions = new Map(); // userId -> permission level
  }
  
  setPermission(userId, level) {
    this.permissions.set(userId, level);
  }
  
  canEdit() {
    const userId = getUserId();
    const level = this.permissions.get(userId);
    return level === 'edit' || level === 'admin';
  }
  
  canDelete() {
    const userId = getUserId();
    const level = this.permissions.get(userId);
    return level === 'admin';
  }
  
  canShare() {
    const userId = getUserId();
    const level = this.permissions.get(userId);
    return level === 'admin';
  }
}

const permissions = new PermissionManager();

// Check permissions before operations
function setCellValue(row, col, value) {
  if (!permissions.canEdit()) {
    showNotification('You do not have permission to edit', 'error');
    return;
  }
  // ... original implementation
}

---

Phase 5: Performance Optimization (Priority: MEDIUM)

5.1 Lazy Formula Evaluation

class FormulaEngine {
  constructor() {
    this.dependencyGraph = new Map(); // cell -> dependent cells
    this.evaluationQueue = new Set();
    this.isEvaluating = false;
  }
  
  scheduleEvaluation(cell) {
    this.evaluationQueue.add(cell);
    this.processQueue();
  }
  
  async processQueue() {
    if (this.isEvaluating) return;
    this.isEvaluating = true;
    
    while (this.evaluationQueue.size > 0) {
      const cell = this.evaluationQueue.values().next().value;
      this.evaluationQueue.delete(cell);
      await this.evaluateCell(cell);
    }
    
    this.isEvaluating = false;
  }
  
  async evaluateCell(cell) {
    const data = getCellData(cell.row, cell.col);
    if (!data?.formula) return;
    
    try {
      const result = evaluateFormula(data.formula, cell.row, cell.col);
      // Update display without full re-render
      if (virtualGrid) {
        virtualGrid.setCellValue(cell.row, cell.col, result);
      }
    } catch (e) {
      // Handle formula error
    }
  }
}

5.2 Web Worker for Heavy Calculations

// sheet.worker.js - separate worker file
self.onmessage = function(e) {
  const { type, data } = e.data;
  
  switch (type) {
    case 'evaluate_formula':
      const result = evaluateFormulaInWorker(data.formula, data.references);
      self.postMessage({ type: 'formula_result', result });
      break;
      
    case 'batch_render':
      const html = batchRenderCells(data.cells);
      self.postMessage({ type: 'batch_result', html });
      break;
  }
};

5.3 Debounced Auto-save

function debounce(func, wait) {
  let timeout;
  return function executedFunction(...args) {
    const later = () => {
      clearTimeout(timeout);
      func(...args);
    };
    clearTimeout(timeout);
    timeout = setTimeout(later, wait);
  };
}

const debouncedSave = debounce(saveSheet, 2000);

function scheduleAutoSave() {
  state.isDirty = true;
  debouncedSave();
}

---

Phase 6: Advanced Features (Priority: LOW)

6.1 Pivot Tables

function createPivotTable(sourceRange, rowFields, colFields, valueField, aggregation) {
  const data = extractRangeData(sourceRange);
  
  // Group by row fields
  const grouped = {};
  data.forEach(row => {
    const key = rowFields.map(f => row[f]).join('|');
    if (!grouped[key]) grouped[key] = [];
    grouped[key].push(row);
  });
  
  // Aggregate values
  const result = {};
  for (const [key, rows] of Object.entries(grouped)) {
    const values = rows.map(r => parseFloat(r[valueField])).filter(v => !isNaN(v));
    result[key] = aggregate(values, aggregation);
  }
  
  return result;
}

function aggregate(values, type) {
  switch (type) {
    case 'sum': return values.reduce((a, b) => a + b, 0);
    case 'avg': return values.reduce((a, b) => a + b, 0) / values.length;
    case 'count': return values.length;
    case 'min': return Math.min(...values);
    case 'max': return Math.max(...values);
    default: return 0;
  }
}

6.2 Macros/Scripts

class MacroEngine {
  constructor() {
    this.macros = new Map();
  }
  
  registerMacro(name, fn) {
    this.macros.set(name, fn);
  }
  
  runMacro(name) {
    const macro = this.macros.get(name);
    if (macro) {
      macro();
      auditLog.log('macro_run', { name });
    }
  }
}

// Built-in macros
const macroEngine = new MacroEngine();
macroEngine.registerMacro('clearAll', () => {
  state.worksheets[state.activeWorksheet].data = {};
  renderAllCells();
});

macroEngine.registerMacro('autoSum', () => {
  // Auto-sum selected range
});

---

Implementation Roadmap

Step 1: Virtual Scrolling (Week 1) ✅ COMPLETED

• Implement VirtualGrid class (sheet.js:253-448)
• Update renderGrid() to use virtual grid (sheet.js:378-430)
• Auto-enable for sheets > 500 rows
• Ensure existing functionality works (backward compatible)
• Test with 10K+ rows

Step 2: xlsx Support (Week 1-2) ✅ COMPLETED

• Use existing backend APIs (/api/sheet/import, /api/sheet/export)
• Implement importXlsx() (sheet.js:1893-1947)
• Implement exportXlsx() (sheet.js:1949-1992)
• Implement exportCsv() (sheet.js:1994-2021)
• Add UI buttons (sheet.html:209-225)
• Test with real xlsx files

Step 3: Collaboration (Week 2-3) 🔄 IN PROGRESS

• Existing WebSocket handler (connectWebSocket)
• Enhance with operation transformation
• Add cursor tracking for remote users
• Test multi-user scenarios

Step 4: Enterprise (Week 3-4) ✅ COMPLETED

• Add audit logging (AuditLog class, sheet.js:299-338)
• Implement version history (VersionManager class, sheet.js:340-418)
• Add permission system (PermissionManager class, sheet.js:420-452)
• Add admin UI (version history dialog)

Step 5: Performance (Week 4-5) 🔄 IN PROGRESS

• Add debounced save (CONFIG.AUTOSAVE_DELAY)
• Add formula lazy evaluation
• Implement Web Worker
• Profile and optimize

---

Backward Compatibility

All changes must maintain backward compatibility:

1. Keep existing CONFIG structure
2. Preserve current API (setCellValue, getCellValue, etc.)
3. VirtualGrid should wrap existing cell data
4. Feature flags for new functionality

// Feature detection
const features = {
  virtualScrolling: true,      // Default on for large sheets
  xlsxSupport: typeof XLSX !== 'undefined',
  collaboration: state.sheetId !== null,
  auditLog: true
};

// Use features to determine behavior
if (features.virtualScrolling && CONFIG.ROWS > 1000) {
  // Use virtual grid
} else {
  // Use traditional grid
}

---

Testing Strategy

Unit Tests

• VirtualGrid scroll behavior
• xlsx import/export round-trip
• Formula evaluation
• Permission checks

Integration Tests

• Multi-user collaboration
• Large file import (100K+ rows)
• Version restore
• Audit log persistence

Performance Tests

• Render time with 1M rows
• Memory usage
• Network bandwidth for collaboration

---

Files Modified

1. ✅ botui/ui/suite/sheet/sheet.js - VirtualGrid, importXlsx/exportXlsx, AuditLog, VersionManager, PermissionManager
2. ✅ botui/ui/suite/sheet/sheet.html - Import/export buttons
3. ⏳ botui/ui/suite/sheet/sheet.css - Virtual grid styles (needed)
4. ⏳ botserver/ - Use existing APIs (already have /api/sheet/import, /api/sheet/export, etc.)

---

Success Metrics

• Virtual scrolling implementation complete (auto-enable >500 rows)
• xlsx import/export via backend APIs
• Support 1M+ rows with smooth scrolling (needs testing)
• Import/export xlsx files up to 50MB (needs testing)
• 10+ concurrent users with sub-100ms sync (needs enhancement)
• <100ms initial load time (needs profiling)
• 0 compilation errors in botui (verified)
• 0 compilation errors in botserver (verifying) (verify)