redis optimization/improvments

2026-01-09 04:08:06 +00:00
parent eaf5149d15
commit e4e3e65ea4
23 changed files with 8365 additions and 0 deletions
@@ -0,0 +1,523 @@
+# Job Info Data Handling Rules & Patterns
+
+**Version:** 1.0  
+**Status:** PRODUCTION  
+**Last Updated:** January 2026
+
+---
+
+## Overview
+
+This document enforces the data handling patterns, caching strategies, and view structure used in Job Info. All developers must follow these rules to maintain consistency, performance, and reliability.
+
+---
+
+## 1. Authentication & Token Rules (DUAL-TOKEN SYSTEM)
+
+### Dual Token Architecture
+
+The app uses **PocketBase OAuth through Microsoft** to obtain TWO tokens:
+
+1. **PocketBase Token** - For app operations (jobs, files, notes)
+   - Obtained via PocketBase OAuth
+   - Stored in `localStorage` under `pocketbase_auth`
+   - Expires: 3600 seconds (1 hour)
+
+2. **Microsoft Graph Token** - For SharePoint/Office 365 integration
+   - Obtained during PocketBase OAuth (Microsoft scopes)
+   - Stored in `localStorage` under `graph_token`
+   - Expires: 3600 seconds (1 hour)
+
+### Critical Requirement: NO User Re-authentication After Login
+
+**RULE:** Once user logs in, they should NEVER be asked to authenticate again (except system restart)
+
+**HOW:**
+- Both tokens obtained once during initial OAuth flow
+- Tokens stored in `localStorage` (frontend) and Valkey cache (backend)
+- Background refresh loop keeps tokens fresh (every 30 minutes)
+- If frontend token expires, backend provides fresh one
+- If both fail, use stale token as fallback
+- Only redirect to login if completely unavailable (system restart)
+
+### Token Storage
+
+**Frontend (localStorage):**
+```javascript
+pocketbase_auth        // PocketBase token + metadata
+graph_token            // Microsoft Graph token
+token_expires_pb       // PocketBase expiry timestamp
+token_expires_graph    // Graph expiry timestamp
+```
+
+**Backend (Valkey/Redis):**
+```
+tokens:${userId}:pb                    // Fresh PocketBase token (TTL: 1 hour)
+tokens:${userId}:graph                 // Fresh Graph token (TTL: 1 hour)
+tokens:${userId}:refresh_pb            // PocketBase refresh token (TTL: 30 days)
+tokens:${userId}:refresh_graph         // Graph refresh token (TTL: 30 days)
+```
+
+### Single Token Check Pattern
+
+Before ANY API call:
+
+```javascript
+// ✅ CORRECT: Check & refresh if needed
+async function ensureToken(type) {
+  // type: 'pb' or 'graph'
+  
+  // 1. Get from localStorage
+  const token = localStorage.getItem(`${type}_token`);
+  const expiry = localStorage.getItem(`token_expires_${type}`);
+  
+  // 2. If fresh (> 15 min remaining), use immediately
+  if (token && expiry > Date.now() + 15*60*1000) {
+    return token;
+  }
+  
+  // 3. If expiring soon, refresh from backend
+  const response = await fetch(`/api/refresh-token?type=${type}`);
+  if (response.ok) {
+    const { token: newToken, expiresAt } = await response.json();
+    localStorage.setItem(`${type}_token`, newToken);
+    localStorage.setItem(`token_expires_${type}`, expiresAt);
+    return newToken;
+  }
+  
+  // 4. Fallback: Use stale token if available (field workers!)
+  if (token) {
+    console.warn(`Using stale ${type} token - backend unreachable`);
+    return token;
+  }
+  
+  // 5. Only last resort: Redirect to login
+  redirectToLogin();
+  throw new Error('Authentication required');
+}
+
+// ❌ WRONG: Refreshing on every request
+const token = await pb.authRefresh(); // DON'T DO THIS
+fetch('/api/endpoint', { headers: { 'Authorization': `Bearer ${token}` } });
+```
+
+### Background Token Refresh (Backend)
+
+```typescript
+// Runs every 30 minutes for each logged-in user
+// Transparently refreshes tokens before they expire
+// Users never see a re-auth prompt
+
+async function refreshTokensInBackground(userId: string) {
+  // Check PocketBase token
+  const pbToken = await cache.get(`tokens:${userId}:pb`);
+  if (isExpiring(pbToken)) {
+    const refreshed = await refreshPocketBaseToken(pbToken.refresh_token);
+    await cache.set(`tokens:${userId}:pb`, refreshed, ttl: 3600);
+  }
+  
+  // Check Graph token
+  const graphToken = await cache.get(`tokens:${userId}:graph`);
+  if (isExpiring(graphToken)) {
+    const refreshed = await refreshMicrosoftGraphToken(graphToken.refresh_token);
+    await cache.set(`tokens:${userId}:graph`, refreshed, ttl: 3600);
+  }
+}
+```
+
+### Sign Out
+- Clear `localStorage` items: `pocketbase_auth`, `graph_token`, expiry times
+- Clear `sessionStorage`
+- Call backend `/api/auth/logout` to stop refresh loops
+- Revoke tokens with PocketBase & Microsoft (if available)
+- Redirect to signin page
+
+---
+
+## 2. Caching Strategy
+
+### Backend Cache (Redis/Valkey)
+
+**Rule:** Cache GET requests with deterministic keys, NOT mutations.
+
+#### Jobs List
+```typescript
+// Cache key format: jobs:page:${page}:perPage:${perPage}:sort:${sort}
+const cacheKey = `jobs:page:${page}:perPage:${perPage}:sort:${sort}`;
+const ttl = 300; // 5 minutes
+
+// Pattern:
+const cached = await getCache(cacheKey);
+if (cached) return c.json(cached);
+
+const data = await fetchFromPocketBase(...);
+await setCache(cacheKey, data, ttl);
+return c.json(data);
+```
+
+#### Cache TTL Guidelines
+| Data Type | TTL | Reason |
+|-----------|-----|--------|
+| Jobs List | 300s (5m) | Frequently searched, moderate change rate |
+| Job Details | 600s (10m) | Less frequently accessed |
+| File Lists | 900s (15m) | Rarely changes, expensive to fetch |
+| Metadata | 1800s (30m) | Static reference data |
+
+#### Cache Invalidation
+- Manual: `clearCachePattern('jobs:*')` after mutations
+- Automatic: TTL expiration
+- **Never cache:** POST, PUT, DELETE requests
+
+### Frontend Cache (In-Memory)
+
+**File Cache Pattern** (Current Implementation)
+```javascript
+const fileCache = new Map();
+
+// Check before fetching
+if (fileCache.has(folderLink)) {
+  const cached = fileCache.get(folderLink);
+  fileListState.items = cached.items;
+  renderFileGroups(folderLink);
+  return;
+}
+
+// Fetch and cache
+const data = await fetch(`/api/job-files?link=${link}`);
+fileCache.set(folderLink, { items: data.items, driveId: data.driveId });
+renderFileGroups(folderLink);
+
+// Clear on sign out
+fileCache.clear();
+```
+
+---
+
+## 3. Data Filtering & Transformation
+
+### File Filtering Rules
+
+**Supported File Types:**
+- PDFs (`.pdf`)
+- Word Documents (`.doc`, `.docx`)
+- Images (`.jpg`, `.jpeg`, `.png`, `.gif`, `.bmp`, `.tiff`, `.webp`)
+
+**Folders:** Excluded (never shown)
+
+**Implementation:**
+```javascript
+function renderFileGroups(folderLink) {
+  const term = (fileListState.filter || '').toLowerCase().trim();
+  
+  const filtered = fileListState.items.filter((f) => {
+    // Exclude folders
+    if (f.isFolder) return false;
+    
+    // Check file type
+    const name = f.name.toLowerCase();
+    const contentType = (f.contentType || '').toLowerCase();
+    const isPdfOrWord = name.endsWith('.pdf') || contentType.includes('pdf') ||
+                       name.endsWith('.doc') || name.endsWith('.docx') || 
+                       contentType.includes('word');
+    const isImage = ['jpg', 'jpeg', 'png', 'gif', 'bmp', 'tiff', 'webp']
+      .some(ext => name.endsWith(`.${ext}`)) || contentType.includes('image');
+    
+    if (!isPdfOrWord && !isImage) return false;
+    
+    // Apply search term filter
+    return name.includes(term);
+  });
+}
+```
+
+### File Categorization
+
+**Categories:** Manager Info → Contracts → Submittals → Plans → Other
+
+**Rules:**
+```javascript
+function categorize(filename) {
+  const name = filename.toLowerCase();
+  if (name.includes('manager')) return 'managerInfo';
+  if (name.includes('contract') || name.includes('estimate')) return 'contracts';
+  if (name.includes('submittal')) return 'submittals';
+  if (name.includes('plan') || name.includes('blueprint')) return 'plans';
+  return 'other';
+}
+```
+
+### Sorting Rules
+
+**Default Sort:** `-Job_Number` (descending)
+
+**Available Sorts:**
+```
+Job_Number, -Job_Number (job #)
+Job_Full_Name, -Job_Full_Name (alphabetical)
+Job_Start_Date, -Job_Start_Date (chronological)
+```
+
+**Backend Implementation:**
+```typescript
+const sort = c.req.query('sort') || '-Job_Number';
+const pbUrl = `...?sort=${encodeURIComponent(sort)}`;
+```
+
+---
+
+## 4. State Management
+
+### Frontend State Object
+```javascript
+// fileListState: Tracks current file view
+const fileListState = {
+  items: [],           // Array of file objects
+  filter: '',          // Current search term
+  jobNumber: '',       // Current job number
+  jobName: '',         // Current job name
+};
+
+// Do NOT create other state stores
+// All state flows through this single object
+```
+
+### Rules
+- Single source of truth per view
+- State modified only by explicit functions
+- No hidden/global state
+- Clear state on logout
+
+---
+
+## 5. API Response Format
+
+### Success Response
+```json
+{
+  "items": [...],      // Array of records
+  "total": 50,         // Total record count
+  "page": 1,
+  "perPage": 10,
+  "source": "cache|fetch"  // For debugging
+}
+```
+
+### Error Response
+```json
+{
+  "error": "User-friendly error message",
+  "status": 400
+}
+```
+
+### File List Response
+```json
+{
+  "items": [
+    {
+      "id": "...",
+      "name": "document.pdf",
+      "url": "https://...",
+      "driveId": "...",
+      "size": 1024,
+      "modified": "2026-01-01T00:00:00Z",
+      "contentType": "application/pdf",
+      "isFolder": false,
+      "path": "/folder/path"
+    }
+  ],
+  "total": 5,
+  "driveId": "...",
+  "source": "walk|search"
+}
+```
+
+---
+
+## 6. View Architecture (Industry Standard)
+
+### View Structure
+
+```
+AuthView
+  └── Login Form
+      └── Redirect to LandingView on success
+
+LandingView (Home)
+  ├── Header (User info, Sign Out)
+  ├── JobCardView
+  │   ├── Search Input
+  │   ├── Filters (Status, Date Range)
+  │   └── Job Cards (Grid)
+  │       └── Click → ManagerInfoView
+  └── Loading/Error States
+
+ManagerInfoView (Job Detail)
+  ├── Job Header (Job #, Name, Status)
+  ├── Job Info Panel (dates, manager, contacts)
+  ├── File List Section
+  │   ├── File Search
+  │   ├── File Groups (categorized)
+  │   └── Preview/Download Buttons
+  └── Notes Section
+      └── Click → NotesView
+
+NotesView (Expanded Notes)
+  ├── Sticky Notes Panel (read-only for now)
+  ├── Note Preview
+  └── Back to ManagerInfoView
+```
+
+---
+
+## 7. Naming Conventions
+
+### File/Folder Naming
+- `auth/` - Authentication module
+- `backend/` - Server code
+- `frontend/` - Client code
+- `docs/` - Documentation
+- `logs/` - Runtime logs (gitignored)
+- `tests/` - Test files
+
+### Variable Naming
+- `pb` - PocketBase instance
+- `authHeaders` - Auth token from PocketBase
+- `fileCache` - Frontend file cache (Map)
+- `fileListState` - Current file view state object
+- `cacheKey` - Redis key (format: `noun:filter:value`)
+
+### Class/Type Naming
+- `JobCard` - Individual job card component
+- `FileGroup` - Categorized file group
+- `FileItem` - Single file object
+- `AuthSession` - User session
+
+---
+
+## 8. Testing & Validation Rules
+
+### What to Test
+- Cache hit/miss behavior
+- File filtering (correct types shown)
+- Search functionality (partial matches work)
+- Categorization (files in correct group)
+- Sorting (correct order)
+- Auth session (token validity)
+
+### What NOT to Test
+- External API responses (mock them)
+- Redis connection (too fragile)
+- Third-party library behavior
+
+---
+
+## 9. Performance Rules
+
+### Frontend
+- Lazy load job files (don't fetch all at once)
+- Use file cache to avoid redundant fetches
+- Debounce search input (300ms)
+- Virtualize long lists (50+ items)
+
+### Backend
+- Cache GET requests always
+- Limit page size to 50 records max
+- Use pagination (default: 10 per page)
+- Timeout file searches after 30 seconds
+
+---
+
+## 10. Error Handling
+
+### Frontend
+```javascript
+try {
+  const resp = await fetch('/api/endpoint');
+  if (!resp.ok) throw new Error(`Server error: ${resp.status}`);
+  const data = await resp.json();
+  // Process data
+} catch (err) {
+  console.error('Operation failed:', err);
+  showErrorUI(err.message);
+}
+```
+
+### Backend
+```typescript
+try {
+  // Operation
+} catch (err) {
+  const message = (err as Error)?.message || String(err);
+  console.error('[endpoint] ERROR:', message);
+  logLine('logs/error.log', `error message`);
+  return c.json({ error: message }, 500);
+}
+```
+
+---
+
+## 11. Logging Rules
+
+### What to Log
+- Cache HIT/MISS (for debugging)
+- API request start/end
+- Errors with full stack trace
+- Authentication events (login, logout)
+
+### What NOT to Log
+- Tokens or secrets
+- User passwords
+- Personal data (except user ID)
+- Verbose debug output in production
+
+### Log Format
+```
+[endpoint] message
+[cache] Cache HIT for key
+[auth] User logged in: user123
+```
+
+---
+
+## 12. Version Control Rules
+
+### Commits
+- Small, atomic commits
+- Clear commit messages: `feat: add caching` or `fix: file filter bug`
+- One feature per commit
+
+### Branches
+- `main` - Production ready
+- `develop` - Integration branch
+- `feature/*` - New features
+- `fix/*` - Bug fixes
+
+### .gitignore
+```
+node_modules/
+.env
+.env.local
+logs/
+*.log
+dist/
+.bun/
+.DS_Store
+```
+
+---
+
+## Summary
+
+| Topic | Rule |
+|-------|------|
+| Authentication | Single PocketBase token, no refresh loops |
+| Caching | Redis for GET, TTL-based, cache key format: `noun:filters` |
+| Files | Only PDF, Word, Image types; categorize by filename |
+| State | Single state object per view |
+| Views | Auth → Landing → Detail → Notes |
+| Performance | Lazy load, paginate, debounce |
+| Errors | Catch, log, return JSON response |
+| Testing | Mock external APIs, test core logic |
+
+**All new code must follow these rules. Violations will be caught in code review.**