Job-List/REDIS_IMPLEMENTATION.md

Redis/Valkey Caching & Search Implementation

Overview

This implementation adds Redis/Valkey caching and search functionality to the Job-List application, following the same patterns used in Job-Info-Prod.

Features Implemented

1. Redis/Valkey Connection Module (redis.ts)

Connection Management:

• Automatic connection to Redis/Valkey using REDIS_URL environment variable
• Defaults to redis://localhost:6379 if not configured
• Connection pooling and automatic reconnection on failures
• Error handling and logging

Cache Operations:

• getCached<T>(key) - Retrieve cached data by key
• setCached(key, data, options) - Store data with TTL (default 5 minutes)
• deleteCached(keyOrPattern) - Delete cache by key or pattern (supports wildcards)
• invalidateSearchCache() - Clear all search results cache

Search Operations:

• cacheSearchResults(query, results, options) - Cache search results
• getCachedSearchResults(query) - Retrieve cached search results

2. API Endpoints (server.ts)

GET /api/jobs

Purpose: Fetch all job records with Redis caching

Headers:

• X-PocketBase-Token - User authentication token (required)

Response:

{
  "success": true,
  "data": [...],
  "cached": true/false
}

Caching:

• Cache key: jobs:all:Job_Info_TestEnv
• TTL: 5 minutes (300 seconds)
• Automatically fetches from PocketBase if cache miss

GET /api/jobs/search?q=<query>

Purpose: Search jobs across multiple fields with caching

Headers:

• X-PocketBase-Token - User authentication token (required)

Query Parameters:

• q - Search query (required)

Searchable Fields:

• Job_Number
• Job_Full_Name
• Job_Name
• Company_Client
• Contact_Person
• Estimator
• Job_Status
• Project_Manager
• Notes

Response:

{
  "success": true,
  "data": [...],
  "query": "search term",
  "cached": true/false
}

Caching:

• Cache key: search:<lowercase_query>
• TTL: 3 minutes (180 seconds)
• Case-insensitive query matching

POST /api/cache/clear

Purpose: Manually clear all caches (admin/debugging)

Headers:

• X-PocketBase-Token - User authentication token (required)

Response:

{
  "success": true,
  "message": "Cache cleared successfully"
}

3. Cache Invalidation

Automatic Cache Clearing:

• When data is submitted via /api/submit, all job caches are automatically cleared
• Ensures cache consistency with PocketBase data
• Clears both job data cache and search result caches

Manual Cache Clearing:

• Use /api/cache/clear endpoint for manual cache invalidation

Configuration

Environment Variables

Add to your .env file (or secrets location):

# Redis/Valkey Configuration
REDIS_URL=redis://localhost:6379

# For production, connect to the same instance as Job-Info-Prod
# REDIS_URL=redis://redis-server:6379
# With authentication: REDIS_URL=redis://:password@redis-server:6379

Dependencies Added

{
  "dependencies": {
    "ioredis": "^5.8.2"
  },
  "devDependencies": {
    "@types/ioredis": "^5.0.0"
  }
}

Architecture

┌──────────────┐
│   Frontend   │
│ (HTML/JS)    │
└──────┬───────┘
       │
       │ HTTP Requests
       │ X-PocketBase-Token header
       │
┌──────▼────────────────────────────────┐
│          Server (server.ts)           │
│  ┌────────────────────────────────┐   │
│  │  1. Check Redis Cache          │   │
│  │     ↓ (if miss)                │   │
│  │  2. Fetch from PocketBase      │   │
│  │     ↓                           │   │
│  │  3. Store in Redis Cache       │   │
│  │     ↓                           │   │
│  │  4. Return to client           │   │
│  └────────────────────────────────┘   │
└───────────┬───────────────────────────┘
            │
            ├─────────────┬──────────────┐
            │             │              │
    ┌───────▼──────┐  ┌──▼──────────┐  │
    │    Redis/    │  │  PocketBase │  │
    │   Valkey     │  │  Database   │  │
    │  (Cache)     │  │  (Source)   │  │
    └──────────────┘  └─────────────┘  │
                                        │
                      Same Redis instance
                      as Job-Info-Prod

Cache Strategy

Read Path

1. Client requests data via /api/jobs or /api/jobs/search
2. Server checks Redis cache first
3. On cache hit: Return cached data immediately
4. On cache miss: Fetch from PocketBase, cache it, then return

Write Path

1. Client submits update via /api/submit
2. Server updates PocketBase directly
3. Server invalidates all related caches
4. Next read will fetch fresh data from PocketBase and cache it

Benefits

• Reduced Load: Minimizes PocketBase queries
• Faster Response: Cached responses are ~10-100x faster
• Scalability: Multiple instances can share the same cache
• Consistency: Cache invalidation ensures data freshness

Future Migration to Job-Info-Prod

The current implementation:

• ✅ Uses Redis/Valkey for caching and search
• ✅ Keeps PocketBase as source of truth for writes
• ✅ Can connect to the same Redis instance as Job-Info-Prod

Future migration path:

1. Connect to the same Redis instance used by Job-Info-Prod
2. Implement Redis PubSub or webhooks for real-time cache invalidation
3. Gradually transition writes to Job-Info-Prod backend
4. Eventually use Job-Info-Prod as primary data source

Testing

Test Cache Functionality

1. Test cached reads:

# First request (cache miss)
curl -H "X-PocketBase-Token: YOUR_TOKEN" http://localhost:3025/api/jobs

# Second request within 5 minutes (cache hit)
curl -H "X-PocketBase-Token: YOUR_TOKEN" http://localhost:3025/api/jobs

2. Test search:

# Search for jobs
curl -H "X-PocketBase-Token: YOUR_TOKEN" "http://localhost:3025/api/jobs/search?q=Estimating"

3. Test cache invalidation:

# Clear cache
curl -X POST -H "X-PocketBase-Token: YOUR_TOKEN" http://localhost:3025/api/cache/clear

Verify Redis Connection

# Check if Redis is running
redis-cli ping
# Should return: PONG

# Monitor Redis operations
redis-cli monitor
# Then make API requests to see cache operations

Monitoring

Redis operations are logged to console:

• "Jobs fetched from cache" - Data served from cache
• "Jobs fetched from PocketBase and cached" - Cache miss, data fetched and cached
• "Search results fetched from cache" - Search results from cache
• "Cache cleared successfully" - Manual cache clear

Troubleshooting

Redis Connection Issues

If Redis connection fails:

1. Check REDIS_URL environment variable
2. Verify Redis/Valkey is running: redis-cli ping
3. Check network connectivity to Redis server
4. Review server logs for connection errors

Cache Not Working

If cache doesn't seem to work:

1. Verify Redis is accessible
2. Check server logs for cache errors
3. Try manually clearing cache: POST /api/cache/clear
4. Verify TTL values are appropriate for your use case

Stale Data

If seeing stale data:

1. Cache invalidation should happen automatically on updates
2. Manually clear cache if needed
3. Consider reducing TTL values for more frequent updates
4. Check if updates are going through /api/submit endpoint