Job-List/REDIS_IMPLEMENTATION.md

# Redis/Valkey Caching & Search Implementation

## Overview

This implementation adds Redis/Valkey caching and search functionality to the Job-List application, following the same patterns used in Job-Info-Prod.

## Features Implemented

### 1. Redis/Valkey Connection Module (`redis.ts`)

**Connection Management:**
- Automatic connection to Redis/Valkey using `REDIS_URL` environment variable
- Defaults to `redis://localhost:6379` if not configured
- Connection pooling and automatic reconnection on failures
- Error handling and logging

**Cache Operations:**
- `getCached<T>(key)` - Retrieve cached data by key
- `setCached(key, data, options)` - Store data with TTL (default 5 minutes)
- `deleteCached(keyOrPattern)` - Delete cache by key or pattern (supports wildcards)
- `invalidateSearchCache()` - Clear all search results cache

**Search Operations:**
- `cacheSearchResults(query, results, options)` - Cache search results
- `getCachedSearchResults(query)` - Retrieve cached search results

### 2. API Endpoints (`server.ts`)

#### GET `/api/jobs`
**Purpose:** Fetch all job records with Redis caching

**Headers:**
- `X-PocketBase-Token` - User authentication token (required)

**Response:**
```json
{
  "success": true,
  "data": [...],
  "cached": true/false
}
```

**Caching:**
- Cache key: `jobs:all:Job_Info_TestEnv`
- TTL: 5 minutes (300 seconds)
- Automatically fetches from PocketBase if cache miss

#### GET `/api/jobs/search?q=<query>`
**Purpose:** Search jobs across multiple fields with caching

**Headers:**
- `X-PocketBase-Token` - User authentication token (required)

**Query Parameters:**
- `q` - Search query (required)

**Searchable Fields:**
- Job_Number
- Job_Full_Name
- Job_Name
- Company_Client
- Contact_Person
- Estimator
- Job_Status
- Project_Manager
- Notes

**Response:**
```json
{
  "success": true,
  "data": [...],
  "query": "search term",
  "cached": true/false
}
```

**Caching:**
- Cache key: `search:<lowercase_query>`
- TTL: 3 minutes (180 seconds)
- Case-insensitive query matching

#### POST `/api/cache/clear`
**Purpose:** Manually clear all caches (admin/debugging)

**Headers:**
- `X-PocketBase-Token` - User authentication token (required)

**Response:**
```json
{
  "success": true,
  "message": "Cache cleared successfully"
}
```

### 3. Cache Invalidation

**Automatic Cache Clearing:**
- When data is submitted via `/api/submit`, all job caches are automatically cleared
- Ensures cache consistency with PocketBase data
- Clears both job data cache and search result caches

**Manual Cache Clearing:**
- Use `/api/cache/clear` endpoint for manual cache invalidation

## Configuration

### Environment Variables

Add to your `.env` file (or secrets location):

```bash
# Redis/Valkey Configuration
REDIS_URL=redis://localhost:6379

# For production, connect to the same instance as Job-Info-Prod
# REDIS_URL=redis://redis-server:6379
# With authentication: REDIS_URL=redis://:password@redis-server:6379
```

### Dependencies Added

```json
{
  "dependencies": {
    "ioredis": "^5.8.2"
  },
  "devDependencies": {
    "@types/ioredis": "^5.0.0"
  }
}
```

## Architecture

```
┌──────────────┐
│   Frontend   │
│ (HTML/JS)    │
└──────┬───────┘
       │
       │ HTTP Requests
       │ X-PocketBase-Token header
       │
┌──────▼────────────────────────────────┐
│          Server (server.ts)           │
│  ┌────────────────────────────────┐   │
│  │  1. Check Redis Cache          │   │
│  │     ↓ (if miss)                │   │
│  │  2. Fetch from PocketBase      │   │
│  │     ↓                           │   │
│  │  3. Store in Redis Cache       │   │
│  │     ↓                           │   │
│  │  4. Return to client           │   │
│  └────────────────────────────────┘   │
└───────────┬───────────────────────────┘
            │
            ├─────────────┬──────────────┐
            │             │              │
    ┌───────▼──────┐  ┌──▼──────────┐  │
    │    Redis/    │  │  PocketBase │  │
    │   Valkey     │  │  Database   │  │
    │  (Cache)     │  │  (Source)   │  │
    └──────────────┘  └─────────────┘  │
                                        │
                      Same Redis instance
                      as Job-Info-Prod
```

## Cache Strategy

### Read Path
1. Client requests data via `/api/jobs` or `/api/jobs/search`
2. Server checks Redis cache first
3. On cache hit: Return cached data immediately
4. On cache miss: Fetch from PocketBase, cache it, then return

### Write Path
1. Client submits update via `/api/submit`
2. Server updates PocketBase directly
3. Server invalidates all related caches
4. Next read will fetch fresh data from PocketBase and cache it

### Benefits
- **Reduced Load:** Minimizes PocketBase queries
- **Faster Response:** Cached responses are ~10-100x faster
- **Scalability:** Multiple instances can share the same cache
- **Consistency:** Cache invalidation ensures data freshness

## Future Migration to Job-Info-Prod

The current implementation:
- ✅ Uses Redis/Valkey for caching and search
- ✅ Keeps PocketBase as source of truth for writes
- ✅ Can connect to the same Redis instance as Job-Info-Prod

Future migration path:
1. Connect to the same Redis instance used by Job-Info-Prod
2. Implement Redis PubSub or webhooks for real-time cache invalidation
3. Gradually transition writes to Job-Info-Prod backend
4. Eventually use Job-Info-Prod as primary data source

## Testing

### Test Cache Functionality

1. **Test cached reads:**
```bash
# First request (cache miss)
curl -H "X-PocketBase-Token: YOUR_TOKEN" http://localhost:3025/api/jobs

# Second request within 5 minutes (cache hit)
curl -H "X-PocketBase-Token: YOUR_TOKEN" http://localhost:3025/api/jobs
```

2. **Test search:**
```bash
# Search for jobs
curl -H "X-PocketBase-Token: YOUR_TOKEN" "http://localhost:3025/api/jobs/search?q=Estimating"
```

3. **Test cache invalidation:**
```bash
# Clear cache
curl -X POST -H "X-PocketBase-Token: YOUR_TOKEN" http://localhost:3025/api/cache/clear
```

### Verify Redis Connection

```bash
# Check if Redis is running
redis-cli ping
# Should return: PONG

# Monitor Redis operations
redis-cli monitor
# Then make API requests to see cache operations
```

## Monitoring

Redis operations are logged to console:
- "Jobs fetched from cache" - Data served from cache
- "Jobs fetched from PocketBase and cached" - Cache miss, data fetched and cached
- "Search results fetched from cache" - Search results from cache
- "Cache cleared successfully" - Manual cache clear

## Troubleshooting

### Redis Connection Issues

If Redis connection fails:
1. Check `REDIS_URL` environment variable
2. Verify Redis/Valkey is running: `redis-cli ping`
3. Check network connectivity to Redis server
4. Review server logs for connection errors

### Cache Not Working

If cache doesn't seem to work:
1. Verify Redis is accessible
2. Check server logs for cache errors
3. Try manually clearing cache: `POST /api/cache/clear`
4. Verify TTL values are appropriate for your use case

### Stale Data

If seeing stale data:
1. Cache invalidation should happen automatically on updates
2. Manually clear cache if needed
3. Consider reducing TTL values for more frequent updates
4. Check if updates are going through `/api/submit` endpoint