v1.0.0-alpha4: Add Redis/Valkey caching and search with principal agent auth for global settings
This commit is contained in:
@@ -0,0 +1,274 @@
|
||||
# Redis/Valkey Caching & Search Implementation
|
||||
|
||||
## Overview
|
||||
|
||||
This implementation adds Redis/Valkey caching and search functionality to the Job-List application, following the same patterns used in Job-Info-Prod.
|
||||
|
||||
## Features Implemented
|
||||
|
||||
### 1. Redis/Valkey Connection Module (`redis.ts`)
|
||||
|
||||
**Connection Management:**
|
||||
- Automatic connection to Redis/Valkey using `REDIS_URL` environment variable
|
||||
- Defaults to `redis://localhost:6379` if not configured
|
||||
- Connection pooling and automatic reconnection on failures
|
||||
- Error handling and logging
|
||||
|
||||
**Cache Operations:**
|
||||
- `getCached<T>(key)` - Retrieve cached data by key
|
||||
- `setCached(key, data, options)` - Store data with TTL (default 5 minutes)
|
||||
- `deleteCached(keyOrPattern)` - Delete cache by key or pattern (supports wildcards)
|
||||
- `invalidateSearchCache()` - Clear all search results cache
|
||||
|
||||
**Search Operations:**
|
||||
- `cacheSearchResults(query, results, options)` - Cache search results
|
||||
- `getCachedSearchResults(query)` - Retrieve cached search results
|
||||
|
||||
### 2. API Endpoints (`server.ts`)
|
||||
|
||||
#### GET `/api/jobs`
|
||||
**Purpose:** Fetch all job records with Redis caching
|
||||
|
||||
**Headers:**
|
||||
- `X-PocketBase-Token` - User authentication token (required)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": [...],
|
||||
"cached": true/false
|
||||
}
|
||||
```
|
||||
|
||||
**Caching:**
|
||||
- Cache key: `jobs:all:Job_Info_TestEnv`
|
||||
- TTL: 5 minutes (300 seconds)
|
||||
- Automatically fetches from PocketBase if cache miss
|
||||
|
||||
#### GET `/api/jobs/search?q=<query>`
|
||||
**Purpose:** Search jobs across multiple fields with caching
|
||||
|
||||
**Headers:**
|
||||
- `X-PocketBase-Token` - User authentication token (required)
|
||||
|
||||
**Query Parameters:**
|
||||
- `q` - Search query (required)
|
||||
|
||||
**Searchable Fields:**
|
||||
- Job_Number
|
||||
- Job_Full_Name
|
||||
- Job_Name
|
||||
- Company_Client
|
||||
- Contact_Person
|
||||
- Estimator
|
||||
- Job_Status
|
||||
- Project_Manager
|
||||
- Notes
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": [...],
|
||||
"query": "search term",
|
||||
"cached": true/false
|
||||
}
|
||||
```
|
||||
|
||||
**Caching:**
|
||||
- Cache key: `search:<lowercase_query>`
|
||||
- TTL: 3 minutes (180 seconds)
|
||||
- Case-insensitive query matching
|
||||
|
||||
#### POST `/api/cache/clear`
|
||||
**Purpose:** Manually clear all caches (admin/debugging)
|
||||
|
||||
**Headers:**
|
||||
- `X-PocketBase-Token` - User authentication token (required)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Cache cleared successfully"
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Cache Invalidation
|
||||
|
||||
**Automatic Cache Clearing:**
|
||||
- When data is submitted via `/api/submit`, all job caches are automatically cleared
|
||||
- Ensures cache consistency with PocketBase data
|
||||
- Clears both job data cache and search result caches
|
||||
|
||||
**Manual Cache Clearing:**
|
||||
- Use `/api/cache/clear` endpoint for manual cache invalidation
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Add to your `.env` file (or secrets location):
|
||||
|
||||
```bash
|
||||
# Redis/Valkey Configuration
|
||||
REDIS_URL=redis://localhost:6379
|
||||
|
||||
# For production, connect to the same instance as Job-Info-Prod
|
||||
# REDIS_URL=redis://redis-server:6379
|
||||
# With authentication: REDIS_URL=redis://:password@redis-server:6379
|
||||
```
|
||||
|
||||
### Dependencies Added
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"ioredis": "^5.8.2"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/ioredis": "^5.0.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
│ Frontend │
|
||||
│ (HTML/JS) │
|
||||
└──────┬───────┘
|
||||
│
|
||||
│ HTTP Requests
|
||||
│ X-PocketBase-Token header
|
||||
│
|
||||
┌──────▼────────────────────────────────┐
|
||||
│ Server (server.ts) │
|
||||
│ ┌────────────────────────────────┐ │
|
||||
│ │ 1. Check Redis Cache │ │
|
||||
│ │ ↓ (if miss) │ │
|
||||
│ │ 2. Fetch from PocketBase │ │
|
||||
│ │ ↓ │ │
|
||||
│ │ 3. Store in Redis Cache │ │
|
||||
│ │ ↓ │ │
|
||||
│ │ 4. Return to client │ │
|
||||
│ └────────────────────────────────┘ │
|
||||
└───────────┬───────────────────────────┘
|
||||
│
|
||||
├─────────────┬──────────────┐
|
||||
│ │ │
|
||||
┌───────▼──────┐ ┌──▼──────────┐ │
|
||||
│ Redis/ │ │ PocketBase │ │
|
||||
│ Valkey │ │ Database │ │
|
||||
│ (Cache) │ │ (Source) │ │
|
||||
└──────────────┘ └─────────────┘ │
|
||||
│
|
||||
Same Redis instance
|
||||
as Job-Info-Prod
|
||||
```
|
||||
|
||||
## Cache Strategy
|
||||
|
||||
### Read Path
|
||||
1. Client requests data via `/api/jobs` or `/api/jobs/search`
|
||||
2. Server checks Redis cache first
|
||||
3. On cache hit: Return cached data immediately
|
||||
4. On cache miss: Fetch from PocketBase, cache it, then return
|
||||
|
||||
### Write Path
|
||||
1. Client submits update via `/api/submit`
|
||||
2. Server updates PocketBase directly
|
||||
3. Server invalidates all related caches
|
||||
4. Next read will fetch fresh data from PocketBase and cache it
|
||||
|
||||
### Benefits
|
||||
- **Reduced Load:** Minimizes PocketBase queries
|
||||
- **Faster Response:** Cached responses are ~10-100x faster
|
||||
- **Scalability:** Multiple instances can share the same cache
|
||||
- **Consistency:** Cache invalidation ensures data freshness
|
||||
|
||||
## Future Migration to Job-Info-Prod
|
||||
|
||||
The current implementation:
|
||||
- ✅ Uses Redis/Valkey for caching and search
|
||||
- ✅ Keeps PocketBase as source of truth for writes
|
||||
- ✅ Can connect to the same Redis instance as Job-Info-Prod
|
||||
|
||||
Future migration path:
|
||||
1. Connect to the same Redis instance used by Job-Info-Prod
|
||||
2. Implement Redis PubSub or webhooks for real-time cache invalidation
|
||||
3. Gradually transition writes to Job-Info-Prod backend
|
||||
4. Eventually use Job-Info-Prod as primary data source
|
||||
|
||||
## Testing
|
||||
|
||||
### Test Cache Functionality
|
||||
|
||||
1. **Test cached reads:**
|
||||
```bash
|
||||
# First request (cache miss)
|
||||
curl -H "X-PocketBase-Token: YOUR_TOKEN" http://localhost:3025/api/jobs
|
||||
|
||||
# Second request within 5 minutes (cache hit)
|
||||
curl -H "X-PocketBase-Token: YOUR_TOKEN" http://localhost:3025/api/jobs
|
||||
```
|
||||
|
||||
2. **Test search:**
|
||||
```bash
|
||||
# Search for jobs
|
||||
curl -H "X-PocketBase-Token: YOUR_TOKEN" "http://localhost:3025/api/jobs/search?q=Estimating"
|
||||
```
|
||||
|
||||
3. **Test cache invalidation:**
|
||||
```bash
|
||||
# Clear cache
|
||||
curl -X POST -H "X-PocketBase-Token: YOUR_TOKEN" http://localhost:3025/api/cache/clear
|
||||
```
|
||||
|
||||
### Verify Redis Connection
|
||||
|
||||
```bash
|
||||
# Check if Redis is running
|
||||
redis-cli ping
|
||||
# Should return: PONG
|
||||
|
||||
# Monitor Redis operations
|
||||
redis-cli monitor
|
||||
# Then make API requests to see cache operations
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
Redis operations are logged to console:
|
||||
- "Jobs fetched from cache" - Data served from cache
|
||||
- "Jobs fetched from PocketBase and cached" - Cache miss, data fetched and cached
|
||||
- "Search results fetched from cache" - Search results from cache
|
||||
- "Cache cleared successfully" - Manual cache clear
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Redis Connection Issues
|
||||
|
||||
If Redis connection fails:
|
||||
1. Check `REDIS_URL` environment variable
|
||||
2. Verify Redis/Valkey is running: `redis-cli ping`
|
||||
3. Check network connectivity to Redis server
|
||||
4. Review server logs for connection errors
|
||||
|
||||
### Cache Not Working
|
||||
|
||||
If cache doesn't seem to work:
|
||||
1. Verify Redis is accessible
|
||||
2. Check server logs for cache errors
|
||||
3. Try manually clearing cache: `POST /api/cache/clear`
|
||||
4. Verify TTL values are appropriate for your use case
|
||||
|
||||
### Stale Data
|
||||
|
||||
If seeing stale data:
|
||||
1. Cache invalidation should happen automatically on updates
|
||||
2. Manually clear cache if needed
|
||||
3. Consider reducing TTL values for more frequent updates
|
||||
4. Check if updates are going through `/api/submit` endpoint
|
||||
Reference in New Issue
Block a user