Job-Info/DEPLOYMENT_GUIDE.md

# Deployment Guide - Job Info Application

## Phase 9: Production Deployment

This guide covers preparing the Job Info application for production deployment including Docker configuration, environment setup, and CI/CD pipeline.

---

## 1. Deployment Architecture

### Production Stack
```
┌─────────────────────────────────────────┐
│        Client Browser (Edge)            │
│  - Svelte SPA (hash-based routing)      │
│  - Service Worker (offline + refresh)   │
│  - IndexedDB (client-side cache)        │
└─────────────────┬───────────────────────┘
                  │ HTTPS
                  ↓
┌─────────────────────────────────────────┐
│       CDN (CloudFlare/CloudFront)       │
│  - Cache static assets (CSS, JS, HTML)  │
│  - Serve from edge location             │
│  - 30-day cache for assets              │
└─────────────────┬───────────────────────┘
                  │ HTTPS
                  ↓
┌─────────────────────────────────────────┐
│   Load Balancer (ALB/NLB or Ingress)    │
│  - Route /api to backend                │
│  - Route static to CDN                  │
│  - Health checks every 30s              │
└─────────────────┬───────────────────────┘
                  │ HTTPS
          ┌───────┴──────────┐
          ↓                  ↓
    ┌──────────────┐  ┌──────────────┐
    │  Backend #1  │  │  Backend #2  │
    │  Hono Server │  │  Hono Server │
    │  Port 3005   │  │  Port 3005   │
    └──────┬───────┘  └──────┬───────┘
           │                 │
           └────────┬────────┘
                    ↓
        ┌──────────────────────┐
        │  PostgreSQL/Redis    │
        │  (Persistent Storage)│
        └──────────────────────┘
```

---

## 2. Docker Setup

### 2.1 Dockerfile - Backend

Create `Mode3Test/Dockerfile`:

```dockerfile
# Build stage
FROM oven/bun:latest as builder

WORKDIR /app

# Copy package files
COPY package.json bun.lockb ./

# Install dependencies
RUN bun install --production

# Copy source
COPY backend/ ./backend/
COPY .env ./

# Runtime stage
FROM oven/bun:latest

WORKDIR /app

# Copy from builder
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/backend ./backend
COPY --from=builder /app/.env ./

# Expose port
EXPOSE 3005

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
  CMD bun --eval "fetch('http://localhost:3005/health').then(r => r.status === 200 ? process.exit(0) : process.exit(1))"

# Start server
CMD ["bun", "run", "backend/server.ts"]
```

### 2.2 Dockerfile - Frontend

Create `frontend/Dockerfile`:

```dockerfile
# Build stage
FROM oven/bun:latest as builder

WORKDIR /app

# Copy package files
COPY package.json bun.lockb ./

# Install dependencies
RUN bun install

# Copy source
COPY . .

# Build
RUN bun run build

# Runtime stage - lightweight
FROM nginx:alpine

# Copy built assets
COPY --from=builder /app/dist /usr/share/nginx/html

# Copy nginx config
COPY nginx.conf /etc/nginx/conf.d/default.conf

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
  CMD wget --quiet --tries=1 --spider http://localhost:80/index.html || exit 1

# Start nginx
CMD ["nginx", "-g", "daemon off;"]
```

### 2.3 Nginx Configuration

Create `frontend/nginx.conf`:

```nginx
server {
    listen 80;
    server_name _;

    # Security headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'wasm-unsafe-eval'; style-src 'self' 'unsafe-inline'; connect-src 'self' https://graph.microsoft.com https://*.microsoft.com" always;

    # Compression
    gzip on;
    gzip_vary on;
    gzip_min_length 1000;
    gzip_types text/plain text/css text/xml text/javascript application/json application/javascript application/xml+rss;

    # Root location
    root /usr/share/nginx/html;

    # SPA routing - serve index.html for all non-asset requests
    location / {
        try_files $uri $uri/ /index.html;
        expires -1;
        add_header Cache-Control "public, must-revalidate, proxy-revalidate";
    }

    # Static assets with long cache
    location ~* \.(js|css|woff|woff2|eot|ttf|otf|svg|png|jpg|jpeg|gif|ico)$ {
        expires 30d;
        add_header Cache-Control "public, immutable";
    }

    # API proxy
    location /api/ {
        proxy_pass http://backend:3005;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # CORS headers
        add_header Access-Control-Allow-Origin "*" always;
        add_header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS" always;
        add_header Access-Control-Allow-Headers "Content-Type, Authorization" always;
        
        if ($request_method = 'OPTIONS') {
            return 204;
        }
    }

    # Health check endpoint
    location /health {
        access_log off;
        return 200 "healthy\n";
        add_header Content-Type text/plain;
    }
}
```

### 2.4 Docker Compose - Development

Create `docker-compose.yml`:

```yaml
version: '3.8'

services:
  # PostgreSQL Database
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: job_info_user
      POSTGRES_PASSWORD: ${DB_PASSWORD:-dev_password}
      POSTGRES_DB: job_info_db
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U job_info_user"]
      interval: 10s
      timeout: 5s
      retries: 5

  # Redis Cache
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

  # Backend API
  backend:
    build:
      context: .
      dockerfile: Mode3Test/Dockerfile
    environment:
      NODE_ENV: production
      PORT: 3005
      DATABASE_URL: postgresql://job_info_user:${DB_PASSWORD:-dev_password}@postgres:5432/job_info_db
      REDIS_URL: redis://redis:6379
      MICROSOFT_TENANT_ID: ${MICROSOFT_TENANT_ID}
      MICROSOFT_CLIENT_ID: ${MICROSOFT_CLIENT_ID}
      MICROSOFT_CLIENT_SECRET: ${MICROSOFT_CLIENT_SECRET}
      MICROSOFT_REDIRECT_URI: ${MICROSOFT_REDIRECT_URI:-http://localhost:3005/api/auth/callback}
    ports:
      - "3005:3005"
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3005/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  # Frontend Web Server
  frontend:
    build:
      context: .
      dockerfile: frontend/Dockerfile
    ports:
      - "80:80"
    depends_on:
      - backend
    environment:
      BACKEND_URL: http://backend:3005
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost/index.html"]
      interval: 30s
      timeout: 10s
      retries: 3

volumes:
  postgres_data:

networks:
  default:
    name: job-info-network
```

---

## 3. Environment Configuration

### 3.1 Production .env

Create `.env.production`:

```bash
# Application
NODE_ENV=production
PORT=3005

# Database (update with production credentials)
DATABASE_URL=postgresql://user:password@prod-db:5432/job_info_prod
REDIS_URL=redis://prod-redis:6379

# Microsoft OAuth (register app in Azure)
MICROSOFT_TENANT_ID=your-tenant-id
MICROSOFT_CLIENT_ID=your-client-id
MICROSOFT_CLIENT_SECRET=your-client-secret
MICROSOFT_REDIRECT_URI=https://app.example.com/api/auth/callback

# Security
JWT_SECRET=your-super-secret-key-min-32-chars
SESSION_SECRET=your-session-secret-key

# Frontend
VITE_API_URL=https://app.example.com/api
VITE_GRAPH_API_BASE=https://graph.microsoft.com/v1.0
VITE_OAUTH_CLIENT_ID=your-client-id
VITE_OAUTH_TENANT_ID=your-tenant-id
VITE_OAUTH_REDIRECT_URI=https://app.example.com/#/callback

# Logging
LOG_LEVEL=info
SENTRY_DSN=https://your-sentry-key@sentry.io/project-id

# Monitoring
DATADOG_API_KEY=your-datadog-key (optional)
```

### 3.2 Secrets Management

**Option A: AWS Secrets Manager**
```bash
# Store in AWS Secrets Manager
aws secretsmanager create-secret \
  --name job-info/prod \
  --secret-string '{
    "db_password": "...",
    "jwt_secret": "...",
    "microsoft_client_secret": "..."
  }'

# Reference in deployment
aws secretsmanager get-secret-value --secret-id job-info/prod
```

**Option B: Kubernetes Secrets**
```yaml
apiVersion: v1
kind: Secret
metadata:
  name: job-info-secrets
type: Opaque
stringData:
  database_password: "..."
  jwt_secret: "..."
  microsoft_client_secret: "..."
```

**Option C: Environment Variables**
```bash
# Set directly in container runtime
docker run \
  -e DATABASE_URL="postgresql://..." \
  -e JWT_SECRET="..." \
  job-info-backend:latest
```

---

## 4. Kubernetes Deployment

### 4.1 Deployment Manifests

Create `k8s/backend-deployment.yaml`:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: job-info-backend
  labels:
    app: job-info-backend
spec:
  replicas: 3  # Run 3 backend instances
  selector:
    matchLabels:
      app: job-info-backend
  template:
    metadata:
      labels:
        app: job-info-backend
    spec:
      containers:
      - name: backend
        image: your-registry/job-info-backend:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 3005
          name: http
        env:
        - name: NODE_ENV
          value: "production"
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: job-info-secrets
              key: database_url
        - name: MICROSOFT_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: job-info-secrets
              key: microsoft_client_secret
        resources:
          requests:
            memory: "256Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 3005
          initialDelaySeconds: 10
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /health
            port: 3005
          initialDelaySeconds: 5
          periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: job-info-backend-service
spec:
  selector:
    app: job-info-backend
  ports:
  - port: 3005
    targetPort: 3005
  type: ClusterIP
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: job-info-backend-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: job-info-backend
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
```

Create `k8s/frontend-deployment.yaml`:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: job-info-frontend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: job-info-frontend
  template:
    metadata:
      labels:
        app: job-info-frontend
    spec:
      containers:
      - name: frontend
        image: your-registry/job-info-frontend:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "128Mi"
            cpu: "50m"
          limits:
            memory: "256Mi"
            cpu: "250m"
        livenessProbe:
          httpGet:
            path: /index.html
            port: 80
          initialDelaySeconds: 10
          periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  name: job-info-frontend-service
spec:
  selector:
    app: job-info-frontend
  ports:
  - port: 80
    targetPort: 80
  type: LoadBalancer
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: job-info-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: job-info-backend-service
            port:
              number: 3005
      - path: /
        pathType: Prefix
        backend:
          service:
            name: job-info-frontend-service
            port:
              number: 80
```

---

## 5. CI/CD Pipeline

### 5.1 GitHub Actions Workflow

Create `.github/workflows/deploy.yml`:

```yaml
name: Deploy to Production

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    
    - name: Setup Bun
      uses: oven-sh/setup-bun@v1
    
    - name: Install dependencies
      run: |
        cd frontend && bun install
        cd ../Mode3Test && bun install
    
    - name: Run tests
      run: bun test
    
    - name: Lint
      run: |
        cd frontend && bun run lint || true
        cd ../Mode3Test && bun run lint || true

  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    
    - name: Setup Bun
      uses: oven-sh/setup-bun@v1
    
    - name: Build frontend
      run: |
        cd frontend
        bun install
        bun run build
    
    - name: Build Docker images
      run: |
        docker build -t job-info-backend:${{ github.sha }} -f Mode3Test/Dockerfile .
        docker build -t job-info-frontend:${{ github.sha }} -f frontend/Dockerfile .
    
    - name: Push to registry
      if: github.event_name == 'push' && github.ref == 'refs/heads/main'
      run: |
        docker tag job-info-backend:${{ github.sha }} your-registry/job-info-backend:latest
        docker tag job-info-frontend:${{ github.sha }} your-registry/job-info-frontend:latest
        
        docker push your-registry/job-info-backend:latest
        docker push your-registry/job-info-frontend:latest

  deploy:
    needs: build
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    
    - name: Deploy to Kubernetes
      run: |
        kubectl apply -f k8s/
        kubectl set image deployment/job-info-backend \
          job-info-backend=your-registry/job-info-backend:latest
        kubectl set image deployment/job-info-frontend \
          job-info-frontend=your-registry/job-info-frontend:latest
```

---

## 6. Monitoring & Observability

### 6.1 Logging

Setup structured logging:

```typescript
// backend/logging.ts
import pino from 'pino';

export const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  transport: {
    targets: [
      {
        target: 'pino/file',
        options: { destination: '/var/log/job-info.log' }
      },
      {
        target: 'pino-stackdriver',
        options: { projectId: 'your-gcp-project' }
      }
    ]
  }
});
```

### 6.2 Metrics

Setup Prometheus metrics:

```typescript
// backend/metrics.ts
import promClient from 'prom-client';

export const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
});

export const activeAuthSessions = new promClient.Gauge({
  name: 'active_auth_sessions',
  help: 'Number of active authentication sessions',
});
```

### 6.3 Error Tracking

Setup Sentry:

```typescript
// backend/sentry.ts
import * as Sentry from "@sentry/node";

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  environment: process.env.NODE_ENV,
  tracesSampleRate: 1.0,
});
```

---

## 7. Backup & Disaster Recovery

### 7.1 Database Backup

```bash
#!/bin/bash
# backup-postgres.sh

BACKUP_DIR="/backups/postgres"
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup
pg_dump -U job_info_user job_info_db | gzip > $BACKUP_DIR/backup_$DATE.sql.gz

# Upload to S3
aws s3 cp $BACKUP_DIR/backup_$DATE.sql.gz s3://job-info-backups/

# Keep only last 30 days
find $BACKUP_DIR -mtime +30 -delete
```

### 7.2 Recovery Procedure

```bash
# Restore from backup
gunzip < /backups/postgres/backup_20240101_000000.sql.gz | \
  psql -U job_info_user job_info_db
```

---

## 8. Security Hardening

### 8.1 TLS/SSL Certificate

```bash
# Using Let's Encrypt with Certbot
certbot certonly --standalone \
  -d app.example.com \
  -d www.app.example.com

# Auto-renew
certbot renew --dry-run
```

### 8.2 DDoS Protection

```nginx
# In nginx.conf
limit_req_zone $binary_remote_addr zone=general:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=api:10m rate=100r/s;

location / {
    limit_req zone=general burst=20 nodelay;
}

location /api/ {
    limit_req zone=api burst=200 nodelay;
}
```

### 8.3 Security Headers

```yaml
# Kubernetes SecurityPolicy
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: job-info-psp
spec:
  privileged: false
  allowPrivilegeEscalation: false
  requiredDropCapabilities:
    - ALL
  volumes:
    - 'configMap'
    - 'emptyDir'
    - 'projected'
    - 'secret'
    - 'downwardAPI'
    - 'persistentVolumeClaim'
  hostNetwork: false
  hostIPC: false
  hostPID: false
  runAsUser:
    rule: 'MustRunAsNonRoot'
  seLinux:
    rule: 'MustRunAs'
    seLinuxOptions:
      level: "s0:c123,c456"
```

---

## 9. Scaling & Performance Tuning

### 9.1 Database Optimization

```sql
-- Create indexes for common queries
CREATE INDEX idx_jobs_location ON jobs(location);
CREATE INDEX idx_jobs_company ON jobs(company_id);
CREATE INDEX idx_users_email ON users(email);

-- Monitor slow queries
SET log_min_duration_statement = 1000; -- Log queries > 1s

-- Connection pooling with PgBouncer
[databases]
job_info_db = host=prod-db port=5432 user=job_info_user password=...
```

### 9.2 Redis Caching

```typescript
// Implement caching layer
async function getJobsWithCache(userId: string): Promise<Job[]> {
  const cacheKey = `jobs:${userId}`;
  
  // Check cache first
  const cached = await redis.get(cacheKey);
  if (cached) return JSON.parse(cached);
  
  // Query database
  const jobs = await database.query(`SELECT * FROM jobs WHERE user_id = $1`, [userId]);
  
  // Cache for 5 minutes
  await redis.setex(cacheKey, 300, JSON.stringify(jobs));
  
  return jobs;
}
```

---

## 10. Deployment Checklist

**Pre-Deployment**:
- [ ] All tests passing (unit, integration, e2e)
- [ ] Code review approved
- [ ] Security scan completed (SAST, dependency check)
- [ ] Database migration tested
- [ ] Environment variables configured
- [ ] Backup taken
- [ ] Rollback plan prepared

**Deployment**:
- [ ] Deploy to staging first
- [ ] Run smoke tests on staging
- [ ] Verify metrics and logs
- [ ] Deploy to production (canary)
- [ ] Monitor error rate (< 0.1%)
- [ ] Monitor response time (< 200ms p95)
- [ ] Monitor CPU/Memory usage

**Post-Deployment**:
- [ ] Verify all endpoints responding
- [ ] Test OAuth flow
- [ ] Verify token refresh
- [ ] Check job loading
- [ ] Validate user experience
- [ ] Monitor for 24 hours
- [ ] Document any issues

---

## 11. Rollback Procedure

If deployment fails or causes issues:

```bash
# Kubernetes rollback
kubectl rollout undo deployment/job-info-backend
kubectl rollout undo deployment/job-info-frontend

# Docker rollback
docker service update --image job-info-backend:previous job-info-backend

# Database rollback (if needed)
psql -U job_info_user job_info_db < /backups/postgres/backup_20240101_000000.sql
```

---

## 12. Production Monitoring Dashboard

Setup Grafana dashboard with:
- Request rate (req/sec)
- Error rate (%)
- Response time (p50, p95, p99)
- CPU/Memory usage
- Database connections
- Redis hit rate
- Active sessions

**Alerts to configure**:
- Error rate > 1%
- Response time p95 > 1s
- CPU > 80%
- Memory > 85%
- Database connections > 80%
- Disk space < 10%

---

## Summary

Deployment checklist:
✅ Docker images built and tested
✅ Environment variables configured
✅ Database initialized with backups
✅ Kubernetes manifests created
✅ CI/CD pipeline configured
✅ Monitoring and logging setup
✅ Security hardening applied
✅ Scaling policies defined
✅ Rollback procedures documented
✅ On-call procedures established

**Next Steps**:
1. Set up production infrastructure
2. Run through deployment checklist
3. Deploy to staging first
4. Execute smoke tests
5. Deploy to production
6. Monitor for first 24 hours