Designing a Scalable Consent Management API

The API Call That Took 8 Seconds

Illustrative scenario: a composite example, not a specific reported case.

A major European news publisher discovered their consent API was adding 8 seconds to page load times during traffic spikes. The root cause: consent checks hitting a single PostgreSQL instance with no connection pooling, no caching, and queries scanning 200 million consent records without proper indexing.

Users abandoned pages. Ad revenue plummeted 34% during peak hours. And when the DPA requested consent records for an investigation, the API timed out trying to export them.

The fix wasn't complicated: read replicas, Redis caching, and proper indexes reduced latency from 8 seconds to 12 milliseconds. But the damage—lost revenue, user frustration, and a compliance investigation—had already been done.

A Consent Management API is the backbone of any privacy program at scale. While the frontend cookie banner gets attention, it's the API that handles the heavy lifting: storing billions of consent records, responding to millions of consent checks per day, propagating consent changes across dozens of downstream systems, and maintaining compliance audit trails.

Why does API design matter for consent management?

Poor API design creates cascading problems: slow consent checks delay page loads (users abandon sites after 3 seconds), unreliable webhooks lead to compliance violations, missing audit fields make regulatory responses impossible, and tight coupling prevents adapting to new privacy regulations.

Well-designed consent APIs share common characteristics: they're fast (sub-100ms for reads), reliable (highly available), auditable (complete history), flexible (support multiple frameworks), and scalable (handle traffic spikes without degradation).

API Architecture Overview

Consent API Design Principles

// consent-api-architecture.ts

/**
 * Core design principles for consent APIs:
 *
 * 1. SEPARATION OF CONCERNS
 *    - Consent storage separate from consent UI
 *    - Consent checks separate from consent updates
 *    - Audit logging separate from operational data
 *
 * 2. EVENTUAL CONSISTENCY WHERE APPROPRIATE
 *    - Consent writes are immediately consistent
 *    - Downstream propagation is eventually consistent
 *    - Analytics can be batch processed
 *
 * 3. IMMUTABILITY FOR COMPLIANCE
 *    - Consent records are append-only
 *    - Changes create new versions, not updates
 *    - Full audit trail is maintained
 *
 * 4. DEFENSE IN DEPTH
 *    - Multiple layers of validation
 *    - Fail closed (deny if uncertain)
 *    - Rate limiting at multiple levels
 */

interface ConsentAPIArchitecture {
  layers: {
    gateway: {
      responsibilities: ['Rate limiting', 'Authentication', 'Request validation', 'Response caching'];
      technology: 'API Gateway (Kong, AWS API Gateway)';
    };
    application: {
      responsibilities: ['Business logic', 'Consent validation', 'Framework compliance'];
      technology: 'Node.js/Express or FastAPI';
    };
    cache: {
      responsibilities: ['Consent state caching', 'Config caching', 'Session management'];
      technology: 'Redis Cluster';
    };
    database: {
      responsibilities: ['Consent storage', 'Audit logging', 'Analytics data'];
      technology: 'PostgreSQL with read replicas';
    };
    messaging: {
      responsibilities: ['Webhook delivery', 'Async processing', 'Event streaming'];
      technology: 'Apache Kafka or AWS SQS';
    };
  };

  scalability: {
    horizontal: 'Stateless application servers behind load balancer';
    vertical: 'Database read replicas for consent checks';
    geographic: 'Multi-region deployment for latency and compliance';
  };
}

Core API Endpoints

RESTful Consent API Design

// consent-api-endpoints.ts

import express, { Router, Request, Response, NextFunction } from 'express';
import { z } from 'zod';
import { ConsentService } from './services/ConsentService';
import { AuditService } from './services/AuditService';
import { WebhookService } from './services/WebhookService';
import { authenticate, authorize } from './middleware/auth';
import { rateLimit } from './middleware/rateLimit';
import { validateRequest } from './middleware/validation';

const router = Router();
const consentService = new ConsentService();
const auditService = new AuditService();
const webhookService = new WebhookService();

// ============================================
// SCHEMA DEFINITIONS
// ============================================

const ConsentChoiceSchema = z.object({
  purposeId: z.string(),
  vendorId: z.string().optional(),
  status: z.enum(['granted', 'denied', 'not_applicable']),
  timestamp: z.string().datetime().optional()
});

const CreateConsentSchema = z.object({
  userId: z.string().min(1).max(255),
  domainId: z.string().uuid(),
  choices: z.array(ConsentChoiceSchema).min(1),
  metadata: z.object({
    source: z.enum(['banner', 'preference_center', 'api', 'import']),
    userAgent: z.string().optional(),
    ipAddress: z.string().ip().optional(),
    geoLocation: z.object({
      country: z.string().length(2),
      region: z.string().optional()
    }).optional()
  }),
  frameworks: z.object({
    tcf: z.object({
      tcString: z.string().optional(),
      cmpId: z.number().optional(),
      cmpVersion: z.number().optional()
    }).optional(),
    gpp: z.object({
      gppString: z.string().optional(),
      sectionIds: z.array(z.number()).optional()
    }).optional(),
    usPrivacy: z.string().length(4).optional()
  }).optional()
});

const UpdateConsentSchema = CreateConsentSchema.partial().required({
  choices: true
});

// ============================================
// CONSENT CRUD ENDPOINTS
// ============================================

/**
 * POST /v1/consent
 * Record a new consent decision
 */
router.post(
  '/v1/consent',
  authenticate,
  rateLimit({ windowMs: 60000, max: 100 }),
  validateRequest(CreateConsentSchema),
  async (req: Request, res: Response, next: NextFunction) => {
    try {
      const consentData = req.body;

      // Validate domain belongs to authenticated client
      await authorize(req.auth.clientId, consentData.domainId);

      // Create consent record
      const consent = await consentService.createConsent({
        ...consentData,
        clientId: req.auth.clientId,
        createdAt: new Date()
      });

      // Log audit event
      await auditService.log({
        action: 'CONSENT_CREATED',
        resourceId: consent.id,
        resourceType: 'consent',
        clientId: req.auth.clientId,
        userId: consentData.userId,
        details: {
          choiceCount: consentData.choices.length,
          source: consentData.metadata.source
        },
        timestamp: new Date()
      });

      // Trigger webhooks asynchronously
      webhookService.triggerAsync('consent.created', consent);

      res.status(201).json({
        success: true,
        data: {
          id: consent.id,
          userId: consent.userId,
          domainId: consent.domainId,
          choices: consent.choices,
          createdAt: consent.createdAt,
          expiresAt: consent.expiresAt,
          tcString: consent.frameworks?.tcf?.tcString
        }
      });
    } catch (error) {
      next(error);
    }
  }
);

/**
 * GET /v1/consent/:userId/:domainId
 * Retrieve current consent state for a user on a domain
 */
router.get(
  '/v1/consent/:userId/:domainId',
  authenticate,
  rateLimit({ windowMs: 60000, max: 1000 }), // Higher limit for reads
  async (req: Request, res: Response, next: NextFunction) => {
    try {
      const { userId, domainId } = req.params;

      // Authorize access
      await authorize(req.auth.clientId, domainId);

      // Try cache first
      const cached = await consentService.getCachedConsent(userId, domainId);
      if (cached) {
        res.setHeader('X-Cache', 'HIT');
        res.setHeader('Cache-Control', 'private, max-age=60');
        return res.json({
          success: true,
          data: cached
        });
      }

      // Fetch from database
      const consent = await consentService.getConsent(userId, domainId);

      if (!consent) {
        return res.status(404).json({
          success: false,
          error: {
            code: 'CONSENT_NOT_FOUND',
            message: 'No consent record found for this user and domain'
          }
        });
      }

      // Cache for future requests
      await consentService.cacheConsent(consent);

      res.setHeader('X-Cache', 'MISS');
      res.json({
        success: true,
        data: consent
      });
    } catch (error) {
      next(error);
    }
  }
);

/**
 * PUT /v1/consent/:consentId
 * Update an existing consent record
 */
router.put(
  '/v1/consent/:consentId',
  authenticate,
  rateLimit({ windowMs: 60000, max: 100 }),
  validateRequest(UpdateConsentSchema),
  async (req: Request, res: Response, next: NextFunction) => {
    try {
      const { consentId } = req.params;
      const updateData = req.body;

      // Get existing consent
      const existing = await consentService.getConsentById(consentId);
      if (!existing) {
        return res.status(404).json({
          success: false,
          error: {
            code: 'CONSENT_NOT_FOUND',
            message: 'Consent record not found'
          }
        });
      }

      // Authorize
      await authorize(req.auth.clientId, existing.domainId);

      // Create new version (immutable updates)
      const updated = await consentService.updateConsent(consentId, {
        ...updateData,
        previousVersion: existing.id
      });

      // Audit log
      await auditService.log({
        action: 'CONSENT_UPDATED',
        resourceId: updated.id,
        resourceType: 'consent',
        clientId: req.auth.clientId,
        userId: existing.userId,
        details: {
          previousId: existing.id,
          changedPurposes: getChangedPurposes(existing.choices, updateData.choices)
        },
        timestamp: new Date()
      });

      // Invalidate cache
      await consentService.invalidateCache(existing.userId, existing.domainId);

      // Trigger webhooks
      webhookService.triggerAsync('consent.updated', {
        previous: existing,
        current: updated
      });

      res.json({
        success: true,
        data: updated
      });
    } catch (error) {
      next(error);
    }
  }
);

/**
 * DELETE /v1/consent/:userId/:domainId
 * Process a deletion request (GDPR Article 17)
 */
router.delete(
  '/v1/consent/:userId/:domainId',
  authenticate,
  rateLimit({ windowMs: 60000, max: 10 }), // Low limit for deletions
  async (req: Request, res: Response, next: NextFunction) => {
    try {
      const { userId, domainId } = req.params;
      const { verificationToken } = req.body;

      // Verify deletion request
      const verified = await consentService.verifyDeletionRequest(
        userId,
        domainId,
        verificationToken
      );

      if (!verified) {
        return res.status(403).json({
          success: false,
          error: {
            code: 'DELETION_NOT_VERIFIED',
            message: 'Deletion request could not be verified'
          }
        });
      }

      // Soft delete (mark for deletion, actual deletion async)
      const deletionJob = await consentService.scheduleConsentDeletion(
        userId,
        domainId,
        {
          requestedAt: new Date(),
          scheduledFor: new Date(Date.now() + 24 * 60 * 60 * 1000), // 24hr grace period
          clientId: req.auth.clientId
        }
      );

      // Audit log
      await auditService.log({
        action: 'CONSENT_DELETION_REQUESTED',
        resourceId: deletionJob.id,
        resourceType: 'deletion_job',
        clientId: req.auth.clientId,
        userId,
        details: { domainId, scheduledFor: deletionJob.scheduledFor },
        timestamp: new Date()
      });

      res.status(202).json({
        success: true,
        data: {
          deletionJobId: deletionJob.id,
          status: 'scheduled',
          scheduledFor: deletionJob.scheduledFor,
          canCancelUntil: deletionJob.scheduledFor
        }
      });
    } catch (error) {
      next(error);
    }
  }
);

// ============================================
// CONSENT CHECK ENDPOINTS (HIGH PERFORMANCE)
// ============================================

/**
 * GET /v1/consent/check
 * Fast consent check for specific purposes
 * Optimized for inline use in ad requests, analytics calls, etc.
 */
router.get(
  '/v1/consent/check',
  authenticate,
  rateLimit({ windowMs: 1000, max: 100 }), // Very high limit
  async (req: Request, res: Response, next: NextFunction) => {
    try {
      const { userId, domainId, purposes } = req.query;

      if (!userId || !domainId || !purposes) {
        return res.status(400).json({
          success: false,
          error: {
            code: 'MISSING_PARAMETERS',
            message: 'userId, domainId, and purposes are required'
          }
        });
      }

      const purposeList = (purposes as string).split(',');

      // Fast path: check cache only
      const result = await consentService.checkConsentFast(
        userId as string,
        domainId as string,
        purposeList
      );

      // Return minimal response for speed
      res.setHeader('Cache-Control', 'private, max-age=30');
      res.json({
        consented: result.consented,
        purposes: result.purposeResults,
        cached: result.fromCache,
        checkedAt: new Date().toISOString()
      });
    } catch (error) {
      // Fail closed: deny consent on error
      res.json({
        consented: false,
        purposes: {},
        error: true
      });
    }
  }
);

/**
 * POST /v1/consent/check/batch
 * Check consent for multiple users at once
 * Useful for backend batch processing
 */
router.post(
  '/v1/consent/check/batch',
  authenticate,
  rateLimit({ windowMs: 60000, max: 100 }),
  async (req: Request, res: Response, next: NextFunction) => {
    try {
      const { checks } = req.body;

      if (!Array.isArray(checks) || checks.length > 100) {
        return res.status(400).json({
          success: false,
          error: {
            code: 'INVALID_BATCH',
            message: 'Batch must be an array with max 100 items'
          }
        });
      }

      const results = await consentService.checkConsentBatch(checks);

      res.json({
        success: true,
        results
      });
    } catch (error) {
      next(error);
    }
  }
);

// ============================================
// CONFIGURATION ENDPOINTS
// ============================================

/**
 * GET /v1/config/:domainId
 * Get widget configuration for a domain
 */
router.get(
  '/v1/config/:domainId',
  authenticate,
  rateLimit({ windowMs: 60000, max: 100 }),
  async (req: Request, res: Response, next: NextFunction) => {
    try {
      const { domainId } = req.params;
      const { locale = 'en' } = req.query;

      const config = await consentService.getDomainConfig(
        domainId,
        locale as string
      );

      if (!config) {
        return res.status(404).json({
          success: false,
          error: {
            code: 'CONFIG_NOT_FOUND',
            message: 'Configuration not found for this domain'
          }
        });
      }

      // Long cache for config
      res.setHeader('Cache-Control', 'public, max-age=3600');
      res.json({
        success: true,
        data: config
      });
    } catch (error) {
      next(error);
    }
  }
);

/**
 * GET /v1/tcf/gvl
 * Get IAB Global Vendor List
 */
router.get(
  '/v1/tcf/gvl',
  rateLimit({ windowMs: 60000, max: 60 }),
  async (req: Request, res: Response, next: NextFunction) => {
    try {
      const { version = 'latest' } = req.query;

      const gvl = await consentService.getGlobalVendorList(version as string);

      res.setHeader('Cache-Control', 'public, max-age=86400'); // Cache for 24hr
      res.json(gvl);
    } catch (error) {
      next(error);
    }
  }
);

export default router;

Database Schema Design

PostgreSQL Schema for Consent Storage

-- consent-database-schema.sql

-- Enable required extensions
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "pgcrypto";

-- ============================================
-- CORE TABLES
-- ============================================

-- Domains/properties that collect consent
CREATE TABLE domains (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    client_id UUID NOT NULL REFERENCES clients(id),
    domain_name VARCHAR(255) NOT NULL,
    display_name VARCHAR(255),

    -- Configuration
    config JSONB NOT NULL DEFAULT '{}',
    supported_frameworks VARCHAR(50)[] DEFAULT ARRAY['basic'],
    default_language VARCHAR(10) DEFAULT 'en',

    -- Status
    is_active BOOLEAN DEFAULT true,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),

    UNIQUE(client_id, domain_name)
);

CREATE INDEX idx_domains_client ON domains(client_id);
CREATE INDEX idx_domains_name ON domains(domain_name);

-- Consent purposes/categories
CREATE TABLE consent_purposes (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    domain_id UUID NOT NULL REFERENCES domains(id),

    -- Purpose identification
    purpose_key VARCHAR(100) NOT NULL, -- e.g., 'analytics', 'marketing'
    external_id VARCHAR(100), -- TCF purpose ID, etc.

    -- Display
    name JSONB NOT NULL, -- {"en": "Analytics", "de": "Analyse"}
    description JSONB NOT NULL,

    -- Legal basis
    legal_basis VARCHAR(50) NOT NULL, -- consent, legitimate_interest, contract, etc.
    is_required BOOLEAN DEFAULT false,

    -- Categorization
    category VARCHAR(50), -- necessary, functional, analytics, marketing
    sort_order INTEGER DEFAULT 0,

    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),

    UNIQUE(domain_id, purpose_key)
);

CREATE INDEX idx_purposes_domain ON consent_purposes(domain_id);

-- Main consent records table (immutable, append-only)
CREATE TABLE consent_records (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),

    -- User identification (hashed for privacy)
    user_id VARCHAR(255) NOT NULL,
    user_id_hash VARCHAR(64) NOT NULL, -- SHA-256 for indexing

    -- Domain
    domain_id UUID NOT NULL REFERENCES domains(id),

    -- Version chain
    version INTEGER NOT NULL DEFAULT 1,
    previous_version_id UUID REFERENCES consent_records(id),

    -- Consent choices stored as JSONB for flexibility
    choices JSONB NOT NULL,
    /*
    Example structure:
    {
        "purposes": {
            "analytics": {"status": "granted", "timestamp": "..."},
            "marketing": {"status": "denied", "timestamp": "..."}
        },
        "vendors": {
            "vendor_123": {"status": "granted", "purposes": ["analytics"]}
        }
    }
    */

    -- Framework-specific strings
    tc_string TEXT, -- IAB TCF consent string
    gpp_string TEXT, -- Global Privacy Platform string
    us_privacy_string VARCHAR(4), -- CCPA/USP string

    -- Metadata
    metadata JSONB NOT NULL DEFAULT '{}',
    /*
    {
        "source": "banner",
        "user_agent": "...",
        "ip_country": "DE",
        "ip_region": "BY",
        "consent_version": "1.2.0"
    }
    */

    -- Timestamps
    created_at TIMESTAMPTZ DEFAULT NOW(),
    expires_at TIMESTAMPTZ, -- Consent validity period

    -- Soft delete support
    deleted_at TIMESTAMPTZ,
    deletion_reason VARCHAR(100)
);

-- Optimized indexes for common queries
CREATE INDEX idx_consent_user_domain ON consent_records(user_id_hash, domain_id)
    WHERE deleted_at IS NULL;
CREATE INDEX idx_consent_domain_created ON consent_records(domain_id, created_at DESC);
CREATE INDEX idx_consent_expires ON consent_records(expires_at)
    WHERE expires_at IS NOT NULL AND deleted_at IS NULL;

-- Partitioning for large-scale deployments
-- CREATE TABLE consent_records_2024 PARTITION OF consent_records
--     FOR VALUES FROM ('2024-01-01') TO ('2025-01-01');

-- ============================================
-- AUDIT & COMPLIANCE TABLES
-- ============================================

-- Comprehensive audit log
CREATE TABLE audit_logs (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),

    -- What happened
    action VARCHAR(100) NOT NULL,
    resource_type VARCHAR(50) NOT NULL,
    resource_id UUID,

    -- Who did it
    client_id UUID REFERENCES clients(id),
    user_id VARCHAR(255),
    api_key_id UUID,

    -- Details
    details JSONB DEFAULT '{}',

    -- Request context
    ip_address INET,
    user_agent TEXT,
    request_id UUID,

    -- Timing
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Time-series partitioning for audit logs
CREATE INDEX idx_audit_created ON audit_logs(created_at DESC);
CREATE INDEX idx_audit_resource ON audit_logs(resource_type, resource_id);
CREATE INDEX idx_audit_client ON audit_logs(client_id, created_at DESC);

-- Consent history view (aggregated for reporting)
CREATE TABLE consent_history_daily (
    date DATE NOT NULL,
    domain_id UUID NOT NULL REFERENCES domains(id),

    -- Aggregated metrics
    total_consents INTEGER DEFAULT 0,
    total_updates INTEGER DEFAULT 0,
    total_deletions INTEGER DEFAULT 0,

    -- By source
    consents_from_banner INTEGER DEFAULT 0,
    consents_from_preference_center INTEGER DEFAULT 0,
    consents_from_api INTEGER DEFAULT 0,

    -- By choice
    analytics_granted INTEGER DEFAULT 0,
    analytics_denied INTEGER DEFAULT 0,
    marketing_granted INTEGER DEFAULT 0,
    marketing_denied INTEGER DEFAULT 0,

    -- Rates
    accept_all_rate DECIMAL(5,4),
    reject_all_rate DECIMAL(5,4),
    customize_rate DECIMAL(5,4),

    PRIMARY KEY (date, domain_id)
);

-- ============================================
-- WEBHOOK & INTEGRATION TABLES
-- ============================================

CREATE TABLE webhooks (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    client_id UUID NOT NULL REFERENCES clients(id),

    -- Configuration
    url TEXT NOT NULL,
    events VARCHAR(100)[] NOT NULL, -- ['consent.created', 'consent.updated']
    secret_hash VARCHAR(64) NOT NULL, -- For HMAC signing

    -- Status
    is_active BOOLEAN DEFAULT true,
    last_triggered_at TIMESTAMPTZ,
    failure_count INTEGER DEFAULT 0,

    -- Retry configuration
    max_retries INTEGER DEFAULT 3,
    retry_delay_seconds INTEGER DEFAULT 60,

    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE webhook_deliveries (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    webhook_id UUID NOT NULL REFERENCES webhooks(id),

    -- Event details
    event_type VARCHAR(100) NOT NULL,
    payload JSONB NOT NULL,

    -- Delivery status
    status VARCHAR(20) DEFAULT 'pending', -- pending, delivered, failed
    attempts INTEGER DEFAULT 0,

    -- Response details
    response_status INTEGER,
    response_body TEXT,
    response_time_ms INTEGER,

    -- Timing
    created_at TIMESTAMPTZ DEFAULT NOW(),
    delivered_at TIMESTAMPTZ,
    next_retry_at TIMESTAMPTZ
);

CREATE INDEX idx_webhook_deliveries_pending ON webhook_deliveries(next_retry_at)
    WHERE status = 'pending';

Caching Strategy for Sub-100ms Response Times

// consent-caching-service.ts

import Redis from 'ioredis';
import { createHash } from 'crypto';

interface CacheConfig {
  redis: {
    host: string;
    port: number;
    cluster?: boolean;
    sentinels?: { host: string; port: number }[];
  };
  ttl: {
    consent: number;      // Consent records: 5 minutes
    config: number;       // Domain config: 1 hour
    gvl: number;          // Vendor list: 24 hours
    consentCheck: number; // Fast consent checks: 30 seconds
  };
  compression: boolean;
}

class ConsentCacheService {
  private redis: Redis.Redis | Redis.Cluster;
  private config: CacheConfig;
  private localCache: Map<string, { data: any; expires: number }>;

  constructor(config: CacheConfig) {
    this.config = config;
    this.localCache = new Map();

    // Initialize Redis connection
    if (config.redis.cluster) {
      this.redis = new Redis.Cluster([
        { host: config.redis.host, port: config.redis.port }
      ]);
    } else if (config.redis.sentinels) {
      this.redis = new Redis({
        sentinels: config.redis.sentinels,
        name: 'consent-master'
      });
    } else {
      this.redis = new Redis(config.redis);
    }

    // Start local cache cleanup
    setInterval(() => this.cleanLocalCache(), 10000);
  }

  // ============================================
  // CONSENT STATE CACHING
  // ============================================

  async getConsentState(
    userId: string,
    domainId: string
  ): Promise<ConsentState | null> {
    const key = this.consentKey(userId, domainId);

    // L1: Check local in-memory cache
    const local = this.getLocal(key);
    if (local) return local;

    // L2: Check Redis
    const cached = await this.redis.get(key);
    if (!cached) return null;

    const consent = this.deserialize(cached);

    // Populate L1 cache
    this.setLocal(key, consent, 5000); // 5 second local TTL

    return consent;
  }

  async setConsentState(
    userId: string,
    domainId: string,
    consent: ConsentState
  ): Promise<void> {
    const key = this.consentKey(userId, domainId);
    const serialized = this.serialize(consent);

    // Set in Redis with TTL
    await this.redis.setex(key, this.config.ttl.consent, serialized);

    // Update local cache
    this.setLocal(key, consent, 5000);

    // Also cache the fast-check version
    await this.cacheConsentCheck(userId, domainId, consent);
  }

  async invalidateConsentState(
    userId: string,
    domainId: string
  ): Promise<void> {
    const key = this.consentKey(userId, domainId);
    const checkKey = this.consentCheckKey(userId, domainId);

    // Delete from all cache layers
    await this.redis.del(key, checkKey);
    this.localCache.delete(key);
    this.localCache.delete(checkKey);
  }

  // ============================================
  // FAST CONSENT CHECK CACHING
  // ============================================

  /**
   * Ultra-fast consent check optimized for ad requests
   * Returns only the essential consent signals
   */
  async getConsentCheckFast(
    userId: string,
    domainId: string,
    purposes: string[]
  ): Promise<ConsentCheckResult | null> {
    const key = this.consentCheckKey(userId, domainId);

    // L1: Local cache (fastest)
    const local = this.getLocal(key);
    if (local) {
      return this.filterPurposes(local, purposes);
    }

    // L2: Redis
    const cached = await this.redis.get(key);
    if (!cached) return null;

    const checkData = this.deserialize(cached);
    this.setLocal(key, checkData, 2000); // Very short local TTL for fast checks

    return this.filterPurposes(checkData, purposes);
  }

  private async cacheConsentCheck(
    userId: string,
    domainId: string,
    consent: ConsentState
  ): Promise<void> {
    // Create minimal check data structure
    const checkData: ConsentCheckData = {
      purposes: {},
      tcString: consent.frameworks?.tcf?.tcString,
      updatedAt: consent.updatedAt
    };

    for (const choice of consent.choices) {
      checkData.purposes[choice.purposeId] = choice.status === 'granted';
    }

    const key = this.consentCheckKey(userId, domainId);
    await this.redis.setex(
      key,
      this.config.ttl.consentCheck,
      this.serialize(checkData)
    );
  }

  private filterPurposes(
    checkData: ConsentCheckData,
    purposes: string[]
  ): ConsentCheckResult {
    const result: ConsentCheckResult = {
      consented: true,
      purposeResults: {},
      fromCache: true
    };

    for (const purpose of purposes) {
      const consented = checkData.purposes[purpose] ?? false;
      result.purposeResults[purpose] = consented;
      if (!consented) {
        result.consented = false;
      }
    }

    return result;
  }

  // ============================================
  // DOMAIN CONFIG CACHING
  // ============================================

  async getDomainConfig(
    domainId: string,
    locale: string
  ): Promise<DomainConfig | null> {
    const key = `config:${domainId}:${locale}`;

    // Check local cache
    const local = this.getLocal(key);
    if (local) return local;

    // Check Redis
    const cached = await this.redis.get(key);
    if (!cached) return null;

    const config = this.deserialize(cached);
    this.setLocal(key, config, 60000); // 1 minute local TTL

    return config;
  }

  async setDomainConfig(
    domainId: string,
    locale: string,
    config: DomainConfig
  ): Promise<void> {
    const key = `config:${domainId}:${locale}`;
    await this.redis.setex(key, this.config.ttl.config, this.serialize(config));
    this.setLocal(key, config, 60000);
  }

  // ============================================
  // CACHE KEY GENERATION
  // ============================================

  private consentKey(userId: string, domainId: string): string {
    // Hash user ID for privacy
    const userHash = this.hashUserId(userId);
    return `consent:${domainId}:${userHash}`;
  }

  private consentCheckKey(userId: string, domainId: string): string {
    const userHash = this.hashUserId(userId);
    return `check:${domainId}:${userHash}`;
  }

  private hashUserId(userId: string): string {
    return createHash('sha256').update(userId).digest('hex').substring(0, 16);
  }

  // ============================================
  // SERIALIZATION
  // ============================================

  private serialize(data: any): string {
    const json = JSON.stringify(data);
    if (this.config.compression && json.length > 1000) {
      // Compress large payloads (implement with zlib if needed)
      return json;
    }
    return json;
  }

  private deserialize(data: string): any {
    return JSON.parse(data);
  }

  // ============================================
  // LOCAL CACHE MANAGEMENT
  // ============================================

  private getLocal(key: string): any | null {
    const entry = this.localCache.get(key);
    if (!entry) return null;
    if (Date.now() > entry.expires) {
      this.localCache.delete(key);
      return null;
    }
    return entry.data;
  }

  private setLocal(key: string, data: any, ttlMs: number): void {
    this.localCache.set(key, {
      data,
      expires: Date.now() + ttlMs
    });
  }

  private cleanLocalCache(): void {
    const now = Date.now();
    for (const [key, entry] of this.localCache) {
      if (now > entry.expires) {
        this.localCache.delete(key);
      }
    }
  }

  // ============================================
  // CACHE WARMING
  // ============================================

  async warmCache(domainId: string): Promise<void> {
    // Fetch recent consent records and pre-populate cache
    const recentConsents = await this.fetchRecentConsents(domainId, 1000);

    const pipeline = this.redis.pipeline();
    for (const consent of recentConsents) {
      const key = this.consentKey(consent.userId, domainId);
      pipeline.setex(key, this.config.ttl.consent, this.serialize(consent));
    }
    await pipeline.exec();
  }

  private async fetchRecentConsents(
    domainId: string,
    limit: number
  ): Promise<ConsentState[]> {
    // Implementation would query database
    return [];
  }
}

Webhook Implementation for Real-Time Propagation

// webhook-service.ts

import crypto from 'crypto';
import axios, { AxiosError } from 'axios';
import { Queue, Worker, Job } from 'bullmq';

interface WebhookConfig {
  maxRetries: number;
  retryDelays: number[]; // [60, 300, 900, 3600] seconds
  timeout: number;
  signatureHeader: string;
}

interface WebhookDelivery {
  id: string;
  webhookId: string;
  eventType: string;
  payload: any;
  attempt: number;
  status: 'pending' | 'delivered' | 'failed';
  responseStatus?: number;
  responseBody?: string;
  error?: string;
}

class WebhookService {
  private queue: Queue;
  private worker: Worker;
  private config: WebhookConfig;

  constructor(config: WebhookConfig) {
    this.config = config;

    // Initialize BullMQ queue
    this.queue = new Queue('webhooks', {
      defaultJobOptions: {
        attempts: config.maxRetries,
        backoff: {
          type: 'custom'
        },
        removeOnComplete: { age: 86400 }, // Keep for 24hr
        removeOnFail: { age: 604800 } // Keep failed for 7 days
      }
    });

    // Initialize worker
    this.worker = new Worker('webhooks', this.processWebhook.bind(this), {
      concurrency: 10,
      limiter: {
        max: 100,
        duration: 1000
      }
    });

    this.setupEventHandlers();
  }

  // ============================================
  // WEBHOOK TRIGGERING
  // ============================================

  async triggerAsync(eventType: string, payload: any): Promise<void> {
    // Get all active webhooks for this event type
    const webhooks = await this.getWebhooksForEvent(eventType);

    for (const webhook of webhooks) {
      await this.queue.add(
        'deliver',
        {
          webhookId: webhook.id,
          url: webhook.url,
          secret: webhook.secret,
          eventType,
          payload,
          attempt: 1
        },
        {
          jobId: `${webhook.id}-${Date.now()}`,
          priority: this.getPriority(eventType)
        }
      );
    }
  }

  private getPriority(eventType: string): number {
    // Higher priority for critical events
    if (eventType.startsWith('consent.deleted')) return 1;
    if (eventType.startsWith('consent.')) return 5;
    return 10;
  }

  // ============================================
  // WEBHOOK DELIVERY
  // ============================================

  private async processWebhook(job: Job): Promise<void> {
    const { webhookId, url, secret, eventType, payload, attempt } = job.data;

    const deliveryId = `del_${crypto.randomUUID()}`;
    const timestamp = Math.floor(Date.now() / 1000);

    // Create signed payload
    const signedPayload = this.createSignedPayload(
      payload,
      eventType,
      timestamp,
      deliveryId
    );

    // Generate signature
    const signature = this.generateSignature(
      JSON.stringify(signedPayload),
      secret,
      timestamp
    );

    try {
      const startTime = Date.now();

      const response = await axios.post(url, signedPayload, {
        headers: {
          'Content-Type': 'application/json',
          [this.config.signatureHeader]: signature,
          'X-Webhook-Delivery-Id': deliveryId,
          'X-Webhook-Timestamp': timestamp.toString(),
          'X-Webhook-Event': eventType
        },
        timeout: this.config.timeout,
        validateStatus: () => true // Don't throw on non-2xx
      });

      const responseTime = Date.now() - startTime;

      // Log delivery
      await this.logDelivery({
        id: deliveryId,
        webhookId,
        eventType,
        payload: signedPayload,
        attempt,
        status: response.status >= 200 && response.status < 300 ? 'delivered' : 'failed',
        responseStatus: response.status,
        responseBody: typeof response.data === 'string'
          ? response.data.substring(0, 1000)
          : JSON.stringify(response.data).substring(0, 1000),
        responseTimeMs: responseTime
      });

      // Throw if not successful (triggers retry)
      if (response.status < 200 || response.status >= 300) {
        throw new Error(`Webhook returned status ${response.status}`);
      }

      // Update webhook health
      await this.updateWebhookHealth(webhookId, true);

    } catch (error) {
      const axiosError = error as AxiosError;

      await this.logDelivery({
        id: deliveryId,
        webhookId,
        eventType,
        payload: signedPayload,
        attempt,
        status: 'failed',
        error: axiosError.message
      });

      // Update webhook health
      await this.updateWebhookHealth(webhookId, false);

      throw error; // Let BullMQ handle retry
    }
  }

  // ============================================
  // SIGNATURE GENERATION
  // ============================================

  private createSignedPayload(
    payload: any,
    eventType: string,
    timestamp: number,
    deliveryId: string
  ): WebhookPayload {
    return {
      id: deliveryId,
      event: eventType,
      created_at: new Date(timestamp * 1000).toISOString(),
      data: payload,
      api_version: '2024-01-01'
    };
  }

  private generateSignature(
    payload: string,
    secret: string,
    timestamp: number
  ): string {
    // Format: t=timestamp,v1=signature
    const signedPayload = `${timestamp}.${payload}`;
    const signature = crypto
      .createHmac('sha256', secret)
      .update(signedPayload)
      .digest('hex');

    return `t=${timestamp},v1=${signature}`;
  }

  // ============================================
  // WEBHOOK VERIFICATION (for recipients)
  // ============================================

  static verifyWebhookSignature(
    payload: string,
    signature: string,
    secret: string,
    tolerance: number = 300 // 5 minutes
  ): boolean {
    const parts = signature.split(',').reduce((acc, part) => {
      const [key, value] = part.split('=');
      acc[key] = value;
      return acc;
    }, {} as Record<string, string>);

    const timestamp = parseInt(parts.t, 10);
    const providedSignature = parts.v1;

    // Check timestamp tolerance
    const now = Math.floor(Date.now() / 1000);
    if (Math.abs(now - timestamp) > tolerance) {
      throw new Error('Webhook timestamp outside tolerance');
    }

    // Verify signature
    const signedPayload = `${timestamp}.${payload}`;
    const expectedSignature = crypto
      .createHmac('sha256', secret)
      .update(signedPayload)
      .digest('hex');

    return crypto.timingSafeEqual(
      Buffer.from(providedSignature),
      Buffer.from(expectedSignature)
    );
  }

  // ============================================
  // WEBHOOK HEALTH MANAGEMENT
  // ============================================

  private async updateWebhookHealth(
    webhookId: string,
    success: boolean
  ): Promise<void> {
    if (success) {
      // Reset failure count
      await this.db.query(`
        UPDATE webhooks
        SET
          failure_count = 0,
          last_triggered_at = NOW()
        WHERE id = $1
      `, [webhookId]);
    } else {
      // Increment failure count
      const result = await this.db.query(`
        UPDATE webhooks
        SET
          failure_count = failure_count + 1,
          last_triggered_at = NOW()
        WHERE id = $1
        RETURNING failure_count
      `, [webhookId]);

      // Auto-disable after too many failures
      if (result.rows[0].failure_count >= 10) {
        await this.disableWebhook(webhookId, 'Too many consecutive failures');
      }
    }
  }

  private async disableWebhook(
    webhookId: string,
    reason: string
  ): Promise<void> {
    await this.db.query(`
      UPDATE webhooks
      SET
        is_active = false,
        disabled_reason = $2,
        disabled_at = NOW()
      WHERE id = $1
    `, [webhookId, reason]);

    // Notify webhook owner
    await this.notifyWebhookDisabled(webhookId, reason);
  }
}

// ============================================
// WEBHOOK EVENT TYPES
// ============================================

interface WebhookEvents {
  'consent.created': {
    consent_id: string;
    user_id: string;
    domain_id: string;
    choices: ConsentChoice[];
    tc_string?: string;
    created_at: string;
  };

  'consent.updated': {
    consent_id: string;
    user_id: string;
    domain_id: string;
    previous_choices: ConsentChoice[];
    new_choices: ConsentChoice[];
    changed_purposes: string[];
    updated_at: string;
  };

  'consent.deleted': {
    user_id: string;
    domain_id: string;
    deleted_at: string;
    deletion_type: 'user_request' | 'retention_policy' | 'client_request';
  };

  'consent.expired': {
    consent_id: string;
    user_id: string;
    domain_id: string;
    expired_at: string;
    original_created_at: string;
  };
}

FAQ: Consent Management API Design

How do I handle API versioning as privacy regulations evolve?

Use URL-based versioning (e.g., /v1/consent, /v2/consent) for major changes and header-based versioning for minor updates. Maintain backward compatibility for at least 12 months. When GDPR or similar regulations change, create a new API version rather than modifying existing endpoints. Use feature flags to gradually migrate clients. Document sunset dates and provide migration guides.

What's the best authentication approach for consent APIs?

For server-to-server communication, use API keys with HMAC signature verification. For client-side widgets, use domain-restricted API keys that only work from registered domains. Implement JWT tokens for authenticated user sessions. Rate limit aggressively based on API key. Never expose server-side API keys in client code—use a proxy endpoint instead.

How do I achieve sub-100ms response times for consent checks?

Layer your caching: L1 (in-process memory, 1-5 second TTL), L2 (Redis, 30-60 second TTL), L3 (database). Use read replicas for database queries. Pre-compute consent check results into a minimal data structure optimized for fast lookup. Use connection pooling for both Redis and database. Place edge caches in multiple geographic regions. Return cached results even if slightly stale—eventual consistency is acceptable for consent checks.

How do I handle consent sync across multiple domains?

For same-company multi-domain consent sharing, implement a consent synchronization service that propagates consent changes via webhooks or event streaming. Use a canonical user identifier that works across domains. Be careful with cross-domain consent sharing—GDPR generally requires separate consent per data controller. Document the legal basis for any cross-domain consent sharing.

What audit data should I capture for compliance?

Capture: full consent record, timestamp, source (banner/preference center/API), user agent, IP geolocation (country/region only), consent version, any framework-specific strings (TCF, GPP), request ID for traceability. Do NOT capture: full IP addresses (hash or truncate), device fingerprints, or any data that could identify the user beyond their consent identifier. Retain audit data for at least the consent validity period plus one year.

How do I handle high-volume consent operations during traffic spikes?

Implement queue-based processing for writes while serving reads from cache. Use database connection pooling with queue overflow. Implement circuit breakers that fail gracefully (default to requiring consent). Auto-scale based on queue depth, not just CPU. Pre-warm caches before anticipated traffic spikes. Consider write-behind caching for consent updates—acknowledge immediately, persist asynchronously with guaranteed delivery.

Building for the Future of Privacy Regulation

The consent management API landscape will continue evolving as new regulations emerge and existing ones are updated. Design your APIs with extensibility in mind: use flexible JSONB fields for framework-specific data, maintain loose coupling between your consent storage and framework-specific logic, and build abstraction layers that can accommodate new consent frameworks without database migrations.

The patterns shown here—immutable consent records, comprehensive audit logging, webhook-based propagation, and aggressive caching—provide a foundation that can scale from thousands to billions of consent records while maintaining the performance and reliability that modern privacy compliance demands.