TLDR: Well-designed consent APIs enable seamless integration while maintaining compliance, sub-100ms latency for consent checks, and supporting millions of daily consent operations. Poor API design causes cascading failures—slow checks delay pages, unreliable webhooks cause violations, missing audit fields doom DSAR responses.
Read full summary
This comprehensive guide covers building scalable consent management APIs from the ground up. Learn RESTful design patterns for consent CRUD operations, webhook implementations for real-time consent propagation, rate limiting strategies, authentication and authorization patterns, caching for sub-100ms response times, database schema design, and API versioning for evolving privacy regulations. Includes complete TypeScript/Node.js implementations with PostgreSQL and Redis.
*Summary by Claude AI*
## The API Call That Took 8 Seconds
A major European news publisher discovered their consent API was adding 8 seconds to page load times during traffic spikes. The root cause: consent checks hitting a single PostgreSQL instance with no connection pooling, no caching, and queries scanning 200 million consent records without proper indexing.
Users abandoned pages. Ad revenue plummeted 34% during peak hours. And when the DPA requested consent records for an investigation, the API timed out trying to export them.
The fix wasn't complicated: read replicas, Redis caching, and proper indexes reduced latency from 8 seconds to 12 milliseconds. But the damage—lost revenue, user frustration, and a compliance investigation—had already been done.
A Consent Management API is the backbone of any privacy program at scale. While the frontend cookie banner gets attention, it's the API that handles the heavy lifting: storing billions of consent records, responding to millions of consent checks per day, propagating consent changes across dozens of downstream systems, and maintaining compliance audit trails.
**Why does API design matter for consent management?**
Poor API design creates cascading problems: slow consent checks delay page loads (users abandon sites after 3 seconds), unreliable webhooks lead to compliance violations, missing audit fields make regulatory responses impossible, and tight coupling prevents adapting to new privacy regulations.
Well-designed consent APIs share common characteristics: they're fast (sub-100ms for reads), reliable (99.99% uptime), auditable (complete history), flexible (support multiple frameworks), and scalable (handle traffic spikes without degradation).
## API Architecture Overview
### Consent API Design Principles
```typescript
// consent-api-architecture.ts
/**
* Core design principles for consent APIs:
*
* 1. SEPARATION OF CONCERNS
* - Consent storage separate from consent UI
* - Consent checks separate from consent updates
* - Audit logging separate from operational data
*
* 2. EVENTUAL CONSISTENCY WHERE APPROPRIATE
* - Consent writes are immediately consistent
* - Downstream propagation is eventually consistent
* - Analytics can be batch processed
*
* 3. IMMUTABILITY FOR COMPLIANCE
* - Consent records are append-only
* - Changes create new versions, not updates
* - Full audit trail is maintained
*
* 4. DEFENSE IN DEPTH
* - Multiple layers of validation
* - Fail closed (deny if uncertain)
* - Rate limiting at multiple levels
*/
interface ConsentAPIArchitecture {
layers: {
gateway: {
responsibilities: ['Rate limiting', 'Authentication', 'Request validation', 'Response caching'];
technology: 'API Gateway (Kong, AWS API Gateway)';
};
application: {
responsibilities: ['Business logic', 'Consent validation', 'Framework compliance'];
technology: 'Node.js/Express or FastAPI';
};
cache: {
responsibilities: ['Consent state caching', 'Config caching', 'Session management'];
technology: 'Redis Cluster';
};
database: {
responsibilities: ['Consent storage', 'Audit logging', 'Analytics data'];
technology: 'PostgreSQL with read replicas';
};
messaging: {
responsibilities: ['Webhook delivery', 'Async processing', 'Event streaming'];
technology: 'Apache Kafka or AWS SQS';
};
};
scalability: {
horizontal: 'Stateless application servers behind load balancer';
vertical: 'Database read replicas for consent checks';
geographic: 'Multi-region deployment for latency and compliance';
};
}
```
## Core API Endpoints
### RESTful Consent API Design
```typescript
// consent-api-endpoints.ts
import express, { Router, Request, Response, NextFunction } from 'express';
import { z } from 'zod';
import { ConsentService } from './services/ConsentService';
import { AuditService } from './services/AuditService';
import { WebhookService } from './services/WebhookService';
import { authenticate, authorize } from './middleware/auth';
import { rateLimit } from './middleware/rateLimit';
import { validateRequest } from './middleware/validation';
const router = Router();
const consentService = new ConsentService();
const auditService = new AuditService();
const webhookService = new WebhookService();
// ============================================
// SCHEMA DEFINITIONS
// ============================================
const ConsentChoiceSchema = z.object({
purposeId: z.string(),
vendorId: z.string().optional(),
status: z.enum(['granted', 'denied', 'not_applicable']),
timestamp: z.string().datetime().optional()
});
const CreateConsentSchema = z.object({
userId: z.string().min(1).max(255),
domainId: z.string().uuid(),
choices: z.array(ConsentChoiceSchema).min(1),
metadata: z.object({
source: z.enum(['banner', 'preference_center', 'api', 'import']),
userAgent: z.string().optional(),
ipAddress: z.string().ip().optional(),
geoLocation: z.object({
country: z.string().length(2),
region: z.string().optional()
}).optional()
}),
frameworks: z.object({
tcf: z.object({
tcString: z.string().optional(),
cmpId: z.number().optional(),
cmpVersion: z.number().optional()
}).optional(),
gpp: z.object({
gppString: z.string().optional(),
sectionIds: z.array(z.number()).optional()
}).optional(),
usPrivacy: z.string().length(4).optional()
}).optional()
});
const UpdateConsentSchema = CreateConsentSchema.partial().required({
choices: true
});
// ============================================
// CONSENT CRUD ENDPOINTS
// ============================================
/**
* POST /v1/consent
* Record a new consent decision
*/
router.post(
'/v1/consent',
authenticate,
rateLimit({ windowMs: 60000, max: 100 }),
validateRequest(CreateConsentSchema),
async (req: Request, res: Response, next: NextFunction) => {
try {
const consentData = req.body;
// Validate domain belongs to authenticated client
await authorize(req.auth.clientId, consentData.domainId);
// Create consent record
const consent = await consentService.createConsent({
...consentData,
clientId: req.auth.clientId,
createdAt: new Date()
});
// Log audit event
await auditService.log({
action: 'CONSENT_CREATED',
resourceId: consent.id,
resourceType: 'consent',
clientId: req.auth.clientId,
userId: consentData.userId,
details: {
choiceCount: consentData.choices.length,
source: consentData.metadata.source
},
timestamp: new Date()
});
// Trigger webhooks asynchronously
webhookService.triggerAsync('consent.created', consent);
res.status(201).json({
success: true,
data: {
id: consent.id,
userId: consent.userId,
domainId: consent.domainId,
choices: consent.choices,
createdAt: consent.createdAt,
expiresAt: consent.expiresAt,
tcString: consent.frameworks?.tcf?.tcString
}
});
} catch (error) {
next(error);
}
}
);
/**
* GET /v1/consent/:userId/:domainId
* Retrieve current consent state for a user on a domain
*/
router.get(
'/v1/consent/:userId/:domainId',
authenticate,
rateLimit({ windowMs: 60000, max: 1000 }), // Higher limit for reads
async (req: Request, res: Response, next: NextFunction) => {
try {
const { userId, domainId } = req.params;
// Authorize access
await authorize(req.auth.clientId, domainId);
// Try cache first
const cached = await consentService.getCachedConsent(userId, domainId);
if (cached) {
res.setHeader('X-Cache', 'HIT');
res.setHeader('Cache-Control', 'private, max-age=60');
return res.json({
success: true,
data: cached
});
}
// Fetch from database
const consent = await consentService.getConsent(userId, domainId);
if (!consent) {
return res.status(404).json({
success: false,
error: {
code: 'CONSENT_NOT_FOUND',
message: 'No consent record found for this user and domain'
}
});
}
// Cache for future requests
await consentService.cacheConsent(consent);
res.setHeader('X-Cache', 'MISS');
res.json({
success: true,
data: consent
});
} catch (error) {
next(error);
}
}
);
/**
* PUT /v1/consent/:consentId
* Update an existing consent record
*/
router.put(
'/v1/consent/:consentId',
authenticate,
rateLimit({ windowMs: 60000, max: 100 }),
validateRequest(UpdateConsentSchema),
async (req: Request, res: Response, next: NextFunction) => {
try {
const { consentId } = req.params;
const updateData = req.body;
// Get existing consent
const existing = await consentService.getConsentById(consentId);
if (!existing) {
return res.status(404).json({
success: false,
error: {
code: 'CONSENT_NOT_FOUND',
message: 'Consent record not found'
}
});
}
// Authorize
await authorize(req.auth.clientId, existing.domainId);
// Create new version (immutable updates)
const updated = await consentService.updateConsent(consentId, {
...updateData,
previousVersion: existing.id
});
// Audit log
await auditService.log({
action: 'CONSENT_UPDATED',
resourceId: updated.id,
resourceType: 'consent',
clientId: req.auth.clientId,
userId: existing.userId,
details: {
previousId: existing.id,
changedPurposes: getChangedPurposes(existing.choices, updateData.choices)
},
timestamp: new Date()
});
// Invalidate cache
await consentService.invalidateCache(existing.userId, existing.domainId);
// Trigger webhooks
webhookService.triggerAsync('consent.updated', {
previous: existing,
current: updated
});
res.json({
success: true,
data: updated
});
} catch (error) {
next(error);
}
}
);
/**
* DELETE /v1/consent/:userId/:domainId
* Process a deletion request (GDPR Article 17)
*/
router.delete(
'/v1/consent/:userId/:domainId',
authenticate,
rateLimit({ windowMs: 60000, max: 10 }), // Low limit for deletions
async (req: Request, res: Response, next: NextFunction) => {
try {
const { userId, domainId } = req.params;
const { verificationToken } = req.body;
// Verify deletion request
const verified = await consentService.verifyDeletionRequest(
userId,
domainId,
verificationToken
);
if (!verified) {
return res.status(403).json({
success: false,
error: {
code: 'DELETION_NOT_VERIFIED',
message: 'Deletion request could not be verified'
}
});
}
// Soft delete (mark for deletion, actual deletion async)
const deletionJob = await consentService.scheduleConsentDeletion(
userId,
domainId,
{
requestedAt: new Date(),
scheduledFor: new Date(Date.now() + 24 * 60 * 60 * 1000), // 24hr grace period
clientId: req.auth.clientId
}
);
// Audit log
await auditService.log({
action: 'CONSENT_DELETION_REQUESTED',
resourceId: deletionJob.id,
resourceType: 'deletion_job',
clientId: req.auth.clientId,
userId,
details: { domainId, scheduledFor: deletionJob.scheduledFor },
timestamp: new Date()
});
res.status(202).json({
success: true,
data: {
deletionJobId: deletionJob.id,
status: 'scheduled',
scheduledFor: deletionJob.scheduledFor,
canCancelUntil: deletionJob.scheduledFor
}
});
} catch (error) {
next(error);
}
}
);
// ============================================
// CONSENT CHECK ENDPOINTS (HIGH PERFORMANCE)
// ============================================
/**
* GET /v1/consent/check
* Fast consent check for specific purposes
* Optimized for inline use in ad requests, analytics calls, etc.
*/
router.get(
'/v1/consent/check',
authenticate,
rateLimit({ windowMs: 1000, max: 100 }), // Very high limit
async (req: Request, res: Response, next: NextFunction) => {
try {
const { userId, domainId, purposes } = req.query;
if (!userId || !domainId || !purposes) {
return res.status(400).json({
success: false,
error: {
code: 'MISSING_PARAMETERS',
message: 'userId, domainId, and purposes are required'
}
});
}
const purposeList = (purposes as string).split(',');
// Fast path: check cache only
const result = await consentService.checkConsentFast(
userId as string,
domainId as string,
purposeList
);
// Return minimal response for speed
res.setHeader('Cache-Control', 'private, max-age=30');
res.json({
consented: result.consented,
purposes: result.purposeResults,
cached: result.fromCache,
checkedAt: new Date().toISOString()
});
} catch (error) {
// Fail closed: deny consent on error
res.json({
consented: false,
purposes: {},
error: true
});
}
}
);
/**
* POST /v1/consent/check/batch
* Check consent for multiple users at once
* Useful for backend batch processing
*/
router.post(
'/v1/consent/check/batch',
authenticate,
rateLimit({ windowMs: 60000, max: 100 }),
async (req: Request, res: Response, next: NextFunction) => {
try {
const { checks } = req.body;
if (!Array.isArray(checks) || checks.length > 100) {
return res.status(400).json({
success: false,
error: {
code: 'INVALID_BATCH',
message: 'Batch must be an array with max 100 items'
}
});
}
const results = await consentService.checkConsentBatch(checks);
res.json({
success: true,
results
});
} catch (error) {
next(error);
}
}
);
// ============================================
// CONFIGURATION ENDPOINTS
// ============================================
/**
* GET /v1/config/:domainId
* Get widget configuration for a domain
*/
router.get(
'/v1/config/:domainId',
authenticate,
rateLimit({ windowMs: 60000, max: 100 }),
async (req: Request, res: Response, next: NextFunction) => {
try {
const { domainId } = req.params;
const { locale = 'en' } = req.query;
const config = await consentService.getDomainConfig(
domainId,
locale as string
);
if (!config) {
return res.status(404).json({
success: false,
error: {
code: 'CONFIG_NOT_FOUND',
message: 'Configuration not found for this domain'
}
});
}
// Long cache for config
res.setHeader('Cache-Control', 'public, max-age=3600');
res.json({
success: true,
data: config
});
} catch (error) {
next(error);
}
}
);
/**
* GET /v1/tcf/gvl
* Get IAB Global Vendor List
*/
router.get(
'/v1/tcf/gvl',
rateLimit({ windowMs: 60000, max: 60 }),
async (req: Request, res: Response, next: NextFunction) => {
try {
const { version = 'latest' } = req.query;
const gvl = await consentService.getGlobalVendorList(version as string);
res.setHeader('Cache-Control', 'public, max-age=86400'); // Cache for 24hr
res.json(gvl);
} catch (error) {
next(error);
}
}
);
export default router;
```
## Database Schema Design
### PostgreSQL Schema for Consent Storage
```sql
-- consent-database-schema.sql
-- Enable required extensions
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "pgcrypto";
-- ============================================
-- CORE TABLES
-- ============================================
-- Domains/properties that collect consent
CREATE TABLE domains (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
client_id UUID NOT NULL REFERENCES clients(id),
domain_name VARCHAR(255) NOT NULL,
display_name VARCHAR(255),
-- Configuration
config JSONB NOT NULL DEFAULT '{}',
supported_frameworks VARCHAR(50)[] DEFAULT ARRAY['basic'],
default_language VARCHAR(10) DEFAULT 'en',
-- Status
is_active BOOLEAN DEFAULT true,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(client_id, domain_name)
);
CREATE INDEX idx_domains_client ON domains(client_id);
CREATE INDEX idx_domains_name ON domains(domain_name);
-- Consent purposes/categories
CREATE TABLE consent_purposes (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
domain_id UUID NOT NULL REFERENCES domains(id),
-- Purpose identification
purpose_key VARCHAR(100) NOT NULL, -- e.g., 'analytics', 'marketing'
external_id VARCHAR(100), -- TCF purpose ID, etc.
-- Display
name JSONB NOT NULL, -- {"en": "Analytics", "de": "Analyse"}
description JSONB NOT NULL,
-- Legal basis
legal_basis VARCHAR(50) NOT NULL, -- consent, legitimate_interest, contract, etc.
is_required BOOLEAN DEFAULT false,
-- Categorization
category VARCHAR(50), -- necessary, functional, analytics, marketing
sort_order INTEGER DEFAULT 0,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(domain_id, purpose_key)
);
CREATE INDEX idx_purposes_domain ON consent_purposes(domain_id);
-- Main consent records table (immutable, append-only)
CREATE TABLE consent_records (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
-- User identification (hashed for privacy)
user_id VARCHAR(255) NOT NULL,
user_id_hash VARCHAR(64) NOT NULL, -- SHA-256 for indexing
-- Domain
domain_id UUID NOT NULL REFERENCES domains(id),
-- Version chain
version INTEGER NOT NULL DEFAULT 1,
previous_version_id UUID REFERENCES consent_records(id),
-- Consent choices stored as JSONB for flexibility
choices JSONB NOT NULL,
/*
Example structure:
{
"purposes": {
"analytics": {"status": "granted", "timestamp": "..."},
"marketing": {"status": "denied", "timestamp": "..."}
},
"vendors": {
"vendor_123": {"status": "granted", "purposes": ["analytics"]}
}
}
*/
-- Framework-specific strings
tc_string TEXT, -- IAB TCF consent string
gpp_string TEXT, -- Global Privacy Platform string
us_privacy_string VARCHAR(4), -- CCPA/USP string
-- Metadata
metadata JSONB NOT NULL DEFAULT '{}',
/*
{
"source": "banner",
"user_agent": "...",
"ip_country": "DE",
"ip_region": "BY",
"consent_version": "1.2.0"
}
*/
-- Timestamps
created_at TIMESTAMPTZ DEFAULT NOW(),
expires_at TIMESTAMPTZ, -- Consent validity period
-- Soft delete support
deleted_at TIMESTAMPTZ,
deletion_reason VARCHAR(100)
);
-- Optimized indexes for common queries
CREATE INDEX idx_consent_user_domain ON consent_records(user_id_hash, domain_id)
WHERE deleted_at IS NULL;
CREATE INDEX idx_consent_domain_created ON consent_records(domain_id, created_at DESC);
CREATE INDEX idx_consent_expires ON consent_records(expires_at)
WHERE expires_at IS NOT NULL AND deleted_at IS NULL;
-- Partitioning for large-scale deployments
-- CREATE TABLE consent_records_2024 PARTITION OF consent_records
-- FOR VALUES FROM ('2024-01-01') TO ('2025-01-01');
-- ============================================
-- AUDIT & COMPLIANCE TABLES
-- ============================================
-- Comprehensive audit log
CREATE TABLE audit_logs (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
-- What happened
action VARCHAR(100) NOT NULL,
resource_type VARCHAR(50) NOT NULL,
resource_id UUID,
-- Who did it
client_id UUID REFERENCES clients(id),
user_id VARCHAR(255),
api_key_id UUID,
-- Details
details JSONB DEFAULT '{}',
-- Request context
ip_address INET,
user_agent TEXT,
request_id UUID,
-- Timing
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Time-series partitioning for audit logs
CREATE INDEX idx_audit_created ON audit_logs(created_at DESC);
CREATE INDEX idx_audit_resource ON audit_logs(resource_type, resource_id);
CREATE INDEX idx_audit_client ON audit_logs(client_id, created_at DESC);
-- Consent history view (aggregated for reporting)
CREATE TABLE consent_history_daily (
date DATE NOT NULL,
domain_id UUID NOT NULL REFERENCES domains(id),
-- Aggregated metrics
total_consents INTEGER DEFAULT 0,
total_updates INTEGER DEFAULT 0,
total_deletions INTEGER DEFAULT 0,
-- By source
consents_from_banner INTEGER DEFAULT 0,
consents_from_preference_center INTEGER DEFAULT 0,
consents_from_api INTEGER DEFAULT 0,
-- By choice
analytics_granted INTEGER DEFAULT 0,
analytics_denied INTEGER DEFAULT 0,
marketing_granted INTEGER DEFAULT 0,
marketing_denied INTEGER DEFAULT 0,
-- Rates
accept_all_rate DECIMAL(5,4),
reject_all_rate DECIMAL(5,4),
customize_rate DECIMAL(5,4),
PRIMARY KEY (date, domain_id)
);
-- ============================================
-- WEBHOOK & INTEGRATION TABLES
-- ============================================
CREATE TABLE webhooks (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
client_id UUID NOT NULL REFERENCES clients(id),
-- Configuration
url TEXT NOT NULL,
events VARCHAR(100)[] NOT NULL, -- ['consent.created', 'consent.updated']
secret_hash VARCHAR(64) NOT NULL, -- For HMAC signing
-- Status
is_active BOOLEAN DEFAULT true,
last_triggered_at TIMESTAMPTZ,
failure_count INTEGER DEFAULT 0,
-- Retry configuration
max_retries INTEGER DEFAULT 3,
retry_delay_seconds INTEGER DEFAULT 60,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE webhook_deliveries (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
webhook_id UUID NOT NULL REFERENCES webhooks(id),
-- Event details
event_type VARCHAR(100) NOT NULL,
payload JSONB NOT NULL,
-- Delivery status
status VARCHAR(20) DEFAULT 'pending', -- pending, delivered, failed
attempts INTEGER DEFAULT 0,
-- Response details
response_status INTEGER,
response_body TEXT,
response_time_ms INTEGER,
-- Timing
created_at TIMESTAMPTZ DEFAULT NOW(),
delivered_at TIMESTAMPTZ,
next_retry_at TIMESTAMPTZ
);
CREATE INDEX idx_webhook_deliveries_pending ON webhook_deliveries(next_retry_at)
WHERE status = 'pending';
```
## Caching Strategy for Sub-100ms Response Times
```typescript
// consent-caching-service.ts
import Redis from 'ioredis';
import { createHash } from 'crypto';
interface CacheConfig {
redis: {
host: string;
port: number;
cluster?: boolean;
sentinels?: { host: string; port: number }[];
};
ttl: {
consent: number; // Consent records: 5 minutes
config: number; // Domain config: 1 hour
gvl: number; // Vendor list: 24 hours
consentCheck: number; // Fast consent checks: 30 seconds
};
compression: boolean;
}
class ConsentCacheService {
private redis: Redis.Redis | Redis.Cluster;
private config: CacheConfig;
private localCache: Map;
constructor(config: CacheConfig) {
this.config = config;
this.localCache = new Map();
// Initialize Redis connection
if (config.redis.cluster) {
this.redis = new Redis.Cluster([
{ host: config.redis.host, port: config.redis.port }
]);
} else if (config.redis.sentinels) {
this.redis = new Redis({
sentinels: config.redis.sentinels,
name: 'consent-master'
});
} else {
this.redis = new Redis(config.redis);
}
// Start local cache cleanup
setInterval(() => this.cleanLocalCache(), 10000);
}
// ============================================
// CONSENT STATE CACHING
// ============================================
async getConsentState(
userId: string,
domainId: string
): Promise {
const key = this.consentKey(userId, domainId);
// L1: Check local in-memory cache
const local = this.getLocal(key);
if (local) return local;
// L2: Check Redis
const cached = await this.redis.get(key);
if (!cached) return null;
const consent = this.deserialize(cached);
// Populate L1 cache
this.setLocal(key, consent, 5000); // 5 second local TTL
return consent;
}
async setConsentState(
userId: string,
domainId: string,
consent: ConsentState
): Promise {
const key = this.consentKey(userId, domainId);
const serialized = this.serialize(consent);
// Set in Redis with TTL
await this.redis.setex(key, this.config.ttl.consent, serialized);
// Update local cache
this.setLocal(key, consent, 5000);
// Also cache the fast-check version
await this.cacheConsentCheck(userId, domainId, consent);
}
async invalidateConsentState(
userId: string,
domainId: string
): Promise {
const key = this.consentKey(userId, domainId);
const checkKey = this.consentCheckKey(userId, domainId);
// Delete from all cache layers
await this.redis.del(key, checkKey);
this.localCache.delete(key);
this.localCache.delete(checkKey);
}
// ============================================
// FAST CONSENT CHECK CACHING
// ============================================
/**
* Ultra-fast consent check optimized for ad requests
* Returns only the essential consent signals
*/
async getConsentCheckFast(
userId: string,
domainId: string,
purposes: string[]
): Promise {
const key = this.consentCheckKey(userId, domainId);
// L1: Local cache (fastest)
const local = this.getLocal(key);
if (local) {
return this.filterPurposes(local, purposes);
}
// L2: Redis
const cached = await this.redis.get(key);
if (!cached) return null;
const checkData = this.deserialize(cached);
this.setLocal(key, checkData, 2000); // Very short local TTL for fast checks
return this.filterPurposes(checkData, purposes);
}
private async cacheConsentCheck(
userId: string,
domainId: string,
consent: ConsentState
): Promise {
// Create minimal check data structure
const checkData: ConsentCheckData = {
purposes: {},
tcString: consent.frameworks?.tcf?.tcString,
updatedAt: consent.updatedAt
};
for (const choice of consent.choices) {
checkData.purposes[choice.purposeId] = choice.status === 'granted';
}
const key = this.consentCheckKey(userId, domainId);
await this.redis.setex(
key,
this.config.ttl.consentCheck,
this.serialize(checkData)
);
}
private filterPurposes(
checkData: ConsentCheckData,
purposes: string[]
): ConsentCheckResult {
const result: ConsentCheckResult = {
consented: true,
purposeResults: {},
fromCache: true
};
for (const purpose of purposes) {
const consented = checkData.purposes[purpose] ?? false;
result.purposeResults[purpose] = consented;
if (!consented) {
result.consented = false;
}
}
return result;
}
// ============================================
// DOMAIN CONFIG CACHING
// ============================================
async getDomainConfig(
domainId: string,
locale: string
): Promise {
const key = `config:${domainId}:${locale}`;
// Check local cache
const local = this.getLocal(key);
if (local) return local;
// Check Redis
const cached = await this.redis.get(key);
if (!cached) return null;
const config = this.deserialize(cached);
this.setLocal(key, config, 60000); // 1 minute local TTL
return config;
}
async setDomainConfig(
domainId: string,
locale: string,
config: DomainConfig
): Promise {
const key = `config:${domainId}:${locale}`;
await this.redis.setex(key, this.config.ttl.config, this.serialize(config));
this.setLocal(key, config, 60000);
}
// ============================================
// CACHE KEY GENERATION
// ============================================
private consentKey(userId: string, domainId: string): string {
// Hash user ID for privacy
const userHash = this.hashUserId(userId);
return `consent:${domainId}:${userHash}`;
}
private consentCheckKey(userId: string, domainId: string): string {
const userHash = this.hashUserId(userId);
return `check:${domainId}:${userHash}`;
}
private hashUserId(userId: string): string {
return createHash('sha256').update(userId).digest('hex').substring(0, 16);
}
// ============================================
// SERIALIZATION
// ============================================
private serialize(data: any): string {
const json = JSON.stringify(data);
if (this.config.compression && json.length > 1000) {
// Compress large payloads (implement with zlib if needed)
return json;
}
return json;
}
private deserialize(data: string): any {
return JSON.parse(data);
}
// ============================================
// LOCAL CACHE MANAGEMENT
// ============================================
private getLocal(key: string): any | null {
const entry = this.localCache.get(key);
if (!entry) return null;
if (Date.now() > entry.expires) {
this.localCache.delete(key);
return null;
}
return entry.data;
}
private setLocal(key: string, data: any, ttlMs: number): void {
this.localCache.set(key, {
data,
expires: Date.now() + ttlMs
});
}
private cleanLocalCache(): void {
const now = Date.now();
for (const [key, entry] of this.localCache) {
if (now > entry.expires) {
this.localCache.delete(key);
}
}
}
// ============================================
// CACHE WARMING
// ============================================
async warmCache(domainId: string): Promise {
// Fetch recent consent records and pre-populate cache
const recentConsents = await this.fetchRecentConsents(domainId, 1000);
const pipeline = this.redis.pipeline();
for (const consent of recentConsents) {
const key = this.consentKey(consent.userId, domainId);
pipeline.setex(key, this.config.ttl.consent, this.serialize(consent));
}
await pipeline.exec();
}
private async fetchRecentConsents(
domainId: string,
limit: number
): Promise {
// Implementation would query database
return [];
}
}
```
## Webhook Implementation for Real-Time Propagation
```typescript
// webhook-service.ts
import crypto from 'crypto';
import axios, { AxiosError } from 'axios';
import { Queue, Worker, Job } from 'bullmq';
interface WebhookConfig {
maxRetries: number;
retryDelays: number[]; // [60, 300, 900, 3600] seconds
timeout: number;
signatureHeader: string;
}
interface WebhookDelivery {
id: string;
webhookId: string;
eventType: string;
payload: any;
attempt: number;
status: 'pending' | 'delivered' | 'failed';
responseStatus?: number;
responseBody?: string;
error?: string;
}
class WebhookService {
private queue: Queue;
private worker: Worker;
private config: WebhookConfig;
constructor(config: WebhookConfig) {
this.config = config;
// Initialize BullMQ queue
this.queue = new Queue('webhooks', {
defaultJobOptions: {
attempts: config.maxRetries,
backoff: {
type: 'custom'
},
removeOnComplete: { age: 86400 }, // Keep for 24hr
removeOnFail: { age: 604800 } // Keep failed for 7 days
}
});
// Initialize worker
this.worker = new Worker('webhooks', this.processWebhook.bind(this), {
concurrency: 10,
limiter: {
max: 100,
duration: 1000
}
});
this.setupEventHandlers();
}
// ============================================
// WEBHOOK TRIGGERING
// ============================================
async triggerAsync(eventType: string, payload: any): Promise {
// Get all active webhooks for this event type
const webhooks = await this.getWebhooksForEvent(eventType);
for (const webhook of webhooks) {
await this.queue.add(
'deliver',
{
webhookId: webhook.id,
url: webhook.url,
secret: webhook.secret,
eventType,
payload,
attempt: 1
},
{
jobId: `${webhook.id}-${Date.now()}`,
priority: this.getPriority(eventType)
}
);
}
}
private getPriority(eventType: string): number {
// Higher priority for critical events
if (eventType.startsWith('consent.deleted')) return 1;
if (eventType.startsWith('consent.')) return 5;
return 10;
}
// ============================================
// WEBHOOK DELIVERY
// ============================================
private async processWebhook(job: Job): Promise {
const { webhookId, url, secret, eventType, payload, attempt } = job.data;
const deliveryId = `del_${crypto.randomUUID()}`;
const timestamp = Math.floor(Date.now() / 1000);
// Create signed payload
const signedPayload = this.createSignedPayload(
payload,
eventType,
timestamp,
deliveryId
);
// Generate signature
const signature = this.generateSignature(
JSON.stringify(signedPayload),
secret,
timestamp
);
try {
const startTime = Date.now();
const response = await axios.post(url, signedPayload, {
headers: {
'Content-Type': 'application/json',
[this.config.signatureHeader]: signature,
'X-Webhook-Delivery-Id': deliveryId,
'X-Webhook-Timestamp': timestamp.toString(),
'X-Webhook-Event': eventType
},
timeout: this.config.timeout,
validateStatus: () => true // Don't throw on non-2xx
});
const responseTime = Date.now() - startTime;
// Log delivery
await this.logDelivery({
id: deliveryId,
webhookId,
eventType,
payload: signedPayload,
attempt,
status: response.status >= 200 && response.status < 300 ? 'delivered' : 'failed',
responseStatus: response.status,
responseBody: typeof response.data === 'string'
? response.data.substring(0, 1000)
: JSON.stringify(response.data).substring(0, 1000),
responseTimeMs: responseTime
});
// Throw if not successful (triggers retry)
if (response.status < 200 || response.status >= 300) {
throw new Error(`Webhook returned status ${response.status}`);
}
// Update webhook health
await this.updateWebhookHealth(webhookId, true);
} catch (error) {
const axiosError = error as AxiosError;
await this.logDelivery({
id: deliveryId,
webhookId,
eventType,
payload: signedPayload,
attempt,
status: 'failed',
error: axiosError.message
});
// Update webhook health
await this.updateWebhookHealth(webhookId, false);
throw error; // Let BullMQ handle retry
}
}
// ============================================
// SIGNATURE GENERATION
// ============================================
private createSignedPayload(
payload: any,
eventType: string,
timestamp: number,
deliveryId: string
): WebhookPayload {
return {
id: deliveryId,
event: eventType,
created_at: new Date(timestamp * 1000).toISOString(),
data: payload,
api_version: '2024-01-01'
};
}
private generateSignature(
payload: string,
secret: string,
timestamp: number
): string {
// Format: t=timestamp,v1=signature
const signedPayload = `${timestamp}.${payload}`;
const signature = crypto
.createHmac('sha256', secret)
.update(signedPayload)
.digest('hex');
return `t=${timestamp},v1=${signature}`;
}
// ============================================
// WEBHOOK VERIFICATION (for recipients)
// ============================================
static verifyWebhookSignature(
payload: string,
signature: string,
secret: string,
tolerance: number = 300 // 5 minutes
): boolean {
const parts = signature.split(',').reduce((acc, part) => {
const [key, value] = part.split('=');
acc[key] = value;
return acc;
}, {} as Record);
const timestamp = parseInt(parts.t, 10);
const providedSignature = parts.v1;
// Check timestamp tolerance
const now = Math.floor(Date.now() / 1000);
if (Math.abs(now - timestamp) > tolerance) {
throw new Error('Webhook timestamp outside tolerance');
}
// Verify signature
const signedPayload = `${timestamp}.${payload}`;
const expectedSignature = crypto
.createHmac('sha256', secret)
.update(signedPayload)
.digest('hex');
return crypto.timingSafeEqual(
Buffer.from(providedSignature),
Buffer.from(expectedSignature)
);
}
// ============================================
// WEBHOOK HEALTH MANAGEMENT
// ============================================
private async updateWebhookHealth(
webhookId: string,
success: boolean
): Promise {
if (success) {
// Reset failure count
await this.db.query(`
UPDATE webhooks
SET
failure_count = 0,
last_triggered_at = NOW()
WHERE id = $1
`, [webhookId]);
} else {
// Increment failure count
const result = await this.db.query(`
UPDATE webhooks
SET
failure_count = failure_count + 1,
last_triggered_at = NOW()
WHERE id = $1
RETURNING failure_count
`, [webhookId]);
// Auto-disable after too many failures
if (result.rows[0].failure_count >= 10) {
await this.disableWebhook(webhookId, 'Too many consecutive failures');
}
}
}
private async disableWebhook(
webhookId: string,
reason: string
): Promise {
await this.db.query(`
UPDATE webhooks
SET
is_active = false,
disabled_reason = $2,
disabled_at = NOW()
WHERE id = $1
`, [webhookId, reason]);
// Notify webhook owner
await this.notifyWebhookDisabled(webhookId, reason);
}
}
// ============================================
// WEBHOOK EVENT TYPES
// ============================================
interface WebhookEvents {
'consent.created': {
consent_id: string;
user_id: string;
domain_id: string;
choices: ConsentChoice[];
tc_string?: string;
created_at: string;
};
'consent.updated': {
consent_id: string;
user_id: string;
domain_id: string;
previous_choices: ConsentChoice[];
new_choices: ConsentChoice[];
changed_purposes: string[];
updated_at: string;
};
'consent.deleted': {
user_id: string;
domain_id: string;
deleted_at: string;
deletion_type: 'user_request' | 'retention_policy' | 'client_request';
};
'consent.expired': {
consent_id: string;
user_id: string;
domain_id: string;
expired_at: string;
original_created_at: string;
};
}
```
## FAQ: Consent Management API Design
### How do I handle API versioning as privacy regulations evolve?
Use URL-based versioning (e.g., `/v1/consent`, `/v2/consent`) for major changes and header-based versioning for minor updates. Maintain backward compatibility for at least 12 months. When GDPR or similar regulations change, create a new API version rather than modifying existing endpoints. Use feature flags to gradually migrate clients. Document sunset dates and provide migration guides.
### What's the best authentication approach for consent APIs?
For server-to-server communication, use API keys with HMAC signature verification. For client-side widgets, use domain-restricted API keys that only work from registered domains. Implement JWT tokens for authenticated user sessions. Rate limit aggressively based on API key. Never expose server-side API keys in client code—use a proxy endpoint instead.
### How do I achieve sub-100ms response times for consent checks?
Layer your caching: L1 (in-process memory, 1-5 second TTL), L2 (Redis, 30-60 second TTL), L3 (database). Use read replicas for database queries. Pre-compute consent check results into a minimal data structure optimized for fast lookup. Use connection pooling for both Redis and database. Place edge caches in multiple geographic regions. Return cached results even if slightly stale—eventual consistency is acceptable for consent checks.
### How do I handle consent sync across multiple domains?
For same-company multi-domain consent sharing, implement a consent synchronization service that propagates consent changes via webhooks or event streaming. Use a canonical user identifier that works across domains. Be careful with cross-domain consent sharing—GDPR generally requires separate consent per data controller. Document the legal basis for any cross-domain consent sharing.
### What audit data should I capture for compliance?
Capture: full consent record, timestamp, source (banner/preference center/API), user agent, IP geolocation (country/region only), consent version, any framework-specific strings (TCF, GPP), request ID for traceability. Do NOT capture: full IP addresses (hash or truncate), device fingerprints, or any data that could identify the user beyond their consent identifier. Retain audit data for at least the consent validity period plus one year.
### How do I handle high-volume consent operations during traffic spikes?
Implement queue-based processing for writes while serving reads from cache. Use database connection pooling with queue overflow. Implement circuit breakers that fail gracefully (default to requiring consent). Auto-scale based on queue depth, not just CPU. Pre-warm caches before anticipated traffic spikes. Consider write-behind caching for consent updates—acknowledge immediately, persist asynchronously with guaranteed delivery.
## Building for the Future of Privacy Regulation
The consent management API landscape will continue evolving as new regulations emerge and existing ones are updated. Design your APIs with extensibility in mind: use flexible JSONB fields for framework-specific data, maintain loose coupling between your consent storage and framework-specific logic, and build abstraction layers that can accommodate new consent frameworks without database migrations.
The patterns shown here—immutable consent records, comprehensive audit logging, webhook-based propagation, and aggressive caching—provide a foundation that can scale from thousands to billions of consent records while maintaining the performance and reliability that modern privacy compliance demands.