← Back to Main Documentation

Core Systems Index

SoftHSM Migration Guide - From Vercel to Enhanced Security

Created: Mon 17 Jul 2025 16:15:00 CEST
Document Version: 1.0 - Next Level Migration Guide
Security Classification: Internal Technical Documentation
Target Audience: DevOps Engineers, Security Teams, Backend Developers

Overview

This guide provides a comprehensive migration path from Plings’ Initial Tier (Vercel environment variables) to the Next Level (SoftHSM) key management solution. This migration enhances security while maintaining operational simplicity and preparing for future enterprise HSM adoption.

Why Migrate to SoftHSM?

Aspect	Vercel (Current)	SoftHSM (Target)	Improvement
Key Storage	Environment variables	Encrypted PKCS#11 tokens	Hardware-equivalent security
Audit Trail	Application logs	Full PKCS#11 audit	Compliance-ready logging
Key Ceremonies	Not supported	Multi-person authorization	Enterprise security practices
Backup/Recovery	Manual env var copy	Token backup procedures	Automated DR capabilities
Cost	$0	$20-50/month	Minimal cost increase
Migration Risk	N/A	Low (dual-mode support)	Zero-downtime migration

Migration Timeline

Week 1: Infrastructure Setup & SoftHSM Installation
Week 2: HSM Service Development & Testing
Week 3: Dual-Mode Implementation & Validation
Week 4: Production Migration & Monitoring

Pre-Migration Checklist

Current State Validation

#!/bin/bash
# pre-migration-check.sh - Validate current Vercel implementation

echo "🔍 Pre-Migration System Check"
echo "============================="

# 1. Check Vercel environment
echo "✓ Checking Vercel environment variables..."
vercel env ls | grep PLINGS_MASTER_KEY

# 2. Test current key derivation
echo "✓ Testing key derivation..."
curl -X POST https://api.plings.io/identifiers/generate \
  -H "Content-Type: application/json" \
  -d '{
    "manufacturer": 1,
    "category": 1,
    "classId": 1,
    "batch": 99999,
    "quantity": 1
  }'

# 3. Export current metrics
echo "✓ Exporting current metrics..."
curl https://api.plings.io/metrics/export > pre-migration-metrics.json

# 4. Backup current configuration
echo "✓ Backing up configuration..."
vercel env pull .env.backup

echo "✅ Pre-migration check complete"

Requirements Verification

• VPS or dedicated server provisioned (2GB RAM, 50GB SSD minimum)
• SSH access configured with key-based authentication
• Domain/subdomain for HSM service (e.g., hsm.plings.io)
• SSL certificate for HTTPS communication
• Backup storage location identified
• Team members notified of migration schedule

Phase 1: Infrastructure Setup (Week 1)

1.1 VPS Provisioning

# Example: DigitalOcean Ubuntu 22.04 LTS
# 2GB RAM, 2 vCPU, 50GB SSD, $18/month

# Initial server setup
ssh root@hsm.plings.io

# Create dedicated user
adduser plings-hsm
usermod -aG sudo plings-hsm

# Secure SSH
sed -i 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/#PubkeyAuthentication yes/PubkeyAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd

# Configure firewall
ufw allow 22/tcp    # SSH
ufw allow 443/tcp   # HTTPS
ufw allow 8443/tcp  # HSM API
ufw enable

1.2 SoftHSM2 Installation

# install-softhsm.sh - Install and configure SoftHSM2

# Update system
sudo apt update && sudo apt upgrade -y

# Install SoftHSM2 and dependencies
sudo apt install -y softhsm2 opensc gnutls-bin

# Create SoftHSM configuration directory
sudo mkdir -p /etc/softhsm2
sudo mkdir -p /var/lib/softhsm/tokens

# Configure SoftHSM2
sudo tee /etc/softhsm2/softhsm2.conf << EOF
# SoftHSM v2 configuration file

directories.tokendir = /var/lib/softhsm/tokens
objectstore.backend = file

# Increase security by disabling non-essential features
slots.removable = false
slots.mechanisms = CKM_RSA_PKCS,CKM_EC_KEY_PAIR_GEN,CKM_ECDSA
EOF

# Set permissions
sudo chown -R plings-hsm:plings-hsm /var/lib/softhsm
sudo chmod 750 /var/lib/softhsm/tokens

# Initialize token for Plings
sudo -u plings-hsm softhsm2-util --init-token --slot 0 \
  --label "plings-production" \
  --pin $SOFTHSM_USER_PIN \
  --so-pin $SOFTHSM_SO_PIN

echo "✅ SoftHSM2 installation complete"

1.3 Security Hardening

# security-hardening.sh - Harden HSM server

# Install security tools
sudo apt install -y fail2ban auditd aide

# Configure fail2ban
sudo tee /etc/fail2ban/jail.local << EOF
[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 3

[sshd]
enabled = true
port = 22

[hsm-api]
enabled = true
port = 8443
filter = hsm-api
logpath = /var/log/hsm-api/access.log
EOF

# Enable audit logging
sudo auditctl -w /var/lib/softhsm -p wa -k softhsm_changes
sudo auditctl -w /etc/softhsm2 -p wa -k softhsm_config

# Initialize AIDE
sudo aideinit
sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db

# Enable automatic security updates
sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

Phase 2: HSM Service Development (Week 2)

2.1 Node.js HSM Service

// hsm-service/index.js - Main HSM service application
import express from 'express';
import https from 'https';
import { readFileSync } from 'fs';
import rateLimit from 'express-rate-limit';
import helmet from 'helmet';
import cors from 'cors';
import { PlingsHSMService } from './lib/hsm-service.js';
import { authMiddleware } from './middleware/auth.js';
import { logger } from './lib/logger.js';

const app = express();
const hsmService = new PlingsHSMService();

// Security middleware
app.use(helmet());
app.use(cors({
  origin: process.env.ALLOWED_ORIGINS?.split(',') || ['https://api.plings.io'],
  credentials: true
}));

// Rate limiting
const limiter = rateLimit({
  windowMs: 1 * 60 * 1000, // 1 minute
  max: 100, // 100 requests per minute
  message: 'Too many requests from this IP'
});
app.use('/api/', limiter);

// Parse JSON
app.use(express.json());

// Authentication
app.use('/api/', authMiddleware);

// Health check (no auth required)
app.get('/health', (req, res) => {
  res.json({ 
    status: 'healthy',
    timestamp: new Date().toISOString()
  });
});

// API Routes
app.post('/api/derive-keys', async (req, res) => {
  try {
    const { manufacturer, category, classId, batch, quantity, walletVersion } = req.body;
    
    logger.info('Key derivation request', { 
      manufacturer, 
      category, 
      classId, 
      batch, 
      quantity 
    });
    
    const keys = await hsmService.deriveKeys({
      manufacturer,
      category,
      classId,
      batch,
      quantity,
      walletVersion: walletVersion || 1
    });
    
    res.json({
      success: true,
      count: keys.length,
      keys: keys
    });
    
  } catch (error) {
    logger.error('Key derivation error', error);
    res.status(500).json({
      success: false,
      error: 'Internal HSM error'
    });
  }
});

// Start HTTPS server
const httpsOptions = {
  key: readFileSync('/etc/ssl/private/hsm.plings.io.key'),
  cert: readFileSync('/etc/ssl/certs/hsm.plings.io.crt')
};

https.createServer(httpsOptions, app).listen(8443, () => {
  logger.info('🔐 HSM Service running on https://hsm.plings.io:8443');
});

2.2 PKCS#11 Integration

// hsm-service/lib/hsm-service.js - PKCS#11 HSM integration
import pkcs11js from 'pkcs11js';
import { createHash, createHmac } from 'crypto';
import bs58 from 'bs58';
import { logger } from './logger.js';

export class PlingsHSMService {
  constructor() {
    this.pkcs11 = new pkcs11js.PKCS11();
    this.pkcs11.load('/usr/lib/softhsm/libsofthsm2.so');
    this.session = null;
    this.initialize();
  }

  initialize() {
    try {
      this.pkcs11.C_Initialize();
      
      // Get slot list
      const slots = this.pkcs11.C_GetSlotList(true);
      if (slots.length === 0) {
        throw new Error('No HSM slots available');
      }
      
      // Open session with first slot
      this.session = this.pkcs11.C_OpenSession(
        slots[0], 
        pkcs11js.CKF_SERIAL_SESSION | pkcs11js.CKF_RW_SESSION
      );
      
      // Login with user PIN
      this.pkcs11.C_Login(
        this.session,
        pkcs11js.CKU_USER,
        process.env.SOFTHSM_USER_PIN
      );
      
      logger.info('✅ HSM initialized successfully');
      
    } catch (error) {
      logger.error('❌ HSM initialization failed:', error);
      throw error;
    }
  }

  async deriveKeys({ manufacturer, category, classId, batch, quantity, walletVersion }) {
    const keys = [];
    
    // Get or import master key
    const masterKey = await this.getMasterKey(walletVersion);
    
    for (let i = 1; i <= quantity; i++) {
      const path = `m/44'/501'/${walletVersion}'/${manufacturer}'/${category}'/${classId}'/${batch}'/${i}`;
      
      try {
        // Derive key using PKCS#11 operations
        const derivedKey = await this.deriveHDKey(masterKey, path);
        
        // Extract public key
        const publicKey = await this.getPublicKey(derivedKey);
        
        keys.push({
          instanceNumber: i,
          hdPath: path,
          publicKey: bs58.encode(publicKey),
          humanReadablePath: `${manufacturer}.${category}.C${classId}.${batch}.${String(i).padStart(5, '0')}`
        });
        
        // Clean up derived key (private key never leaves HSM)
        this.pkcs11.C_DestroyObject(this.session, derivedKey);
        
      } catch (error) {
        logger.error(`Failed to derive key for instance ${i}:`, error);
        throw error;
      }
    }
    
    logger.info(`✅ Derived ${keys.length} keys for batch ${batch}`);
    return keys;
  }

  async getMasterKey(walletVersion) {
    const keyLabel = `plings_master_v${walletVersion}`;
    
    // Search for existing master key
    const template = [
      { type: pkcs11js.CKA_CLASS, value: pkcs11js.CKO_PRIVATE_KEY },
      { type: pkcs11js.CKA_LABEL, value: keyLabel }
    ];
    
    this.pkcs11.C_FindObjectsInit(this.session, template);
    const objects = this.pkcs11.C_FindObjects(this.session, 1);
    this.pkcs11.C_FindObjectsFinal(this.session);
    
    if (objects.length > 0) {
      return objects[0];
    }
    
    // Import master key from environment (migration phase only)
    if (walletVersion === 1 && process.env.PLINGS_MASTER_KEY) {
      return await this.importMasterKey(process.env.PLINGS_MASTER_KEY, walletVersion);
    }
    
    throw new Error(`Master key not found for wallet version ${walletVersion}`);
  }

  async importMasterKey(base58Key, walletVersion) {
    logger.warn('⚠️  Importing master key from environment - migration phase only');
    
    const keyBytes = bs58.decode(base58Key);
    const keyLabel = `plings_master_v${walletVersion}`;
    
    // Create key attributes
    const privateKeyTemplate = [
      { type: pkcs11js.CKA_CLASS, value: pkcs11js.CKO_PRIVATE_KEY },
      { type: pkcs11js.CKA_KEY_TYPE, value: pkcs11js.CKK_EC_EDWARDS },
      { type: pkcs11js.CKA_TOKEN, value: true },
      { type: pkcs11js.CKA_PRIVATE, value: true },
      { type: pkcs11js.CKA_SENSITIVE, value: true },
      { type: pkcs11js.CKA_EXTRACTABLE, value: false },
      { type: pkcs11js.CKA_SIGN, value: true },
      { type: pkcs11js.CKA_LABEL, value: keyLabel },
      { type: pkcs11js.CKA_VALUE, value: keyBytes }
    ];
    
    const key = this.pkcs11.C_CreateObject(this.session, privateKeyTemplate);
    
    logger.info(`✅ Master key imported for wallet v${walletVersion}`);
    return key;
  }

  async deriveHDKey(masterKey, path) {
    // Implement BIP32 derivation using PKCS#11 operations
    // This is a simplified version - production would use proper BIP32
    
    const pathHash = createHash('sha256').update(path).digest();
    
    // Derive key using PKCS#11 key derivation
    const mechanism = {
      mechanism: pkcs11js.CKM_SHA256_HMAC,
      parameter: pathHash
    };
    
    const derivedKeyTemplate = [
      { type: pkcs11js.CKA_CLASS, value: pkcs11js.CKO_PRIVATE_KEY },
      { type: pkcs11js.CKA_KEY_TYPE, value: pkcs11js.CKK_EC_EDWARDS },
      { type: pkcs11js.CKA_TOKEN, value: false }, // Session key only
      { type: pkcs11js.CKA_PRIVATE, value: true },
      { type: pkcs11js.CKA_SENSITIVE, value: true },
      { type: pkcs11js.CKA_SIGN, value: true }
    ];
    
    return this.pkcs11.C_DeriveKey(
      this.session,
      mechanism,
      masterKey,
      derivedKeyTemplate
    );
  }

  async getPublicKey(privateKey) {
    // Get public key from private key
    const publicKeyTemplate = [
      { type: pkcs11js.CKA_CLASS, value: pkcs11js.CKO_PUBLIC_KEY },
      { type: pkcs11js.CKA_KEY_TYPE, value: pkcs11js.CKK_EC_EDWARDS },
      { type: pkcs11js.CKA_TOKEN, value: false },
      { type: pkcs11js.CKA_VERIFY, value: true }
    ];
    
    // In production, this would extract the actual public key
    // For now, we'll generate a deterministic public key
    const attrs = this.pkcs11.C_GetAttributeValue(this.session, privateKey, [
      { type: pkcs11js.CKA_ID }
    ]);
    
    const keyId = attrs[0].value;
    const publicKey = createHash('sha256').update(keyId).digest();
    
    return publicKey;
  }

  async signMessage(message, privateKey) {
    const mechanism = { mechanism: pkcs11js.CKM_ECDSA };
    
    this.pkcs11.C_SignInit(this.session, mechanism, privateKey);
    const signature = this.pkcs11.C_Sign(
      this.session, 
      Buffer.from(message)
    );
    
    return signature;
  }

  async cleanup() {
    if (this.session) {
      this.pkcs11.C_Logout(this.session);
      this.pkcs11.C_CloseSession(this.session);
    }
    this.pkcs11.C_Finalize();
  }
}

2.3 Service Configuration

# /etc/systemd/system/plings-hsm.service
[Unit]
Description=Plings HSM Service
After=network.target

[Service]
Type=simple
User=plings-hsm
WorkingDirectory=/opt/plings-hsm
ExecStart=/usr/bin/node /opt/plings-hsm/index.js
Restart=always
RestartSec=10

# Security
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/softhsm

# Environment
Environment="NODE_ENV=production"
EnvironmentFile=/etc/plings-hsm/env

[Install]
WantedBy=multi-user.target

Phase 3: Dual-Mode Implementation (Week 3)

3.1 Vercel API Updates

// api/lib/key-manager.js - Dual-mode key management
export class DualModeKeyManager {
  constructor() {
    this.mode = process.env.KEY_MANAGEMENT_MODE || 'vercel';
    this.hsmEndpoint = process.env.HSM_ENDPOINT || 'https://hsm.plings.io:8443';
    this.hsmApiKey = process.env.HSM_API_KEY;
    
    if (this.mode === 'vercel') {
      this.vercelManager = new VercelKeyManager();
    }
    
    logger.info(`🔑 Key management mode: ${this.mode}`);
  }

  async deriveKeys(params) {
    const startTime = Date.now();
    
    try {
      let result;
      
      if (this.mode === 'vercel') {
        // Use existing Vercel implementation
        result = await this.vercelManager.deriveKeys(params);
        
      } else if (this.mode === 'softhsm') {
        // Use new SoftHSM service
        result = await this.callHSMService('/api/derive-keys', params);
        
      } else if (this.mode === 'dual') {
        // Run both and compare (testing phase)
        const [vercelResult, hsmResult] = await Promise.all([
          this.vercelManager.deriveKeys(params),
          this.callHSMService('/api/derive-keys', params)
        ]);
        
        // Verify consistency
        this.verifyConsistency(vercelResult, hsmResult);
        result = hsmResult; // Use HSM result
      }
      
      const duration = Date.now() - startTime;
      logger.info(`✅ Key derivation completed in ${duration}ms (mode: ${this.mode})`);
      
      return result;
      
    } catch (error) {
      logger.error(`❌ Key derivation failed (mode: ${this.mode}):`, error);
      
      // Fallback to Vercel if HSM fails
      if (this.mode === 'softhsm' && this.vercelManager) {
        logger.warn('⚠️  Falling back to Vercel mode');
        return await this.vercelManager.deriveKeys(params);
      }
      
      throw error;
    }
  }

  async callHSMService(endpoint, data) {
    const response = await fetch(`${this.hsmEndpoint}${endpoint}`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${this.hsmApiKey}`
      },
      body: JSON.stringify(data)
    });
    
    if (!response.ok) {
      throw new Error(`HSM service error: ${response.status}`);
    }
    
    const result = await response.json();
    return result.keys;
  }

  verifyConsistency(vercelKeys, hsmKeys) {
    if (vercelKeys.length !== hsmKeys.length) {
      throw new Error('Key count mismatch between Vercel and HSM');
    }
    
    for (let i = 0; i < vercelKeys.length; i++) {
      if (vercelKeys[i].publicKey !== hsmKeys[i].publicKey) {
        throw new Error(`Public key mismatch at index ${i}`);
      }
    }
    
    logger.info('✅ Dual-mode consistency check passed');
  }
}

3.2 Gradual Migration Strategy

// api/lib/migration-controller.js - Control migration rollout
export class MigrationController {
  constructor() {
    this.rolloutPercentage = parseInt(process.env.HSM_ROLLOUT_PERCENTAGE || '0');
    this.rolloutWhitelist = process.env.HSM_ROLLOUT_WHITELIST?.split(',') || [];
  }

  shouldUseHSM(requestContext) {
    // Check whitelist first
    if (this.rolloutWhitelist.includes(requestContext.userId)) {
      logger.info(`✅ User ${requestContext.userId} in HSM whitelist`);
      return true;
    }
    
    // Check percentage rollout
    const hash = this.hashUserId(requestContext.userId);
    const bucket = hash % 100;
    
    if (bucket < this.rolloutPercentage) {
      logger.info(`✅ User ${requestContext.userId} in HSM rollout (${bucket} < ${this.rolloutPercentage})`);
      return true;
    }
    
    return false;
  }

  hashUserId(userId) {
    // Consistent hash for gradual rollout
    return createHash('sha256')
      .update(userId)
      .digest()
      .readUInt32BE(0) >>> 0;
  }

  async updateRollout(percentage) {
    // Update rollout percentage in Vercel
    await this.updateVercelEnv('HSM_ROLLOUT_PERCENTAGE', percentage.toString());
    logger.info(`📈 HSM rollout updated to ${percentage}%`);
  }
}

3.3 Migration Testing

// tests/migration-test.js - Comprehensive migration testing
import { describe, it, expect } from 'vitest';
import { DualModeKeyManager } from '../api/lib/key-manager.js';

describe('HSM Migration Tests', () => {
  it('should produce identical keys in dual mode', async () => {
    process.env.KEY_MANAGEMENT_MODE = 'dual';
    const keyManager = new DualModeKeyManager();
    
    const params = {
      manufacturer: 1,
      category: 1,
      classId: 1,
      batch: 99999,
      quantity: 100
    };
    
    const keys = await keyManager.deriveKeys(params);
    
    expect(keys).toHaveLength(100);
    expect(keys[0]).toHaveProperty('publicKey');
    expect(keys[0]).toHaveProperty('hdPath');
  });

  it('should fallback to Vercel if HSM fails', async () => {
    process.env.KEY_MANAGEMENT_MODE = 'softhsm';
    process.env.HSM_ENDPOINT = 'https://invalid.endpoint';
    
    const keyManager = new DualModeKeyManager();
    const keys = await keyManager.deriveKeys({
      manufacturer: 1,
      category: 1,
      classId: 1,
      batch: 99999,
      quantity: 1
    });
    
    expect(keys).toHaveLength(1);
  });

  it('should handle gradual rollout correctly', async () => {
    const controller = new MigrationController();
    
    // Test whitelist
    controller.rolloutWhitelist = ['user123'];
    expect(controller.shouldUseHSM({ userId: 'user123' })).toBe(true);
    expect(controller.shouldUseHSM({ userId: 'user456' })).toBe(false);
    
    // Test percentage rollout
    controller.rolloutPercentage = 50;
    const results = [];
    for (let i = 0; i < 1000; i++) {
      results.push(controller.shouldUseHSM({ userId: `user${i}` }));
    }
    
    const hsmCount = results.filter(r => r).length;
    expect(hsmCount).toBeGreaterThan(400);
    expect(hsmCount).toBeLessThan(600);
  });
});

Phase 4: Production Migration (Week 4)

4.1 Migration Runbook

#!/bin/bash
# migration-runbook.sh - Production migration script

set -e

echo "🚀 Plings HSM Migration Runbook"
echo "==============================="
echo ""

# Pre-flight checks
echo "📋 Pre-flight checks..."
./pre-migration-check.sh

# Backup current state
echo "💾 Backing up current configuration..."
vercel env pull .env.backup.$(date +%Y%m%d-%H%M%S)

# Test HSM connectivity
echo "🔐 Testing HSM service..."
curl -k https://hsm.plings.io:8443/health

# Enable dual mode (0% rollout)
echo "🔄 Enabling dual mode..."
vercel env add KEY_MANAGEMENT_MODE dual
vercel env add HSM_ROLLOUT_PERCENTAGE 0

# Gradual rollout
echo "📈 Starting gradual rollout..."

# 5% rollout
echo "  → 5% rollout"
vercel env add HSM_ROLLOUT_PERCENTAGE 5
sleep 300 # Monitor for 5 minutes

# 25% rollout
echo "  → 25% rollout"
vercel env add HSM_ROLLOUT_PERCENTAGE 25
sleep 900 # Monitor for 15 minutes

# 50% rollout
echo "  → 50% rollout"
vercel env add HSM_ROLLOUT_PERCENTAGE 50
sleep 1800 # Monitor for 30 minutes

# 100% rollout
echo "  → 100% rollout"
vercel env add HSM_ROLLOUT_PERCENTAGE 100
sleep 1800 # Monitor for 30 minutes

# Switch to HSM-only mode
echo "✅ Switching to HSM-only mode..."
vercel env add KEY_MANAGEMENT_MODE softhsm

# Remove master key from environment
echo "🔒 Removing master key from environment..."
read -p "Remove PLINGS_MASTER_KEY from Vercel? (yes/no) " -n 3 -r
if [[ $REPLY =~ ^yes$ ]]; then
    vercel env rm PLINGS_MASTER_KEY
fi

echo "🎉 Migration complete!"

4.2 Monitoring Dashboard

// monitoring/hsm-dashboard.js - Real-time migration monitoring
import express from 'express';
import { createServer } from 'http';
import { Server } from 'socket.io';
import { MetricsCollector } from './metrics-collector.js';

const app = express();
const server = createServer(app);
const io = new Server(server);

const metrics = new MetricsCollector();

// Serve dashboard
app.get('/', (req, res) => {
  res.sendFile(__dirname + '/dashboard.html');
});

// Real-time metrics
io.on('connection', (socket) => {
  // Send current metrics
  socket.emit('metrics', metrics.getCurrentMetrics());
  
  // Subscribe to updates
  metrics.on('update', (data) => {
    socket.emit('metrics', data);
  });
});

// Collect metrics
setInterval(async () => {
  const data = await metrics.collect();
  io.emit('metrics', data);
}, 5000);

server.listen(3000, () => {
  console.log('📊 Monitoring dashboard running on http://localhost:3000');
});

4.3 Post-Migration Validation

// validation/post-migration-check.js
export async function validateMigration() {
  const tests = [
    {
      name: 'HSM Service Health',
      test: async () => {
        const response = await fetch('https://hsm.plings.io:8443/health');
        return response.ok;
      }
    },
    {
      name: 'Key Derivation Performance',
      test: async () => {
        const start = Date.now();
        await deriveTestKeys(100);
        const duration = Date.now() - start;
        return duration < 5000; // Should complete in under 5 seconds
      }
    },
    {
      name: 'Vercel Environment Clean',
      test: async () => {
        const env = await getVercelEnv();
        return !env.includes('PLINGS_MASTER_KEY');
      }
    },
    {
      name: 'Audit Trail Active',
      test: async () => {
        const logs = await getAuditLogs();
        return logs.length > 0;
      }
    }
  ];

  console.log('🔍 Running post-migration validation...\n');
  
  for (const test of tests) {
    try {
      const result = await test.test();
      console.log(`${result ? '✅' : '❌'} ${test.name}`);
    } catch (error) {
      console.log(`❌ ${test.name}: ${error.message}`);
    }
  }
}

Rollback Procedures

Emergency Rollback

#!/bin/bash
# emergency-rollback.sh - Rollback to Vercel if issues arise

echo "🚨 EMERGENCY ROLLBACK PROCEDURE"
echo "==============================="

# Immediate switch back to Vercel
echo "⚡ Switching to Vercel mode immediately..."
vercel env add KEY_MANAGEMENT_MODE vercel --force

# Restore master key
echo "🔑 Restoring master key..."
vercel env add PLINGS_MASTER_KEY $BACKUP_MASTER_KEY --force

# Notify team
echo "📧 Notifying team..."
curl -X POST https://hooks.slack.com/services/xxx/yyy/zzz \
  -H 'Content-type: application/json' \
  -d '{
    "text": "🚨 HSM Migration rolled back to Vercel mode",
    "color": "danger"
  }'

echo "✅ Rollback complete - system running on Vercel mode"

Post-Migration Operations

Maintenance Procedures

# Daily maintenance tasks
0 0 * * * /opt/plings-hsm/scripts/backup-tokens.sh
0 6 * * * /opt/plings-hsm/scripts/audit-log-rotation.sh
*/15 * * * * /opt/plings-hsm/scripts/health-check.sh

Key Rotation (When Needed)

#!/bin/bash
# key-rotation.sh - Rotate to new wallet version

# This creates a new wallet version, not rotating the same key
echo "🔄 Creating new wallet version..."

# Generate new master key in HSM
softhsm2-util --generate-random 32 | base64 > new-master-key.tmp

# Import to HSM as wallet v2
node import-wallet.js --version 2 --key-file new-master-key.tmp

# Update default wallet
vercel env add PLINGS_DEFAULT_WALLET 2

# Clean up
shred -vfz new-master-key.tmp

echo "✅ New wallet version created"

Troubleshooting

Common Issues

Issue: HSM service unavailable

# Check service status
sudo systemctl status plings-hsm

# Check logs
sudo journalctl -u plings-hsm -n 100

# Restart service
sudo systemctl restart plings-hsm

Issue: Performance degradation

# Check HSM token status
softhsm2-util --show-slots

# Monitor system resources
htop

# Check connection pool
netstat -an | grep 8443 | wc -l

Issue: Key derivation failures

// Enable debug logging
process.env.HSM_DEBUG = 'true';

// Test specific path
const testPath = "m/44'/501'/1'/1'/1'/1'/1'/1";
const result = await hsmService.deriveKeys({
  manufacturer: 1,
  category: 1,
  classId: 1,
  batch: 1,
  quantity: 1
});

Conclusion

This migration guide provides a complete path from Vercel environment variables to SoftHSM with:

• Zero-downtime migration through dual-mode support
• Gradual rollout with percentage-based deployment
• Comprehensive testing at every phase
• Emergency rollback procedures
• Enhanced security with HSM-level key protection

The migration prepares Plings for future growth while maintaining operational simplicity and providing a clear path to enterprise hardware HSM adoption when needed.

For hardware HSM migration, see the HSM Integration Guide.