Deployment Runbook

This document outlines the procedures for deploying updates to the Lightning Enable API, including pre-deployment checks, deployment steps, verification, and rollback procedures.

Deployment Overview

Lightning Enable API is deployed to Azure App Service using GitHub Actions CI/CD.

Environment	URL	Branch	Auto-Deploy
Production	api.lightningenable.com	main	Yes
Staging	staging-lightning-enable.azurewebsites.net	develop	Yes

Pre-Deployment Checklist

Before deploying to production, verify the following:

Code Quality

Configuration

New environment variables documented
Sensitive values added to Azure Key Vault (not committed to code)
appsettings.json updated if needed
Database migrations prepared (if any)

Testing

Changes tested in staging environment
API endpoints manually verified
Webhook delivery tested (OpenNode, Stripe)
Rate limiting behavior verified (if changed)
No performance regression

Communication

Team notified of deployment
Deployment window scheduled (prefer low-traffic periods)
Rollback plan documented

Deployment Process

Automatic Deployment (Recommended)

Deployments are triggered automatically when changes are pushed to the main branch.

# Merge approved PR to main
git checkout main
git pull origin main
git merge feature/your-feature
git push origin main

GitHub Actions will:

Run build and tests
Create deployment artifact
Deploy to Azure App Service
Run health check verification

Monitor deployment:

GitHub Actions: https://github.com/AdrianRamsey13/lightning-enable/actions
Azure Portal: App Service > Deployment Center

Manual Deployment (Emergency Only)

For urgent fixes when CI/CD is unavailable:

# Build release package
dotnet publish src/LightningEnable.Api -c Release -o ./publish

# Deploy using Azure CLI
az webapp deploy \
  --resource-group lightning-enable-rg \
  --name lightning-enable-api \
  --src-path ./publish \
  --type zip

Or use Azure Portal:

Navigate to App Service > Deployment Center
Select "Manual Deployment"
Upload ZIP file or connect to local Git

Database Migrations

Before Deployment

If the release includes database schema changes:

# Generate migration (if not already created)
cd src/LightningEnable.Data
dotnet ef migrations add MigrationName --startup-project ../LightningEnable.Api

# Review migration SQL
dotnet ef migrations script --startup-project ../LightningEnable.Api -o migration.sql

Applying Migrations

Migrations are applied automatically on application startup via DbContext.Database.Migrate().

For manual migration:

# From local machine with connection string
dotnet ef database update --startup-project ../LightningEnable.Api

# Or using Azure Cloud Shell
sqlcmd -S lightning-enable-db.database.windows.net -d LightningEnable -i migration.sql

Rollback Migration

# Rollback to previous migration
dotnet ef database update PreviousMigrationName --startup-project ../LightningEnable.Api

Warning: Data migrations may not be fully reversible. Test thoroughly in staging first.

Post-Deployment Verification

Immediate Checks (0-5 minutes)

Health Endpoint

curl https://api.lightningenable.com/health
# Expected: {"status":"Healthy"}

Application Insights
- Check Live Metrics for errors
- Verify request rate is normal
- No spike in exception rate

Basic API Test

# Verify API is responding (authenticated endpoint)
curl -H "X-API-Key: YOUR_API_KEY" \
  https://api.lightningenable.com/api/payments?limit=1

Extended Checks (5-30 minutes)

Webhook Delivery
- Create test payment
- Verify webhook received
Stripe Integration
- Access /checkout page
- Verify pricing displays correctly
L402 Protocol
- Test protected endpoint
- Verify 402 response with invoice
Application Insights Metrics
- Response time within baseline
- Error rate < 1%
- No new exception types

Monitoring Period (30 minutes - 2 hours)

Watch error rate trends
Monitor response times
Check for customer-reported issues
Review logs for warnings

Rollback Procedures

When to Rollback

Rollback immediately if:

Health endpoint failing
Error rate > 5%
Critical functionality broken (payments, webhooks)
Security vulnerability discovered

Quick Rollback (Azure Portal)

Navigate to Azure Portal > App Service
Go to Deployment Center > Deployment history
Click on previous successful deployment
Click "Redeploy"

CLI Rollback

# List recent deployments
az webapp deployment list \
  --resource-group lightning-enable-rg \
  --name lightning-enable-api \
  --query "[].{id:id,status:status,time:receivedTime}" \
  --output table

# Rollback to specific deployment
az webapp deployment source config-zip \
  --resource-group lightning-enable-rg \
  --name lightning-enable-api \
  --src previous-deployment.zip

Slot Swap Rollback (If Staging Slot Available)

# Swap production back to staging
az webapp deployment slot swap \
  --resource-group lightning-enable-rg \
  --name lightning-enable-api \
  --slot staging \
  --target-slot production

Database Rollback

If migration caused issues:

Assess data impact - Can migration be reversed without data loss?

Point-in-time restore (if needed):

az sql db restore \
  --resource-group lightning-enable-rg \
  --server lightning-enable-sql \
  --name LightningEnable \
  --dest-name LightningEnable-Restored \
  --time "2026-01-09T12:00:00Z"

Rollback migration code - Revert to previous migration version
Redeploy - Deploy previous working version

Environment Variables

Required for Production

Variable	Description	Secret
`APPLICATIONINSIGHTS_CONNECTION_STRING`	App Insights telemetry	No
`ConnectionStrings__DefaultConnection`	SQL Server connection	Yes
`DB_ENCRYPTION_KEY`	AES-256 encryption key for sensitive fields	Yes
`ADMIN_API_KEY`	Admin API authentication	Yes
`STRIPE_SECRET_KEY`	Stripe API key	Yes
`STRIPE_WEBHOOK_SECRET`	Stripe webhook signature	Yes

Adding New Environment Variables

Azure Portal:
- App Service > Configuration > Application settings
- Add new setting
- Click Save (triggers restart)

Azure CLI:

az webapp config appsettings set \
  --resource-group lightning-enable-rg \
  --name lightning-enable-api \
  --settings "NEW_VARIABLE=value"

Key Vault Reference (for secrets):

@Microsoft.KeyVault(SecretUri=https://lightning-enable-kv.vault.azure.net/secrets/secret-name/)

Deployment Troubleshooting

Deployment Fails in GitHub Actions

Common Causes:

Test failures
Build errors
Azure authentication expired

Resolution:

Check GitHub Actions logs for specific error
Fix code issues locally
If auth issue, re-run failed deployment
Update Azure service principal if credentials expired

Application Not Starting

Symptoms:

500 errors immediately after deployment
Health check failing

Resolution:

Check App Service > Diagnose and solve problems
Review Application Insights > Failures
Check for missing environment variables
Verify connection strings are correct

Common startup errors:

Error	Cause	Solution
`DB_ENCRYPTION_KEY required`	Missing encryption key	Set environment variable
`Connection string not found`	Missing database config	Add ConnectionStrings__DefaultConnection
`ADMIN_API_KEY required`	Missing admin key	Set ADMIN_API_KEY environment variable

Performance Degradation After Deployment

Check Application Insights Profiler
Compare response times before/after
Review new code for N+1 queries, missing caching
Scale up if needed while investigating
Consider rollback if severe

Scheduled Maintenance

Best Practices

Timing: Deploy during low-traffic periods (UTC 06:00-10:00)
Communication: Notify enterprise customers 24 hours in advance for major changes
Monitoring: Have team member available for 2 hours post-deployment
Documentation: Update runbook with new learnings

Maintenance Window Process

24 hours before:
- Notify customers (if major changes)
- Prepare rollback plan
- Stage changes in staging environment
During maintenance:
- Enable enhanced monitoring
- Deploy changes
- Run verification checks
- Monitor for issues
After maintenance:
- Send completion notification
- Continue monitoring for 2 hours
- Document any issues encountered

Deployment Checklist Template

Copy this checklist for each deployment:

## Deployment: [Version/Feature Name]
**Date:** [Date]
**Deployer:** [Name]

### Pre-Deployment
- [ ] Tests passing
- [ ] Code reviewed
- [ ] Staging tested
- [ ] Team notified

### Deployment
- [ ] Merged to main
- [ ] GitHub Actions completed
- [ ] No deployment errors

### Post-Deployment
- [ ] Health check passing
- [ ] API endpoints responding
- [ ] No error spike in Application Insights
- [ ] Webhooks working
- [ ] Monitoring for 30 minutes

### Rollback (if needed)
- [ ] Issue identified: ____________
- [ ] Rollback initiated at: ____________
- [ ] Rollback completed at: ____________
- [ ] Service restored

### Notes
[Any observations, issues, or follow-up items]

Environment Variables - Complete configuration reference
Incident Response - Handling production issues
Backup & Recovery - Database backup procedures

Deployment Overview​

Pre-Deployment Checklist​

Code Quality​

Configuration​

Testing​

Communication​

Deployment Process​

Automatic Deployment (Recommended)​

Manual Deployment (Emergency Only)​

Database Migrations​

Before Deployment​

Applying Migrations​

Rollback Migration​

Post-Deployment Verification​

Immediate Checks (0-5 minutes)​

Extended Checks (5-30 minutes)​

Monitoring Period (30 minutes - 2 hours)​

Rollback Procedures​

When to Rollback​

Quick Rollback (Azure Portal)​

CLI Rollback​

Slot Swap Rollback (If Staging Slot Available)​

Database Rollback​

Environment Variables​

Required for Production​

Adding New Environment Variables​

Deployment Troubleshooting​

Deployment Fails in GitHub Actions​

Application Not Starting​

Performance Degradation After Deployment​

Scheduled Maintenance​

Best Practices​

Maintenance Window Process​

Deployment Checklist Template​

Related Documentation​