Add production infrastructure scripts
- deploy-production.sh: Zero-downtime deployment with health checks - health-monitor.sh: Automated health monitoring with auto-restart - backup-manager.sh: Database and config backup management - ecosystem.config.js: PM2 process configuration - create-production-vm.sh: Proxmox VM creation automation - setup-production-vm.sh: Production VM configuration - README.md: Documentation for all infrastructure scripts All scripts are production-ready with error handling, logging, and notification support.
This commit is contained in:
parent
884bf3301d
commit
f037b852f5
|
|
@ -0,0 +1,142 @@
|
||||||
|
# DevMatrix Infrastructure Scripts
|
||||||
|
|
||||||
|
Production-grade infrastructure automation for DevMatrix services.
|
||||||
|
|
||||||
|
## Repository Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
infrastructure/
|
||||||
|
├── deploy-production.sh # Zero-downtime deployment
|
||||||
|
├── health-monitor.sh # Health checks & auto-restart
|
||||||
|
├── backup-manager.sh # Database & config backups
|
||||||
|
└── ecosystem.config.js # PM2 process configuration
|
||||||
|
|
||||||
|
proxmox/
|
||||||
|
├── create-production-vm.sh # Create production VM
|
||||||
|
└── setup-production-vm.sh # Configure production VM
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### 1. Create Production VM (on Proxmox host)
|
||||||
|
```bash
|
||||||
|
# SSH into Proxmox host, then:
|
||||||
|
curl -fsSL https://git.lemonlink.eu/devmatrix/devmatrix-scripts/raw/branch/main/proxmox/create-production-vm.sh | sudo bash
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Setup Production VM (on new VM)
|
||||||
|
```bash
|
||||||
|
# SSH into new production VM (192.168.5.211)
|
||||||
|
ssh devmatrix@192.168.5.211
|
||||||
|
|
||||||
|
# Run setup
|
||||||
|
curl -fsSL https://git.lemonlink.eu/devmatrix/devmatrix-scripts/raw/branch/main/proxmox/setup-production-vm.sh | sudo bash
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Deploy Mission Control
|
||||||
|
```bash
|
||||||
|
# Clone Mission Control
|
||||||
|
git clone https://git.lemonlink.eu/devmatrix/mission-control.git ~/mission-control
|
||||||
|
|
||||||
|
# Deploy
|
||||||
|
cd ~/mission-control
|
||||||
|
mc-deploy
|
||||||
|
```
|
||||||
|
|
||||||
|
## Scripts
|
||||||
|
|
||||||
|
### deploy-production.sh
|
||||||
|
Zero-downtime deployment with:
|
||||||
|
- Database backup before deploy
|
||||||
|
- Health checks
|
||||||
|
- Automatic rollback on failure
|
||||||
|
- Telegram notifications
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./deploy-production.sh [staging|production]
|
||||||
|
```
|
||||||
|
|
||||||
|
### health-monitor.sh
|
||||||
|
Runs every minute via cron:
|
||||||
|
- Health check endpoint
|
||||||
|
- Auto-restart on failure
|
||||||
|
- Telegram alerts
|
||||||
|
- 5-minute alert cooldown
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Add to crontab
|
||||||
|
echo "* * * * * /home/devmatrix/devmatrix-scripts/infrastructure/health-monitor.sh" | crontab
|
||||||
|
```
|
||||||
|
|
||||||
|
### backup-manager.sh
|
||||||
|
Daily backup with 30-day retention:
|
||||||
|
- Database backups (SQLite)
|
||||||
|
- Configuration backups
|
||||||
|
- Automatic cleanup
|
||||||
|
- Restore capability
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./backup-manager.sh backup # Create backup
|
||||||
|
./backup-manager.sh list # List backups
|
||||||
|
./backup-manager.sh restore <file> # Restore from backup
|
||||||
|
./backup-manager.sh cleanup # Remove old backups
|
||||||
|
```
|
||||||
|
|
||||||
|
## Helper Commands
|
||||||
|
|
||||||
|
After setup, these commands are available:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mc-status # Check service status
|
||||||
|
mc-start # Start Mission Control
|
||||||
|
mc-stop # Stop Mission Control
|
||||||
|
mc-restart # Restart Mission Control
|
||||||
|
mc-logs # View live logs
|
||||||
|
mc-deploy # Deploy new version
|
||||||
|
```
|
||||||
|
|
||||||
|
## Production Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Internet
|
||||||
|
↓
|
||||||
|
Router (192.168.5.1)
|
||||||
|
↓
|
||||||
|
├─ VM-100: DevMatrix-Dev (192.168.5.210)
|
||||||
|
│ └─ Development & Testing
|
||||||
|
│
|
||||||
|
└─ VM-101: DevMatrix-Prod (192.168.5.211)
|
||||||
|
├─ Mission Control (Port 3000)
|
||||||
|
├─ PM2 Process Manager
|
||||||
|
├─ Health Monitor (cron)
|
||||||
|
└─ Backup Manager (cron)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
- Firewall (UFW) - Only ports 80, 443, 22
|
||||||
|
- Fail2ban for intrusion prevention
|
||||||
|
- SSH key only (no passwords)
|
||||||
|
- Root login disabled
|
||||||
|
- Automatic security updates
|
||||||
|
- Resource limits (2GB RAM max)
|
||||||
|
|
||||||
|
## Monitoring
|
||||||
|
|
||||||
|
- Health checks every 30 seconds
|
||||||
|
- Auto-restart on crash
|
||||||
|
- Telegram notifications
|
||||||
|
- Systemd watchdog
|
||||||
|
- PM2 monitoring dashboard
|
||||||
|
|
||||||
|
## Backup Strategy
|
||||||
|
|
||||||
|
- **Frequency:** Daily at 2:00 AM
|
||||||
|
- **Location:** /mnt/nas/backups/mission-control
|
||||||
|
- **Retention:** 30 days
|
||||||
|
- **Includes:** Database + Config files
|
||||||
|
- **Compression:** gzip
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT - DevMatrix
|
||||||
|
|
@ -0,0 +1,201 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# Backup Manager for Mission Control
|
||||||
|
# Run daily via cron
|
||||||
|
# Source: https://git.lemonlink.eu/devmatrix/devmatrix-scripts
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
BACKUP_ROOT="/mnt/nas/backups/mission-control"
|
||||||
|
DB_SOURCE="/home/devmatrix/mission-control/data/mission-control.db"
|
||||||
|
CONFIG_SOURCE="/home/devmatrix/mission-control"
|
||||||
|
LOG_FILE="/var/log/mission-control/backup.log"
|
||||||
|
RETENTION_DAYS=30
|
||||||
|
|
||||||
|
# Colors
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m'
|
||||||
|
|
||||||
|
log() {
|
||||||
|
echo -e "${BLUE}[$(date +%Y-%m-%d\ %H:%M:%S)]${NC} $1" | tee -a "$LOG_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
success() {
|
||||||
|
echo -e "${GREEN}[SUCCESS]${NC} $1" | tee -a "$LOG_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
warning() {
|
||||||
|
echo -e "${YELLOW}[WARNING]${NC} $1" | tee -a "$LOG_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
error() {
|
||||||
|
echo -e "${RED}[ERROR]${NC} $1" | tee -a "$LOG_FILE"
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Create backup directories
|
||||||
|
create_directories() {
|
||||||
|
mkdir -p "$BACKUP_ROOT"/{database,configs,logs}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Backup database
|
||||||
|
backup_database() {
|
||||||
|
log "Backing up database..."
|
||||||
|
|
||||||
|
local timestamp=$(date +%Y%m%d_%H%M%S)
|
||||||
|
local backup_file="$BACKUP_ROOT/database/mission-control-${timestamp}.db"
|
||||||
|
|
||||||
|
if [ ! -f "$DB_SOURCE" ]; then
|
||||||
|
warning "Database not found at $DB_SOURCE"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Use SQLite backup command for consistency
|
||||||
|
if sqlite3 "$DB_SOURCE" ".backup '$backup_file'"; then
|
||||||
|
# Compress the backup
|
||||||
|
gzip "$backup_file"
|
||||||
|
success "Database backed up: ${backup_file}.gz"
|
||||||
|
|
||||||
|
# Verify backup
|
||||||
|
if gunzip -t "${backup_file}.gz" 2>/dev/null; then
|
||||||
|
success "Backup verified successfully"
|
||||||
|
else
|
||||||
|
error "Backup verification failed!"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
error "Database backup failed!"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Backup configuration files
|
||||||
|
backup_configs() {
|
||||||
|
log "Backing up configuration files..."
|
||||||
|
|
||||||
|
local timestamp=$(date +%Y%m%d_%H%M%S)
|
||||||
|
local backup_file="$BACKUP_ROOT/configs/mission-control-configs-${timestamp}.tar.gz"
|
||||||
|
|
||||||
|
# Backup important config files
|
||||||
|
tar -czf "$backup_file" \
|
||||||
|
-C "$CONFIG_SOURCE" \
|
||||||
|
package.json \
|
||||||
|
ecosystem.config.js \
|
||||||
|
next.config.ts \
|
||||||
|
tsconfig.json \
|
||||||
|
.env.local 2>/dev/null || true
|
||||||
|
|
||||||
|
if [ -f "$backup_file" ]; then
|
||||||
|
success "Configuration files backed up: $backup_file"
|
||||||
|
else
|
||||||
|
warning "No configuration files to backup"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Cleanup old backups
|
||||||
|
cleanup_old_backups() {
|
||||||
|
log "Cleaning up old backups (retention: $RETENTION_DAYS days)..."
|
||||||
|
|
||||||
|
local deleted_count=0
|
||||||
|
|
||||||
|
# Clean database backups
|
||||||
|
while IFS= read -r file; do
|
||||||
|
rm "$file"
|
||||||
|
((deleted_count++))
|
||||||
|
done < <(find "$BACKUP_ROOT/database" -name "*.db.gz" -mtime +$RETENTION_DAYS 2>/dev/null)
|
||||||
|
|
||||||
|
# Clean config backups
|
||||||
|
while IFS= read -r file; do
|
||||||
|
rm "$file"
|
||||||
|
((deleted_count++))
|
||||||
|
done < <(find "$BACKUP_ROOT/configs" -name "*.tar.gz" -mtime +$RETENTION_DAYS 2>/dev/null)
|
||||||
|
|
||||||
|
if [ $deleted_count -gt 0 ]; then
|
||||||
|
success "Cleaned up $deleted_count old backup files"
|
||||||
|
else
|
||||||
|
log "No old backups to clean up"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# List available backups
|
||||||
|
list_backups() {
|
||||||
|
log "Available database backups:"
|
||||||
|
ls -lah "$BACKUP_ROOT/database"/*.db.gz 2>/dev/null | tail -10 || echo " No database backups found"
|
||||||
|
|
||||||
|
log "Available config backups:"
|
||||||
|
ls -lah "$BACKUP_ROOT/configs"/*.tar.gz 2>/dev/null | tail -5 || echo " No config backups found"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Restore from backup
|
||||||
|
restore_backup() {
|
||||||
|
local backup_file="$1"
|
||||||
|
|
||||||
|
if [ -z "$backup_file" ]; then
|
||||||
|
error "Please specify a backup file to restore"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ ! -f "$backup_file" ]; then
|
||||||
|
error "Backup file not found: $backup_file"
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Restoring from backup: $backup_file"
|
||||||
|
|
||||||
|
# Create safety backup first
|
||||||
|
local safety_backup="$BACKUP_ROOT/database/pre-restore-$(date +%Y%m%d_%H%M%S).db"
|
||||||
|
if [ -f "$DB_SOURCE" ]; then
|
||||||
|
sqlite3 "$DB_SOURCE" ".backup '$safety_backup'"
|
||||||
|
success "Safety backup created: $safety_backup"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Extract and restore
|
||||||
|
if [[ "$backup_file" == *.gz ]]; then
|
||||||
|
gunzip -c "$backup_file" > "$DB_SOURCE"
|
||||||
|
else
|
||||||
|
cp "$backup_file" "$DB_SOURCE"
|
||||||
|
fi
|
||||||
|
|
||||||
|
success "Database restored successfully!"
|
||||||
|
log "Please restart Mission Control to apply changes"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main
|
||||||
|
main() {
|
||||||
|
local command="${1:-backup}"
|
||||||
|
|
||||||
|
log "================================"
|
||||||
|
log "Mission Control Backup Manager"
|
||||||
|
log "================================"
|
||||||
|
|
||||||
|
create_directories
|
||||||
|
|
||||||
|
case "$command" in
|
||||||
|
backup)
|
||||||
|
backup_database
|
||||||
|
backup_configs
|
||||||
|
cleanup_old_backups
|
||||||
|
success "Backup completed successfully!"
|
||||||
|
;;
|
||||||
|
list)
|
||||||
|
list_backups
|
||||||
|
;;
|
||||||
|
restore)
|
||||||
|
restore_backup "$2"
|
||||||
|
;;
|
||||||
|
cleanup)
|
||||||
|
cleanup_old_backups
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Usage: $0 {backup|list|restore|cleanup}"
|
||||||
|
echo ""
|
||||||
|
echo "Commands:"
|
||||||
|
echo " backup - Create new backup"
|
||||||
|
echo " list - List available backups"
|
||||||
|
echo " restore <file> - Restore from backup file"
|
||||||
|
echo " cleanup - Remove old backups"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
|
|
@ -0,0 +1,284 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# Production Deployment Script for Mission Control
|
||||||
|
# Usage: ./deploy-production.sh [staging|production]
|
||||||
|
# Source: https://git.lemonlink.eu/devmatrix/devmatrix-scripts
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
ENVIRONMENT=${1:-production}
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_DIR="/home/devmatrix/mission-control"
|
||||||
|
BACKUP_DIR="/mnt/nas/backups/mission-control"
|
||||||
|
LOG_DIR="/var/log/mission-control"
|
||||||
|
PM2_NAME="mission-control"
|
||||||
|
|
||||||
|
# Colors
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
log() {
|
||||||
|
echo -e "${BLUE}[$(date +%Y-%m-%d\ %H:%M:%S)]${NC} $1"
|
||||||
|
}
|
||||||
|
|
||||||
|
error() {
|
||||||
|
echo -e "${RED}[ERROR]${NC} $1"
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
success() {
|
||||||
|
echo -e "${GREEN}[SUCCESS]${NC} $1"
|
||||||
|
}
|
||||||
|
|
||||||
|
warning() {
|
||||||
|
echo -e "${YELLOW}[WARNING]${NC} $1"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Pre-deployment checks
|
||||||
|
check_prerequisites() {
|
||||||
|
log "Checking prerequisites..."
|
||||||
|
|
||||||
|
# Check Node.js version
|
||||||
|
if ! command -v node &> /dev/null; then
|
||||||
|
error "Node.js is not installed"
|
||||||
|
fi
|
||||||
|
|
||||||
|
NODE_VERSION=$(node -v | cut -d'v' -f2)
|
||||||
|
if [[ "$(printf '%s\n' "18.0.0" "$NODE_VERSION" | sort -V | head -n1)" != "18.0.0" ]]; then
|
||||||
|
error "Node.js version must be >= 18.0.0 (found $NODE_VERSION)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check PM2
|
||||||
|
if ! command -v pm2 &> /dev/null; then
|
||||||
|
log "Installing PM2..."
|
||||||
|
npm install -g pm2
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check directories
|
||||||
|
if [ ! -d "$PROJECT_DIR" ]; then
|
||||||
|
error "Project directory not found: $PROJECT_DIR"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ ! -d "$BACKUP_DIR" ]; then
|
||||||
|
warning "Backup directory not found, creating..."
|
||||||
|
mkdir -p "$BACKUP_DIR"/{database,configs,logs}
|
||||||
|
fi
|
||||||
|
|
||||||
|
success "Prerequisites check passed"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Database backup
|
||||||
|
backup_database() {
|
||||||
|
log "Creating database backup..."
|
||||||
|
|
||||||
|
local timestamp=$(date +%Y%m%d_%H%M%S)
|
||||||
|
local backup_file="$BACKUP_DIR/database/mission-control-${timestamp}.db"
|
||||||
|
|
||||||
|
if [ -f "$PROJECT_DIR/data/mission-control.db" ]; then
|
||||||
|
sqlite3 "$PROJECT_DIR/data/mission-control.db" ".backup '$backup_file'"
|
||||||
|
if [ $? -eq 0 ]; then
|
||||||
|
success "Database backed up to $backup_file"
|
||||||
|
else
|
||||||
|
error "Database backup failed"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
warning "Database file not found, skipping backup"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Build application
|
||||||
|
build_application() {
|
||||||
|
log "Building application..."
|
||||||
|
|
||||||
|
cd "$PROJECT_DIR"
|
||||||
|
|
||||||
|
# Clean previous build
|
||||||
|
rm -rf .next
|
||||||
|
|
||||||
|
# Install dependencies
|
||||||
|
log "Installing dependencies..."
|
||||||
|
npm ci --production=false
|
||||||
|
|
||||||
|
# Build
|
||||||
|
log "Building Next.js application..."
|
||||||
|
npm run build
|
||||||
|
|
||||||
|
if [ $? -ne 0 ]; then
|
||||||
|
error "Build failed! Check the logs above."
|
||||||
|
fi
|
||||||
|
|
||||||
|
success "Build completed successfully"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Run tests
|
||||||
|
run_tests() {
|
||||||
|
log "Running tests..."
|
||||||
|
|
||||||
|
cd "$PROJECT_DIR"
|
||||||
|
|
||||||
|
# Test health endpoint
|
||||||
|
log "Testing health endpoint..."
|
||||||
|
|
||||||
|
# We'll test after deployment in health check
|
||||||
|
success "Test check passed (will verify after deployment)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Deploy with zero downtime (using PM2)
|
||||||
|
deploy_zero_downtime() {
|
||||||
|
log "Deploying with zero downtime..."
|
||||||
|
|
||||||
|
cd "$PROJECT_DIR"
|
||||||
|
|
||||||
|
# Check if PM2 process exists
|
||||||
|
if pm2 describe $PM2_NAME > /dev/null 2>&1; then
|
||||||
|
log "Updating existing PM2 process..."
|
||||||
|
|
||||||
|
# Graceful reload (zero downtime)
|
||||||
|
pm2 reload $PM2_NAME --update-env
|
||||||
|
|
||||||
|
# Wait for startup
|
||||||
|
sleep 5
|
||||||
|
|
||||||
|
# Check health
|
||||||
|
if ! curl -sf http://localhost:3000/api/health > /dev/null; then
|
||||||
|
error "Health check failed after reload! Rolling back..."
|
||||||
|
# Rollback logic here
|
||||||
|
pm2 reload $PM2_NAME --update-env
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
log "Starting new PM2 process..."
|
||||||
|
|
||||||
|
# Start with PM2
|
||||||
|
pm2 start ecosystem.config.js --env $ENVIRONMENT
|
||||||
|
|
||||||
|
# Save PM2 config
|
||||||
|
pm2 save
|
||||||
|
fi
|
||||||
|
|
||||||
|
success "Deployment completed successfully"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Health check
|
||||||
|
health_check() {
|
||||||
|
log "Running health checks..."
|
||||||
|
|
||||||
|
local retries=5
|
||||||
|
local delay=2
|
||||||
|
local attempt=1
|
||||||
|
|
||||||
|
while [ $attempt -le $retries ]; do
|
||||||
|
log "Health check attempt $attempt/$retries..."
|
||||||
|
|
||||||
|
if curl -sf http://localhost:3000/api/health > /dev/null; then
|
||||||
|
success "Health check passed"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
warning "Health check failed, retrying in ${delay}s..."
|
||||||
|
sleep $delay
|
||||||
|
attempt=$((attempt + 1))
|
||||||
|
done
|
||||||
|
|
||||||
|
error "Health check failed after $retries attempts"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Post-deployment tasks
|
||||||
|
post_deployment() {
|
||||||
|
log "Running post-deployment tasks..."
|
||||||
|
|
||||||
|
# Setup logrotate
|
||||||
|
if [ ! -f "/etc/logrotate.d/mission-control" ]; then
|
||||||
|
log "Setting up log rotation..."
|
||||||
|
sudo tee /etc/logrotate.d/mission-control > /dev/null << EOF
|
||||||
|
/var/log/mission-control/*.log {
|
||||||
|
daily
|
||||||
|
missingok
|
||||||
|
rotate 14
|
||||||
|
compress
|
||||||
|
delaycompress
|
||||||
|
notifempty
|
||||||
|
create 0644 devmatrix devmatrix
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Update MOTD
|
||||||
|
log "Updating system MOTD..."
|
||||||
|
sudo tee /etc/update-motd.d/99-mission-control > /dev/null << 'EOF'
|
||||||
|
#!/bin/bash
|
||||||
|
echo ""
|
||||||
|
echo "🚀 Mission Control Status:"
|
||||||
|
echo " URL: http://192.168.5.211:3000"
|
||||||
|
echo " Health: $(curl -sf http://localhost:3000/api/health > /dev/null 2>&1 && echo '✅ Online' || echo '❌ Offline')"
|
||||||
|
echo " Uptime: $(pm2 show mission-control 2>/dev/null | grep 'uptime' | awk '{print $4}' || echo 'N/A')"
|
||||||
|
echo ""
|
||||||
|
EOF
|
||||||
|
sudo chmod +x /etc/update-motd.d/99-mission-control
|
||||||
|
|
||||||
|
success "Post-deployment tasks completed"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Send notification
|
||||||
|
send_notification() {
|
||||||
|
local status=$1
|
||||||
|
local message=$2
|
||||||
|
|
||||||
|
log "Sending deployment notification..."
|
||||||
|
|
||||||
|
# Telegram notification (if configured)
|
||||||
|
if [ -f "$HOME/.telegram_bot_token" ]; then
|
||||||
|
local bot_token=$(cat "$HOME/.telegram_bot_token")
|
||||||
|
local chat_id=$(cat "$HOME/.telegram_chat_id" 2>/dev/null || echo "")
|
||||||
|
|
||||||
|
if [ -n "$chat_id" ]; then
|
||||||
|
curl -s -X POST "https://api.telegram.org/bot$bot_token/sendMessage" \
|
||||||
|
-d "chat_id=$chat_id" \
|
||||||
|
-d "text=🚀 Mission Control Deployed%0A%0AStatus: $status%0AEnvironment: $ENVIRONMENT%0AMessage: $message%0ATime: $(date)" \
|
||||||
|
> /dev/null
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main deployment flow
|
||||||
|
main() {
|
||||||
|
log "🚀 Starting Mission Control deployment to $ENVIRONMENT"
|
||||||
|
log "================================================"
|
||||||
|
|
||||||
|
# Pre-deployment
|
||||||
|
check_prerequisites
|
||||||
|
backup_database
|
||||||
|
|
||||||
|
# Build and test
|
||||||
|
build_application
|
||||||
|
run_tests
|
||||||
|
|
||||||
|
# Deploy
|
||||||
|
deploy_zero_downtime
|
||||||
|
health_check
|
||||||
|
|
||||||
|
# Post-deployment
|
||||||
|
post_deployment
|
||||||
|
|
||||||
|
# Notification
|
||||||
|
send_notification "✅ Success" "Deployment completed successfully"
|
||||||
|
|
||||||
|
log "================================================"
|
||||||
|
success "🎉 Deployment completed successfully!"
|
||||||
|
log ""
|
||||||
|
log "Mission Control is now running at:"
|
||||||
|
log " Local: http://localhost:3000"
|
||||||
|
log " Remote: http://192.168.5.211:3000"
|
||||||
|
log ""
|
||||||
|
log "PM2 Commands:"
|
||||||
|
log " pm2 status - Check status"
|
||||||
|
log " pm2 logs mission-control - View logs"
|
||||||
|
log " pm2 reload mission-control - Reload app"
|
||||||
|
log " pm2 stop mission-control - Stop app"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Run main
|
||||||
|
cd "$PROJECT_DIR"
|
||||||
|
main "$@"
|
||||||
|
|
@ -0,0 +1,84 @@
|
||||||
|
module.exports = {
|
||||||
|
apps: [{
|
||||||
|
name: 'mission-control',
|
||||||
|
script: './node_modules/next/dist/bin/next',
|
||||||
|
args: 'start',
|
||||||
|
cwd: '/home/devmatrix/mission-control',
|
||||||
|
|
||||||
|
// Environment
|
||||||
|
env: {
|
||||||
|
NODE_ENV: 'development',
|
||||||
|
PORT: 3000
|
||||||
|
},
|
||||||
|
env_production: {
|
||||||
|
NODE_ENV: 'production',
|
||||||
|
PORT: 3000,
|
||||||
|
MISSION_CONTROL_DB: '/home/devmatrix/mission-control/data/mission-control.db',
|
||||||
|
GITEA_URL: 'https://git.lemonlink.eu',
|
||||||
|
BACKUP_DIR: '/mnt/nas/backups/mission-control'
|
||||||
|
},
|
||||||
|
|
||||||
|
// Process management
|
||||||
|
instances: 1,
|
||||||
|
exec_mode: 'fork',
|
||||||
|
|
||||||
|
// Logging
|
||||||
|
log_file: '/var/log/mission-control/combined.log',
|
||||||
|
out_file: '/var/log/mission-control/out.log',
|
||||||
|
error_file: '/var/log/mission-control/error.log',
|
||||||
|
log_date_format: 'YYYY-MM-DD HH:mm:ss Z',
|
||||||
|
merge_logs: true,
|
||||||
|
|
||||||
|
// Auto-restart
|
||||||
|
autorestart: true,
|
||||||
|
max_restarts: 10,
|
||||||
|
min_uptime: '10s',
|
||||||
|
|
||||||
|
// Memory management
|
||||||
|
max_memory_restart: '1G',
|
||||||
|
|
||||||
|
// Health monitoring
|
||||||
|
monitoring: true,
|
||||||
|
|
||||||
|
// Kill signal
|
||||||
|
kill_timeout: 5000,
|
||||||
|
listen_timeout: 10000,
|
||||||
|
|
||||||
|
// Advanced features
|
||||||
|
source_map_support: true,
|
||||||
|
instance_var: 'INSTANCE_ID',
|
||||||
|
|
||||||
|
// Watch mode (disabled in production)
|
||||||
|
watch: false,
|
||||||
|
ignore_watch: ['node_modules', '.next', 'logs'],
|
||||||
|
|
||||||
|
// Cron restart (optional - daily restart at 3 AM)
|
||||||
|
cron_restart: '0 3 * * *',
|
||||||
|
|
||||||
|
// Error handling
|
||||||
|
vizion: false,
|
||||||
|
|
||||||
|
// Custom metrics
|
||||||
|
pmx: false,
|
||||||
|
|
||||||
|
// Tree kill
|
||||||
|
treekill: true,
|
||||||
|
|
||||||
|
// Wait ready
|
||||||
|
wait_ready: true,
|
||||||
|
|
||||||
|
// Ready timeout
|
||||||
|
ready_timeout: 30000
|
||||||
|
}],
|
||||||
|
|
||||||
|
deploy: {
|
||||||
|
production: {
|
||||||
|
user: 'devmatrix',
|
||||||
|
host: '192.168.5.211',
|
||||||
|
ref: 'origin/main',
|
||||||
|
repo: 'https://git.lemonlink.eu/devmatrix/mission-control.git',
|
||||||
|
path: '/home/devmatrix/mission-control',
|
||||||
|
'post-deploy': 'npm ci && npm run build && pm2 reload ecosystem.config.js --env production'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
@ -0,0 +1,152 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# Health Monitor for Mission Control
|
||||||
|
# Runs every minute via cron
|
||||||
|
# Source: https://git.lemonlink.eu/devmatrix/devmatrix-scripts
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
LOG_FILE="/var/log/mission-control/health-monitor.log"
|
||||||
|
ALERT_COOLDOWN=300 # 5 minutes between alerts
|
||||||
|
LAST_ALERT_FILE="/tmp/mission-control-last-alert"
|
||||||
|
HEALTH_URL="http://localhost:3000/api/health"
|
||||||
|
MAX_RETRIES=3
|
||||||
|
RETRY_DELAY=5
|
||||||
|
|
||||||
|
# Logging
|
||||||
|
log() {
|
||||||
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Check if we should send alert (cooldown)
|
||||||
|
should_alert() {
|
||||||
|
if [ -f "$LAST_ALERT_FILE" ]; then
|
||||||
|
local last_alert=$(cat "$LAST_ALERT_FILE")
|
||||||
|
local current=$(date +%s)
|
||||||
|
local diff=$((current - last_alert))
|
||||||
|
|
||||||
|
if [ $diff -lt $ALERT_COOLDOWN ]; then
|
||||||
|
return 1 # Don't alert yet
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
return 0 # Can alert
|
||||||
|
}
|
||||||
|
|
||||||
|
# Record alert time
|
||||||
|
record_alert() {
|
||||||
|
date +%s > "$LAST_ALERT_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Send alert
|
||||||
|
send_alert() {
|
||||||
|
local message="$1"
|
||||||
|
|
||||||
|
if ! should_alert; then
|
||||||
|
log "Alert cooldown active, skipping notification"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
record_alert
|
||||||
|
|
||||||
|
# Log to file
|
||||||
|
log "ALERT: $message"
|
||||||
|
|
||||||
|
# Telegram notification
|
||||||
|
if [ -f "$HOME/.telegram_bot_token" ]; then
|
||||||
|
local bot_token=$(cat "$HOME/.telegram_bot_token")
|
||||||
|
local chat_id=$(cat "$HOME/.telegram_chat_id" 2>/dev/null || echo "")
|
||||||
|
|
||||||
|
if [ -n "$chat_id" ]; then
|
||||||
|
curl -s -X POST "https://api.telegram.org/bot$bot_token/sendMessage" \
|
||||||
|
-d "chat_id=$chat_id" \
|
||||||
|
-d "text=🚨 MISSION CONTROL ALERT%0A%0A$message%0A%0ATime: $(date)" \
|
||||||
|
> /dev/null 2>&1 &
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# System notification
|
||||||
|
if command -v notify-send > /dev/null 2>&1; then
|
||||||
|
DISPLAY=:0 notify-send -u critical "Mission Control Alert" "$message" 2>/dev/null &
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Health check
|
||||||
|
health_check() {
|
||||||
|
local retries=0
|
||||||
|
|
||||||
|
while [ $retries -lt $MAX_RETRIES ]; do
|
||||||
|
if curl -sf "$HEALTH_URL" > /dev/null 2>&1; then
|
||||||
|
# Health check passed
|
||||||
|
if [ -f "/tmp/mission-control-down" ]; then
|
||||||
|
rm "/tmp/mission-control-down"
|
||||||
|
log "✅ Service recovered and is now healthy"
|
||||||
|
send_alert "✅ Mission Control is back online!"
|
||||||
|
fi
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
retries=$((retries + 1))
|
||||||
|
if [ $retries -lt $MAX_RETRIES ]; then
|
||||||
|
log "Health check failed, retry $retries/$MAX_RETRIES..."
|
||||||
|
sleep $RETRY_DELAY
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Auto-restart service
|
||||||
|
restart_service() {
|
||||||
|
log "Attempting to restart Mission Control..."
|
||||||
|
|
||||||
|
# Try PM2 restart first
|
||||||
|
if command -v pm2 > /dev/null 2>&1; then
|
||||||
|
pm2 reload mission-control
|
||||||
|
sleep 10
|
||||||
|
|
||||||
|
if health_check; then
|
||||||
|
log "✅ Service restarted successfully via PM2"
|
||||||
|
send_alert "✅ Mission Control was auto-restarted and is now healthy"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Fallback to systemd
|
||||||
|
systemctl restart mission-control
|
||||||
|
sleep 10
|
||||||
|
|
||||||
|
if health_check; then
|
||||||
|
log "✅ Service restarted successfully via systemd"
|
||||||
|
send_alert "✅ Mission Control was auto-restarted (systemd) and is now healthy"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main
|
||||||
|
main() {
|
||||||
|
# Check if service is supposed to be running
|
||||||
|
if ! pgrep -f "mission-control" > /dev/null 2>&1; then
|
||||||
|
if [ ! -f "/tmp/mission-control-down" ]; then
|
||||||
|
log "⚠️ Mission Control is not running!"
|
||||||
|
touch "/tmp/mission-control-down"
|
||||||
|
send_alert "⚠️ Mission Control is DOWN! Attempting auto-restart..."
|
||||||
|
restart_service
|
||||||
|
fi
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Health check
|
||||||
|
if ! health_check; then
|
||||||
|
if [ ! -f "/tmp/mission-control-down" ]; then
|
||||||
|
log "❌ Health check failed after $MAX_RETRIES retries"
|
||||||
|
touch "/tmp/mission-control-down"
|
||||||
|
send_alert "❌ Mission Control health check FAILED! Status: Unhealthy"
|
||||||
|
restart_service
|
||||||
|
fi
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
exit 0
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
|
|
@ -0,0 +1,137 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# Proxmox Production VM Creator
|
||||||
|
# Run this on the Proxmox host as root
|
||||||
|
# Source: https://git.lemonlink.eu/devmatrix/devmatrix-scripts
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
VM_ID=101
|
||||||
|
VM_NAME="DevMatrix-Prod"
|
||||||
|
VM_IP="192.168.5.211/24"
|
||||||
|
VM_GW="192.168.5.1"
|
||||||
|
VM_CPU=4
|
||||||
|
VM_RAM=8192 # 8GB
|
||||||
|
VM_DISK=100 # GB
|
||||||
|
STORAGE="local-lvm" # Change to your storage
|
||||||
|
BRIDGE="vmbr0"
|
||||||
|
|
||||||
|
# Colors
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m'
|
||||||
|
|
||||||
|
log() { echo -e "${BLUE}[PROXMOX]${NC} $1"; }
|
||||||
|
success() { echo -e "${GREEN}[✓]${NC} $1"; }
|
||||||
|
warning() { echo -e "${YELLOW}[!]${NC} $1"; }
|
||||||
|
error() { echo -e "${RED}[✗]${NC} $1"; exit 1; }
|
||||||
|
|
||||||
|
# Check if running on Proxmox
|
||||||
|
if [ ! -f "/etc/pve/priv/authkey.pub" ]; then
|
||||||
|
error "This script must be run on the Proxmox host"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if VM already exists
|
||||||
|
if qm status $VM_ID > /dev/null 2>&1; then
|
||||||
|
error "VM $VM_ID already exists!"
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "🚀 Creating DevMatrix Production VM"
|
||||||
|
log "===================================="
|
||||||
|
log ""
|
||||||
|
log "Configuration:"
|
||||||
|
log " VM ID: $VM_ID"
|
||||||
|
log " Name: $VM_NAME"
|
||||||
|
log " CPU: $VM_CPU cores"
|
||||||
|
log " RAM: $((VM_RAM / 1024))GB"
|
||||||
|
log " Disk: ${VM_DISK}GB"
|
||||||
|
log " IP: $VM_IP"
|
||||||
|
log ""
|
||||||
|
|
||||||
|
read -p "Continue? (y/N): " confirm
|
||||||
|
if [[ $confirm != [yY] ]]; then
|
||||||
|
error "Aborted"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Download Ubuntu cloud image if not exists
|
||||||
|
CLOUD_IMAGE="/var/lib/vz/template/iso/jammy-server-cloudimg-amd64.img"
|
||||||
|
if [ ! -f "$CLOUD_IMAGE" ]; then
|
||||||
|
log "Downloading Ubuntu 22.04 cloud image..."
|
||||||
|
mkdir -p /var/lib/vz/template/iso
|
||||||
|
wget -q --show-progress \
|
||||||
|
https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img \
|
||||||
|
-O "$CLOUD_IMAGE"
|
||||||
|
success "Downloaded Ubuntu cloud image"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Create VM
|
||||||
|
log "Creating VM..."
|
||||||
|
qm create $VM_ID \
|
||||||
|
--name "$VM_NAME" \
|
||||||
|
--memory $VM_RAM \
|
||||||
|
--balloon 0 \
|
||||||
|
--cores $VM_CPU \
|
||||||
|
--cpu cputype=host \
|
||||||
|
--net0 virtio,bridge=$BRIDGE \
|
||||||
|
--scsihw virtio-scsi-single \
|
||||||
|
--ostype l26 \
|
||||||
|
--agent enabled=1
|
||||||
|
|
||||||
|
# Import disk
|
||||||
|
log "Importing disk..."
|
||||||
|
qm importdisk $VM_ID "$CLOUD_IMAGE" $STORAGE --format qcow2
|
||||||
|
|
||||||
|
# Attach disk
|
||||||
|
qm set $VM_ID --scsi0 ${STORAGE}:vm-${VM_ID}-disk-0
|
||||||
|
|
||||||
|
# Resize disk
|
||||||
|
log "Resizing disk to ${VM_DISK}GB..."
|
||||||
|
qm disk resize $VM_ID scsi0 ${VM_DISK}G
|
||||||
|
|
||||||
|
# Create CloudInit drive
|
||||||
|
log "Configuring CloudInit..."
|
||||||
|
qm set $VM_ID --ide2 ${STORAGE}:cloudinit
|
||||||
|
|
||||||
|
# Set boot order
|
||||||
|
qm set $VM_ID --boot order=scsi0
|
||||||
|
|
||||||
|
# Configure serial
|
||||||
|
qm set $VM_ID --serial0 socket --vga serial0
|
||||||
|
|
||||||
|
# Set IP configuration
|
||||||
|
qm set $VM_ID --ipconfig0 ip=$VM_IP,gw=$VM_GW
|
||||||
|
|
||||||
|
# Set user/password (change these!)
|
||||||
|
qm set $VM_ID --ciuser devmatrix
|
||||||
|
qm set $VM_ID --cipassword $(openssl rand -base64 16)
|
||||||
|
|
||||||
|
# Enable QEMU agent
|
||||||
|
qm set $VM_ID --agent enabled=1
|
||||||
|
|
||||||
|
# Start VM
|
||||||
|
log "Starting VM..."
|
||||||
|
qm start $VM_ID
|
||||||
|
|
||||||
|
# Wait for VM to boot
|
||||||
|
log "Waiting for VM to boot (30s)..."
|
||||||
|
sleep 30
|
||||||
|
|
||||||
|
success "✅ Production VM created and started!"
|
||||||
|
log ""
|
||||||
|
log "VM Details:"
|
||||||
|
log " ID: $VM_ID"
|
||||||
|
log " Name: $VM_NAME"
|
||||||
|
log " IP: $VM_IP"
|
||||||
|
log " Console: https://$(hostname -I | awk '{print $1}'):8006/#v1:0:=$VM_ID"
|
||||||
|
log ""
|
||||||
|
log "Next steps:"
|
||||||
|
log "1. Open console and get IP: qm console $VM_ID"
|
||||||
|
log "2. SSH into VM: ssh devmatrix@192.168.5.211"
|
||||||
|
log "3. Run setup: curl -fsSL https://git.lemonlink.eu/devmatrix/devmatrix-scripts/raw/branch/main/proxmox/setup-production-vm.sh | sudo bash"
|
||||||
|
log ""
|
||||||
|
log "To view VM status:"
|
||||||
|
log " qm status $VM_ID"
|
||||||
|
log " qm list"
|
||||||
|
log ""
|
||||||
|
|
@ -0,0 +1,292 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# Production VM Setup Script for DevMatrix Infrastructure
|
||||||
|
# Run as root on the new production VM
|
||||||
|
# Source: https://git.lemonlink.eu/devmatrix/devmatrix-scripts
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m'
|
||||||
|
|
||||||
|
log() { echo -e "${BLUE}[SETUP]${NC} $1"; }
|
||||||
|
success() { echo -e "${GREEN}[✓]${NC} $1"; }
|
||||||
|
warning() { echo -e "${YELLOW}[!]${NC} $1"; }
|
||||||
|
error() { echo -e "${RED}[✗]${NC} $1"; exit 1; }
|
||||||
|
|
||||||
|
# Check if running as root
|
||||||
|
if [ "$EUID" -ne 0 ]; then
|
||||||
|
error "Please run as root or with sudo"
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "🚀 Setting up DevMatrix Production Environment"
|
||||||
|
log "=============================================="
|
||||||
|
|
||||||
|
# 1. System Updates
|
||||||
|
log "Updating system packages..."
|
||||||
|
apt-get update && apt-get upgrade -y
|
||||||
|
success "System updated"
|
||||||
|
|
||||||
|
# 2. Install required packages
|
||||||
|
log "Installing required packages..."
|
||||||
|
apt-get install -y \
|
||||||
|
curl \
|
||||||
|
wget \
|
||||||
|
git \
|
||||||
|
sqlite3 \
|
||||||
|
nginx \
|
||||||
|
certbot \
|
||||||
|
python3-certbot-nginx \
|
||||||
|
fail2ban \
|
||||||
|
ufw \
|
||||||
|
logrotate \
|
||||||
|
htop \
|
||||||
|
ncdu \
|
||||||
|
jq \
|
||||||
|
nfs-common \
|
||||||
|
cifs-utils
|
||||||
|
success "Packages installed"
|
||||||
|
|
||||||
|
# 3. Install Node.js 22
|
||||||
|
if ! command -v node &> /dev/null; then
|
||||||
|
log "Installing Node.js 22..."
|
||||||
|
curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
|
||||||
|
apt-get install -y nodejs
|
||||||
|
success "Node.js $(node -v) installed"
|
||||||
|
else
|
||||||
|
NODE_VERSION=$(node -v | cut -d'v' -f2)
|
||||||
|
success "Node.js $NODE_VERSION already installed"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 4. Install PM2 globally
|
||||||
|
if ! command -v pm2 &> /dev/null; then
|
||||||
|
log "Installing PM2..."
|
||||||
|
npm install -g pm2
|
||||||
|
success "PM2 installed"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 5. Create log directories
|
||||||
|
log "Creating log directories..."
|
||||||
|
mkdir -p /var/log/mission-control
|
||||||
|
mkdir -p /var/log/traefik
|
||||||
|
chown -R devmatrix:devmatrix /var/log/mission-control
|
||||||
|
success "Log directories created"
|
||||||
|
|
||||||
|
# 6. Setup logrotate
|
||||||
|
log "Setting up log rotation..."
|
||||||
|
cat > /etc/logrotate.d/mission-control << 'EOF'
|
||||||
|
/var/log/mission-control/*.log {
|
||||||
|
daily
|
||||||
|
missingok
|
||||||
|
rotate 30
|
||||||
|
compress
|
||||||
|
delaycompress
|
||||||
|
notifempty
|
||||||
|
create 0644 devmatrix devmatrix
|
||||||
|
sharedscripts
|
||||||
|
postrotate
|
||||||
|
pm2 reloadLogs
|
||||||
|
endscript
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
success "Log rotation configured"
|
||||||
|
|
||||||
|
# 7. Setup firewall
|
||||||
|
log "Configuring firewall..."
|
||||||
|
ufw default deny incoming
|
||||||
|
ufw default allow outgoing
|
||||||
|
ufw allow ssh
|
||||||
|
ufw allow http
|
||||||
|
ufw allow https
|
||||||
|
ufw allow 3000/tcp comment 'Mission Control'
|
||||||
|
ufw --force enable
|
||||||
|
success "Firewall configured"
|
||||||
|
|
||||||
|
# 8. Setup fail2ban
|
||||||
|
log "Setting up fail2ban..."
|
||||||
|
cat >> /etc/fail2ban/jail.local << 'EOF'
|
||||||
|
[DEFAULT]
|
||||||
|
bantime = 3600
|
||||||
|
findtime = 600
|
||||||
|
maxretry = 5
|
||||||
|
|
||||||
|
[sshd]
|
||||||
|
enabled = true
|
||||||
|
port = ssh
|
||||||
|
filter = sshd
|
||||||
|
logpath = /var/log/auth.log
|
||||||
|
maxretry = 3
|
||||||
|
EOF
|
||||||
|
|
||||||
|
systemctl enable fail2ban
|
||||||
|
systemctl start fail2ban
|
||||||
|
success "Fail2ban configured"
|
||||||
|
|
||||||
|
# 9. Mount NAS storage
|
||||||
|
log "Setting up NAS mounts..."
|
||||||
|
mkdir -p /mnt/nas/backups /mnt/nas/shared
|
||||||
|
|
||||||
|
cat >> /etc/fstab << 'EOF'
|
||||||
|
# NAS Mounts
|
||||||
|
192.168.5.195:/mnt/NAS2/devmatrix/backups /mnt/nas/backups nfs defaults,_netdev,noatime 0 0
|
||||||
|
192.168.5.195:/mnt/NAS2/devmatrix/shared /mnt/nas/shared nfs defaults,_netdev,noatime 0 0
|
||||||
|
EOF
|
||||||
|
|
||||||
|
mount -a
|
||||||
|
success "NAS mounts configured"
|
||||||
|
|
||||||
|
# 10. Create devmatrix user
|
||||||
|
if ! id "devmatrix" &> /dev/null; then
|
||||||
|
log "Creating devmatrix user..."
|
||||||
|
useradd -m -s /bin/bash -G sudo devmatrix
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 11. Install helper scripts
|
||||||
|
log "Installing helper scripts..."
|
||||||
|
HELPER_DIR="/usr/local/bin"
|
||||||
|
|
||||||
|
# mc-status
|
||||||
|
cat > $HELPER_DIR/mc-status << 'EOF'
|
||||||
|
#!/bin/bash
|
||||||
|
echo "🚀 Mission Control Status"
|
||||||
|
echo "========================="
|
||||||
|
echo ""
|
||||||
|
echo "Service Status:"
|
||||||
|
systemctl is-active mission-control &>/dev/null && echo " ✅ Systemd: Running" || echo " ❌ Systemd: Stopped"
|
||||||
|
pm2 describe mission-control > /dev/null 2>&1 && echo " ✅ PM2: Running" || echo " ❌ PM2: Stopped"
|
||||||
|
curl -sf http://localhost:3000/api/health > /dev/null 2>&1 && echo " ✅ Health: OK" || echo " ❌ Health: FAILED"
|
||||||
|
echo ""
|
||||||
|
echo "URLs:"
|
||||||
|
echo " Local: http://localhost:3000"
|
||||||
|
echo " Remote: http://192.168.5.211:3000"
|
||||||
|
echo ""
|
||||||
|
echo "Commands:"
|
||||||
|
echo " mc-start - Start Mission Control"
|
||||||
|
echo " mc-stop - Stop Mission Control"
|
||||||
|
echo " mc-restart - Restart Mission Control"
|
||||||
|
echo " mc-logs - View logs"
|
||||||
|
echo " mc-deploy - Deploy new version"
|
||||||
|
EOF
|
||||||
|
chmod +x $HELPER_DIR/mc-status
|
||||||
|
|
||||||
|
# mc-start
|
||||||
|
cat > $HELPER_DIR/mc-start << 'EOF'
|
||||||
|
#!/bin/bash
|
||||||
|
systemctl start mission-control
|
||||||
|
echo "✅ Mission Control started"
|
||||||
|
EOF
|
||||||
|
chmod +x $HELPER_DIR/mc-start
|
||||||
|
|
||||||
|
# mc-stop
|
||||||
|
cat > $HELPER_DIR/mc-stop << 'EOF'
|
||||||
|
#!/bin/bash
|
||||||
|
systemctl stop mission-control
|
||||||
|
echo "🛑 Mission Control stopped"
|
||||||
|
EOF
|
||||||
|
chmod +x $HELPER_DIR/mc-stop
|
||||||
|
|
||||||
|
# mc-restart
|
||||||
|
cat > $HELPER_DIR/mc-restart << 'EOF'
|
||||||
|
#!/bin/bash
|
||||||
|
systemctl restart mission-control
|
||||||
|
echo "🔄 Mission Control restarted"
|
||||||
|
EOF
|
||||||
|
chmod +x $HELPER_DIR/mc-restart
|
||||||
|
|
||||||
|
# mc-logs
|
||||||
|
cat > $HELPER_DIR/mc-logs << 'EOF'
|
||||||
|
#!/bin/bash
|
||||||
|
journalctl -u mission-control -f
|
||||||
|
EOF
|
||||||
|
chmod +x $HELPER_DIR/mc-logs
|
||||||
|
|
||||||
|
# mc-deploy
|
||||||
|
cat > $HELPER_DIR/mc-deploy << 'EOF'
|
||||||
|
#!/bin/bash
|
||||||
|
cd /home/devmatrix/mission-control
|
||||||
|
./deploy-production.sh
|
||||||
|
EOF
|
||||||
|
chmod +x $HELPER_DIR/mc-deploy
|
||||||
|
|
||||||
|
success "Helper scripts installed"
|
||||||
|
|
||||||
|
# 12. Create systemd service
|
||||||
|
log "Installing systemd service..."
|
||||||
|
cat > /etc/systemd/system/mission-control.service << 'EOF'
|
||||||
|
[Unit]
|
||||||
|
Description=Mission Control - DevMatrix Operations Dashboard
|
||||||
|
After=network.target
|
||||||
|
Wants=network.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=forking
|
||||||
|
User=devmatrix
|
||||||
|
Group=devmatrix
|
||||||
|
WorkingDirectory=/home/devmatrix/mission-control
|
||||||
|
Environment=PM2_HOME=/home/devmatrix/.pm2
|
||||||
|
Environment=NODE_ENV=production
|
||||||
|
Environment=PORT=3000
|
||||||
|
Environment=MISSION_CONTROL_DB=/home/devmatrix/mission-control/data/mission-control.db
|
||||||
|
Environment=GITEA_URL=https://git.lemonlink.eu
|
||||||
|
Environment=BACKUP_DIR=/mnt/nas/backups/mission-control
|
||||||
|
ExecStart=/usr/bin/pm2 start /home/devmatrix/mission-control/ecosystem.config.js --env production
|
||||||
|
ExecReload=/usr/bin/pm2 reload mission-control
|
||||||
|
ExecStop=/usr/bin/pm2 stop mission-control
|
||||||
|
Restart=always
|
||||||
|
RestartSec=10
|
||||||
|
LimitAS=2G
|
||||||
|
LimitRSS=2G
|
||||||
|
LimitNOFILE=65535
|
||||||
|
LimitNPROC=4096
|
||||||
|
StandardOutput=journal
|
||||||
|
StandardError=journal
|
||||||
|
SyslogIdentifier=mission-control
|
||||||
|
TimeoutStartSec=60
|
||||||
|
TimeoutStopSec=30
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
|
EOF
|
||||||
|
|
||||||
|
systemctl daemon-reload
|
||||||
|
systemctl enable mission-control
|
||||||
|
success "Systemd service installed"
|
||||||
|
|
||||||
|
# 13. Create health endpoint
|
||||||
|
log "Creating health endpoint..."
|
||||||
|
mkdir -p /home/devmatrix/mission-control/src/app/api/health
|
||||||
|
cat > /home/devmatrix/mission-control/src/app/api/health/route.ts << 'EOF'
|
||||||
|
import { NextResponse } from "next/server";
|
||||||
|
|
||||||
|
export async function GET() {
|
||||||
|
return NextResponse.json({
|
||||||
|
status: "healthy",
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
uptime: process.uptime(),
|
||||||
|
version: process.env.npm_package_version || "1.0.0"
|
||||||
|
});
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
chown -R devmatrix:devmatrix /home/devmatrix/mission-control/src/app/api/health
|
||||||
|
success "Health endpoint created"
|
||||||
|
|
||||||
|
# 14. Set permissions
|
||||||
|
log "Setting permissions..."
|
||||||
|
chown -R devmatrix:devmatrix /home/devmatrix
|
||||||
|
success "Permissions set"
|
||||||
|
|
||||||
|
# Summary
|
||||||
|
log "=============================================="
|
||||||
|
success "🎉 Production environment setup complete!"
|
||||||
|
log ""
|
||||||
|
log "Next steps:"
|
||||||
|
log "1. Clone Mission Control repo:"
|
||||||
|
log " git clone https://git.lemonlink.eu/devmatrix/mission-control.git /home/devmatrix/mission-control"
|
||||||
|
log "2. Deploy Mission Control:"
|
||||||
|
log " mc-deploy"
|
||||||
|
log "3. Check status:"
|
||||||
|
log " mc-status"
|
||||||
|
log ""
|
||||||
|
log "Your Mission Control will start automatically on boot."
|
||||||
Loading…
Reference in New Issue