Production Deployment Guide¶

Complete guide for deploying the Drone Operations application to production on Ubuntu Server using Anaconda, MariaDB, Docker Compose, and Nginx.

Table of Contents¶

Overview
Prerequisites
Server Preparation
Anaconda Installation
MariaDB Installation
Docker and Docker Compose
Nginx Installation
Application Deployment
Database Initialisation
Nginx Configuration
SSL/TLS Setup
Service Management
Monitoring and Maintenance
Backup Strategy
Troubleshooting
Security Hardening

Overview¶

Architecture¶

Ubuntu Server (Host)
│
├── Nginx (Port 80/443)
│   ├── SSL/TLS Termination
│   ├── Static File Serving
│   ├── Reverse Proxy → Gunicorn
│   └── Upload File Serving
│
└── Docker Compose
    ├── MariaDB Container (Internal Port 3306)
    │   └── Persistent Volume: mariadb_data
    └── Web Application Container (Port 8000)
        ├── Gunicorn WSGI Server
        ├── Flask Application
        └── Persistent Volume: uploads

Technology Stack¶

OS: Ubuntu 22.04 LTS (or newer)
Python: 3.12+ (via Anaconda)
Web Server: Nginx
Application Server: Gunicorn
Database: MariaDB 12.x
Containerization: Docker + Docker Compose
SSL/TLS: Let's Encrypt (Certbot)

Prerequisites¶

Hardware Requirements¶

Minimum: - 2 CPU cores - 4 GB RAM - 20 GB disk space - Stable internet connection

Recommended: - 4+ CPU cores - 8 GB RAM - 50+ GB SSD storage - 100 Mbps network

Server Access¶

Root or sudo privileges
SSH access
Domain name (optional, for SSL)
Firewall ports: 22 (SSH), 80 (HTTP), 443 (HTTPS)

Before You Begin¶

Fresh Ubuntu 22.04 LTS installation
Updated system packages
Configured hostname and DNS (if using domain)
Non-root user with sudo privileges (recommended)

Deployment Scenarios¶

This guide supports two deployment scenarios:

Full Control: - You have complete root access including firewall and user management - You can perform all steps independently - Suitable for dedicated servers or VPS where you have full administrative control

Restricted Control (Infrastructure Team Managed): - You have root/sudo access but certain operations are managed by infrastructure team - Firewall configuration handled by infrastructure team - User account creation handled by infrastructure team - All other operations performed independently - Suitable for enterprise environments with centralized infrastructure management

Throughout this guide, sections requiring special permissions are marked with Option A (full control) and Option B (restricted control) alternatives.

Server Preparation¶

1. Update System¶

# Connect to server
ssh user@your-server-ip

# Update package lists
sudo apt update

# Upgrade all packages
sudo apt upgrade -y

# Reboot if kernel was updated
sudo reboot

2. Install Essential Tools¶

# Install basic utilities
sudo apt install -y \
    curl \
    wget \
    git \
    unzip \
    build-essential \
    software-properties-common \
    apt-transport-https \
    ca-certificates \
    gnupg \
    lsb-release

3. Configure Firewall¶

Option A: Full Control (You Manage Firewall)

If you have full control over the server:

# Enable UFW
sudo ufw enable

# Allow SSH (IMPORTANT: Do this first!)
sudo ufw allow 22/tcp

# Allow HTTP and HTTPS
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# Check status
sudo ufw status

# Expected output:
# Status: active
# To                         Action      From
# --                         ------      ----
# 22/tcp                     ALLOW       Anywhere
# 80/tcp                     ALLOW       Anywhere
# 443/tcp                    ALLOW       Anywhere

Option B: Restricted Control (Infrastructure Team Manages Firewall)

If firewall management is handled by your infrastructure team:

Request from infrastructure team: - Port 22/tcp (SSH) - Should already be open - Port 80/tcp (HTTP) - Required for web access and Let's Encrypt verification - Port 443/tcp (HTTPS) - Required for secure web access

Verify ports are open:

# Check if port 80 is accessible
sudo netstat -tlnp | grep :80

# Check if port 443 is accessible
sudo netstat -tlnp | grep :443

# Or use ss command
sudo ss -tlnp | grep -E ':(80|443)'

# Test from external machine (after Nginx installation)
# curl http://your-server-ip

4. Create Application User¶

Option A: Full Control (You Create Users)

If you have permission to create users:

# Create dedicated user for application
sudo useradd -r -m -d /opt/drones -s /bin/bash drones

# Set password for drones user
sudo passwd drones

# Add to docker group (will do this after Docker installation)

Option B: Restricted Control (Infrastructure Team Creates Users)

If user creation is handled by your infrastructure team:

Request from infrastructure team: - Create system user: drones - Home directory: /opt/drones - Shell: /bin/bash - Groups: docker (to be added after Docker installation)

After user is created, verify:

# Check user exists
id drones
# Expected: uid=... gid=... groups=...

# Check home directory
ls -ld /opt/drones
# Expected: drwxr-xr-x ... drones drones ... /opt/drones

# You can still set ownership and permissions
sudo chown -R drones:drones /opt/drones
sudo chmod 755 /opt/drones

If user creation not available: You can skip this step and use your own user account instead of the drones user throughout the deployment. Replace all references to drones with your username.

Anaconda Installation¶

1. Download Anaconda/Miniconda¶

For full Anaconda:

# Download Anaconda installer
cd /tmp
wget https://repo.anaconda.com/archive/Anaconda3-2024.02-1-Linux-x86_64.sh

# Verify checksum (optional but recommended)
sha256sum Anaconda3-2024.02-1-Linux-x86_64.sh

For Miniconda (lighter, recommended for servers):

# Download Miniconda installer
cd /tmp
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

# Make executable
chmod +x Miniconda3-latest-Linux-x86_64.sh

2. Install Anaconda/Miniconda¶

# Run installer with sudo (for system-wide installation)
sudo bash Miniconda3-latest-Linux-x86_64.sh

# Follow prompts:
# - Accept licence: yes
# - Install location: /usr/local/miniconda3 (recommended for system-wide)
# - Initialise conda: no (we'll do this manually for all users)

# Add miniconda to system PATH for all users
echo 'export PATH="/usr/local/miniconda3/bin:$PATH"' | sudo tee /etc/profile.d/miniconda.sh

# Make it executable
sudo chmod +x /etc/profile.d/miniconda.sh

# Load PATH for current session
source /etc/profile.d/miniconda.sh

# Verify installation
conda --version
# Expected: conda 24.x.x

# Update conda
sudo /usr/local/miniconda3/bin/conda update -n base -c defaults conda -y

# Initialise conda for current user (optional, enables conda activate)
conda init bash
source ~/.bashrc

3. Configure Conda¶

# Disable auto-activation of base environment
conda config --set auto_activate_base false

# Add conda-forge channel
conda config --add channels conda-forge

# Set channel priority
conda config --set channel_priority strict

MariaDB Installation¶

Option A: MariaDB via Docker (Recommended)¶

This is handled by Docker Compose. Skip to Docker Installation.

Option B: MariaDB on Host System¶

Only use this if you prefer MariaDB directly on the host instead of in Docker.

# Add MariaDB repository
curl -LsS https://r.mariadb.com/downloads/mariadb_repo_setup | sudo bash -s -- --mariadb-server-version="mariadb-12.0"

# Update package lists
sudo apt update

# Install MariaDB
sudo apt install -y mariadb-server mariadb-client

# Start and enable service
sudo systemctl start mariadb
sudo systemctl enable mariadb

# Secure installation
sudo mysql_secure_installation
# - Set root password
# - Remove anonymous users: Yes
# - Disallow remote root: Yes
# - Remove test database: Yes
# - Reload privileges: Yes

# Configure for production
sudo nano /etc/mysql/mariadb.conf.d/50-server.cnf

# Add/modify under [mysqld]:
[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
max_connections = 200
innodb_buffer_pool_size = 1G
innodb_log_file_size = 256M
wait_timeout = 28800
interactive_timeout = 28800

# Restart MariaDB
sudo systemctl restart mariadb

# Create application database
sudo mysql -u root -p

# Run SQL commands:
CREATE DATABASE drones_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'drones_user'@'localhost' IDENTIFIED BY 'SECURE_PASSWORD_HERE';
GRANT ALL PRIVILEGES ON drones_db.* TO 'drones_user'@'localhost';
FLUSH PRIVILEGES;
EXIT;

Note: If using host MariaDB, you'll need to modify docker-compose.yml to connect to host instead of container.

Docker and Docker Compose¶

1. Install Docker¶

Using official Docker installation method:

# Update package lists and install prerequisites
sudo apt update
sudo apt install -y ca-certificates curl

# Create keyrings directory with proper permissions
sudo install -m 0755 -d /etc/apt/keyrings

# Download Docker's official GPG key
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add Docker repository to Apt sources (DEB822 format)
sudo tee /etc/apt/sources.list.d/docker.sources > /dev/null <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Signed-By: /etc/apt/keyrings/docker.asc
EOF

# Update package lists with new repository
sudo apt update

# Install Docker packages
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Start and enable Docker service
sudo systemctl start docker
sudo systemctl enable docker

# Verify installation
sudo docker --version
# Expected: Docker version 24.x.x or newer

# Test Docker installation
sudo docker run hello-world
# Should download and run successfully

Note: If you encounter PAM authentication errors with sudo curl, see the Troubleshooting section for workarounds. The workaround would be:

# Download without sudo
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /tmp/docker.asc

# Copy with sudo
sudo cp /tmp/docker.asc /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Clean up
rm /tmp/docker.asc

2. Configure Docker¶

Option A: Full Control (You Manage Users/Groups)

If you can manage user groups:

# Add current user to docker group
sudo usermod -aG docker $USER

# Add drones user to docker group (if created in step 4)
sudo usermod -aG docker drones

# Apply group changes (logout/login or run):
newgrp docker

# Test Docker without sudo
docker run hello-world
# Should download and run successfully

Option B: Restricted Control (Infrastructure Team Manages Groups)

If group management requires infrastructure team:

Request from infrastructure team: - Add your user account to docker group - Add drones user to docker group (if user was created)

After groups are configured, verify:

# Check your user's groups
groups
# Should include: docker

# Check drones user's groups (if applicable)
groups drones
# Should include: docker

# Test Docker without sudo
docker run hello-world
# Should download and run successfully

# If you still need sudo, you may need to logout/login
# Or run: newgrp docker

If using your own user instead of drones: Only add your account to the docker group.

3. Verify Docker Compose¶

# Check Docker Compose version
docker compose version
# Expected: Docker Compose version v2.x.x or newer

Nginx Installation¶

1. Install Nginx¶

# Install Nginx
sudo apt install -y nginx

# Start and enable service
sudo systemctl start nginx
sudo systemctl enable nginx

# Check status
sudo systemctl status nginx
# Should show: active (running)

# Test in browser
# Navigate to: http://your-server-ip
# Should see: "Welcome to nginx!"

2. Basic Nginx Configuration¶

# Remove default site
sudo rm /etc/nginx/sites-enabled/default

# Test configuration
sudo nginx -t
# Expected: nginx: configuration file /etc/nginx/nginx.conf test is successful

# Create directory for application config
sudo mkdir -p /etc/nginx/sites-available
sudo mkdir -p /etc/nginx/sites-enabled

Application Deployment¶

1. Clone Repository¶

GitHub Authentication Setup

GitHub requires token-based authentication for repository access. Choose one of the following methods:

Option A: Personal Access Token (HTTPS)

Step 1: Create GitHub Personal Access Token

Go to GitHub: https://github.com/settings/tokens
Click "Generate new token" → "Generate new token (classic)"
Configure token:
Note: "Production server deployment - drones app"
Expiration: Choose appropriate duration (90 days, 1 year, or no expiration)
Scopes: Select repo (Full control of private repositories)
Click "Generate token"
IMPORTANT: Copy the token immediately (you won't see it again)
Format: ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Step 2: Clone Repository with Token

# Create deployment directory
sudo mkdir -p /opt/drones
sudo chown $USER:$USER /opt/drones
cd /opt/drones

# Clone using token (replace YOUR_TOKEN and yourusername/drones)
git clone https://YOUR_TOKEN@github.com/yourusername/drones.git .

# Example:
# git clone https://ghp_ABC123xyz789@github.com/briandavison/drones.git .

# Set ownership
sudo chown -R drones:drones /opt/drones

Step 3: Configure Git Credential Storage (Optional but Recommended)

# Store credentials in encrypted format
git config --global credential.helper store

# Or cache for 1 hour (more secure)
git config --global credential.helper 'cache --timeout=3600'

# Future git operations will prompt for credentials once, then remember

Option B: SSH Key Authentication (More Secure)

Step 1: Generate SSH Key on Server

# Generate SSH key (press Enter for defaults)
ssh-keygen -t ed25519 -C "your_email@example.com"

# Start SSH agent
eval "$(ssh-agent -s)"

# Add key to agent
ssh-add ~/.ssh/id_ed25519

# Display public key
cat ~/.ssh/id_ed25519.pub
# Copy the output

Step 2: Add SSH Key to GitHub

Go to GitHub: https://github.com/settings/keys
Click "New SSH key"
Title: "Production Server - drones"
Paste your public key
Click "Add SSH key"

Step 3: Clone Repository with SSH

# Create deployment directory
sudo mkdir -p /opt/drones
sudo chown $USER:$USER /opt/drones
cd /opt/drones

# Clone using SSH
git clone git@github.com:yourusername/drones.git .

# Set ownership
sudo chown -R drones:drones /opt/drones

Option C: Alternative - Manual File Transfer

If you cannot use Git authentication:

# From your local machine, create a tarball
tar -czf drones.tar.gz --exclude='.git' --exclude='__pycache__' --exclude='.venv' /path/to/local/drones/

# Transfer to server via SCP
scp drones.tar.gz user@server:/tmp/

# On server: Extract to deployment directory
sudo mkdir -p /opt/drones
cd /opt/drones
sudo tar -xzf /tmp/drones.tar.gz --strip-components=1
sudo chown -R drones:drones /opt/drones
rm /tmp/drones.tar.gz

Security Best Practices:

⚠️ Never commit tokens to repository ⚠️ Use environment-specific tokens with minimum required permissions ⚠️ Rotate tokens periodically ⚠️ Delete tokens when no longer needed ⚠️ Prefer SSH keys over tokens for long-term deployments

2. Create Conda Environment¶

# Switch to drones user
sudo su - drones
cd /opt/drones

# Initialise conda for drones user (PATH already set via /etc/profile.d/miniconda.sh)
source /usr/local/miniconda3/etc/profile.d/conda.sh

# Create conda environment
conda create -n drones python=3.12 -y

# Activate environment
conda activate drones

# Install dependencies
pip install -r requirements.txt

# Verify installations
python -c "import flask, pymysql, flask_migrate, gunicorn; print('All packages installed')"

# Exit drones user
exit

3. Create Environment Configuration¶

# Switch to drones user
sudo su - drones
cd /opt/drones

# Copy production environment template
cp .env.example .env

# Edit environment file
nano .env

Configure .env file:

# Flask Configuration
SECRET_KEY=GENERATE_STRONG_SECRET_KEY_HERE
FLASK_ENV=production
FLASK_DEBUG=False

# Database Configuration (Docker)
DB_HOST=db
DB_PORT=3306
DB_USER=drones_user
DB_PASSWORD=CHANGE_TO_SECURE_PASSWORD
DB_NAME=drones_db
DB_ROOT_PASSWORD=CHANGE_TO_SECURE_ROOT_PASSWORD

# Database URL (auto-constructed)
DATABASE_URL=mysql+pymysql://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}?charset=utf8mb4

# Upload Configuration
UPLOAD_FOLDER=/app/uploads
MAX_CONTENT_LENGTH=10485760

Generate SECRET_KEY:

# Generate secure secret key
python -c "import secrets; print(secrets.token_hex(32))"

# Copy output and paste into .env file as SECRET_KEY value

4. Create Required Directories¶

# Still as drones user
mkdir -p /opt/drones/uploads
chmod 755 /opt/drones/uploads

# Exit drones user
exit

5. Set File Permissions¶

# Set ownership
sudo chown -R drones:drones /opt/drones

# Set permissions
sudo chmod -R 755 /opt/drones

# Protect .env file
sudo chmod 600 /opt/drones/.env

Database Initialisation¶

1. Build and Start Docker Containers¶

# Navigate to application directory
cd /opt/drones

# Build Docker images
sudo docker compose build

# Start services (detached mode)
sudo docker compose up -d

# Check container status
sudo docker compose ps

# Expected output:
# NAME          STATUS          PORTS
# drones_db     running (healthy)
# drones_web    running (healthy) 0.0.0.0:8000->8000/tcp

2. Wait for Database to Initialise¶

# Watch database logs until ready
sudo docker compose logs -f db

# Look for: "mariadbd: ready for connections"
# Press Ctrl+C to exit logs

# Or wait 30 seconds
sleep 30

3. Initialise Database Schema¶

# Access web container
sudo docker compose exec web /bin/bash

# Inside container (no conda needed - uses system Python):
# Initialise Flask-Migrate
flask db init

# Create initial migration
flask db migrate -m "Initial migration"

# Apply migration
flask db upgrade

# Seed database
python create_db.py

# Exit container
exit

Note: The Docker container uses Python 3.12 directly, not Miniconda. All packages are installed via pip in the container. Conda is only used on the host system for development (optional).

Expected output from seeding:

⚠️  Run 'flask db upgrade' before seeding!
This script seeds data to MariaDB only.
Adding roles...
Roles added.
Adding users...
Users added.
Project purposes added.
Adding drones...
Drones added.
Database seeding completed successfully.

4. Verify Application Health¶

# Check health endpoint
curl http://localhost:8000/health

# Expected response:
# {"status":"healthy","database":"connected"}

# Check application logs
sudo docker compose logs -f web

# Press Ctrl+C to exit

Nginx Configuration¶

1. Copy Nginx Configuration¶

# Copy configuration file
sudo cp /opt/drones/deploy/nginx.conf /etc/nginx/sites-available/drones

# Edit configuration
sudo nano /etc/nginx/sites-available/drones

2. Customise Configuration¶

Update server_name with your domain:

server {
    listen 80;
    server_name yourdomain.com www.yourdomain.com;  # Change this

    # ... rest of config
}

Complete configuration should include:

upstream drones_app {
    server localhost:8000 fail_timeout=0;
}

server {
    listen 80;
    server_name yourdomain.com www.yourdomain.com;

    client_max_body_size 10M;

    # Static files
    location /static/ {
        alias /opt/drones/app/static/;
        expires 30d;
        add_header Cache-Control "public, immutable";
    }

    # Uploads
    location /uploads/ {
        alias /opt/drones/uploads/;
        expires 7d;
        add_header Cache-Control "public";
    }

    # Proxy to Gunicorn
    location / {
        proxy_pass http://drones_app;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_redirect off;

        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }
}

3. Enable Site¶

# Create symbolic link
sudo ln -s /etc/nginx/sites-available/drones /etc/nginx/sites-enabled/

# Test configuration
sudo nginx -t

# Expected: configuration file test is successful

# Reload Nginx
sudo systemctl reload nginx

4. Test Application¶

# Test via domain (if DNS configured)
curl http://yourdomain.com

# Or test via IP
curl http://your-server-ip

# Should return HTML content of homepage

SSL/TLS Setup¶

1. Install Certbot¶

# Install Certbot
sudo apt install -y certbot python3-certbot-nginx

2. Obtain SSL Certificate¶

Prerequisites: - Domain name points to server IP (A record) - Port 80 accessible from internet - Nginx running

# Obtain certificate
sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com

# Follow prompts:
# - Enter email address
# - Agree to terms: Yes
# - Share email: Your choice
# - Redirect HTTP to HTTPS: 2 (Yes, recommended)

3. Verify SSL Configuration¶

# Check certificate status
sudo certbot certificates

# Test SSL configuration
curl https://yourdomain.com

# Check SSL grade (optional)
# Visit: https://www.ssllabs.com/ssltest/

4. Configure Auto-Renewal¶

# Test renewal process
sudo certbot renew --dry-run

# Expected: Congratulations, all simulated renewals succeeded

# Certbot timer should be enabled by default
sudo systemctl status certbot.timer

# Manual renewal (if needed)
sudo certbot renew

Service Management¶

Docker Compose Management¶

# Start all services
cd /opt/drones
sudo docker compose up -d

# Stop all services
sudo docker compose down

# Restart services
sudo docker compose restart

# View logs
sudo docker compose logs -f

# View specific service logs
sudo docker compose logs -f web
sudo docker compose logs -f db

# Check service status
sudo docker compose ps

# Pull latest images
sudo docker compose pull

# Rebuild containers
sudo docker compose build --no-cache

# Restart specific service
sudo docker compose restart web

Application Updates¶

# Navigate to application directory
cd /opt/drones

# Pull latest code
sudo -u drones git pull

# Stop the service (takes down containers cleanly)
sudo systemctl stop drones

# Rebuild the web container (--no-cache ensures a fresh pip install)
sudo docker compose build --no-cache web

# Start the service
sudo systemctl start drones

# Apply any pending database migrations
sudo docker compose exec web flask db upgrade

# Verify the service is healthy
sudo systemctl status drones
curl http://localhost:8000/health

Note: If using token authentication without credential storage, you'll need to provide your GitHub username and token each time you pull. To avoid this:

# Configure credential storage for drones user
sudo -u drones git config --global credential.helper store

# Or use SSH keys (preferred for production)
# See repository cloning section for SSH setup

Database Migrations¶

# Apply all pending migrations (containers must be running)
sudo docker compose exec web flask db upgrade

# Check current revision
sudo docker compose exec web flask db current

# Show full migration history
sudo docker compose exec web flask db history

# Downgrade one step
sudo docker compose exec web flask db downgrade -1

# Downgrade to a specific revision
sudo docker compose exec web flask db downgrade <revision_id>

If containers are not yet running (e.g. after systemctl stop before rebuilding):

# Start the database only, migrate, then start everything
sudo docker compose up -d db
sudo docker compose run --rm web flask db upgrade
sudo systemctl start drones

Systemd Service (Alternative to Docker Compose CLI)¶

Create systemd service for Docker Compose:

# Create service file
sudo nano /etc/systemd/system/drones.service

Service content:

[Unit]
Description=Drones Flask Application
Requires=docker.service
After=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/drones
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

Enable and use service:

# Reload systemd
sudo systemctl daemon-reload

# Enable service
sudo systemctl enable drones

# Start service
sudo systemctl start drones

# Check status
sudo systemctl status drones

# Stop service
sudo systemctl stop drones

# Restart service
sudo systemctl restart drones

Operations Manual Updates¶

The operations manual is a MkDocs/Material site in opsman_docs/. The build hook (hooks.py) injects live database content (active drones, pilots with active Flyer ID certifications), so it must run inside the Docker container. The opsman_docs/site/ output directory is volume-mounted to the host (see docker-compose.yml), making the built files accessible at /opt/drones/opsman_docs/site/ for deployment to the nginx web root.

When to rebuild: - Any markdown file under opsman_docs/docs/ was edited - opsman_docs/mkdocs.yml was changed - After a deployment that modifies hooks.py or the models it queries

Option A — Admin web interface (recommended)¶

Log in as an admin and use the Rebuild button in the admin panel, or POST directly:

# Trigger rebuild (returns JSON with success/error message)
curl -s -X POST https://yourdomain.com/opsman/rebuild \
  -H "Cookie: <your-session-cookie>"

Diagnostic endpoint if the build fails:

curl -s https://yourdomain.com/opsman/rebuild/diagnostics \
  -H "Cookie: <your-session-cookie>" | python3 -m json.tool

After the rebuild completes, sync the output to the nginx web root:

sudo rsync -av --delete /opt/drones/opsman_docs/site/ /var/www/html/opsman/

Option B — CLI (when the app is not running)¶

# Build inside the web container
sudo docker compose exec web bash -c "cd /app/opsman_docs && mkdocs build"

# Sync to nginx web root
sudo rsync -av --delete /opt/drones/opsman_docs/site/ /var/www/html/opsman/

# Confirm nginx is serving updated content
curl -sI https://yourdomain.com/opsman/ | grep -E "HTTP|Last-Modified"

App Documentation Updates¶

The technical documentation site is a MkDocs/Material site in docs/. Unlike the operations manual it has no database hooks, so the build is purely static. The docs/site/ output directory is volume-mounted to the host (see docker-compose.yml), making built files available at /opt/drones/docs/site/ for deployment.

When to rebuild: - Any markdown file under docs/ was edited - docs/mkdocs.yml was changed

Option A — Admin web interface (recommended)¶

curl -s -X POST https://yourdomain.com/docs/rebuild \
  -H "Cookie: <your-session-cookie>"

sudo rsync -av --delete /opt/drones/docs/site/ /var/www/html/docs/

Option B — CLI¶

sudo docker compose exec web bash -c "cd /app/docs && mkdocs build"
sudo rsync -av --delete /opt/drones/docs/site/ /var/www/html/docs/
curl -sI https://yourdomain.com/docs/ | grep -E "HTTP|Last-Modified"

User Guide Updates¶

The user guide is a MkDocs/Material site in user_guide/. Fully static — no database hooks. The user_guide/site/ output is volume-mounted to the host, making built files available at /opt/drones/user_guide/site/.

When to rebuild: - Any markdown file under user_guide/docs/ was edited - user_guide/mkdocs.yml was changed

Option A — Admin web interface (recommended)¶

curl -s -X POST https://yourdomain.com/user-guide/rebuild \
  -H "Cookie: <your-session-cookie>"

sudo rsync -av --delete /opt/drones/user_guide/site/ /var/www/html/user-guide/

Option B — CLI¶

sudo docker compose exec web bash -c "cd /app/user_guide && mkdocs build"
sudo rsync -av --delete /opt/drones/user_guide/site/ /var/www/html/user-guide/
curl -sI https://yourdomain.com/user-guide/ | grep -E "HTTP|Last-Modified"

Monitoring and Maintenance¶

1. Application Monitoring¶

Check application health:

# Health endpoint
curl http://localhost:8000/health

# Full logs
sudo docker compose logs -f

# Resource usage
sudo docker stats

Monitor disk usage:

# Check disk space
df -h

# Check Docker space usage
sudo docker system df

# Clean up Docker resources
sudo docker system prune -a

2. Database Monitoring¶

# Access MariaDB container
sudo docker compose exec db mysql -u root -p

# Check database size
SELECT table_schema AS "Database",
    ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS "Size (MB)"
FROM information_schema.TABLES
GROUP BY table_schema;

# Check active connections
SHOW PROCESSLIST;

# Check database status
SHOW STATUS;

# Exit
EXIT;

3. Log Management¶

# View application logs
sudo docker compose logs -f web

# View last 100 lines
sudo docker compose logs --tail=100 web

# View Nginx access logs
sudo tail -f /var/log/nginx/access.log

# View Nginx error logs
sudo tail -f /var/log/nginx/error.log

# Rotate logs (configured in logrotate)
sudo logrotate -f /etc/logrotate.d/nginx

4. Performance Tuning¶

Gunicorn workers:

Edit gunicorn.conf.py:

# Adjust based on server resources
# Formula: (2 x $num_cores) + 1

import multiprocessing
workers = multiprocessing.cpu_count() * 2 + 1

# Or set manually for 4 cores:
workers = 9

MariaDB optimization:

Access container and edit config:

sudo docker compose exec db bash

# Edit MariaDB config
nano /etc/mysql/mariadb.conf.d/50-server.cnf

# Adjust based on RAM (example for 8GB server):
[mysqld]
innodb_buffer_pool_size = 2G
innodb_log_file_size = 512M
max_connections = 200
query_cache_size = 32M

Backup Strategy¶

1. Database Backups¶

Manual backup:

# Create backup directory
sudo mkdir -p /opt/backups/database
sudo chown drones:drones /opt/backups/database

# Create backup
sudo docker compose exec db mysqldump \
    -u root -p${DB_ROOT_PASSWORD} \
    --single-transaction \
    --quick \
    --lock-tables=false \
    drones_db > /opt/backups/database/drones_$(date +%Y%m%d_%H%M%S).sql

# Compress backup
gzip /opt/backups/database/drones_*.sql

Automated daily backup script:

# Create backup script
sudo nano /opt/backups/backup_database.sh

Script content:

#!/bin/bash
BACKUP_DIR="/opt/backups/database"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/drones_$DATE.sql"

# Load environment variables
source /opt/drones/.env

# Create backup
docker compose -f /opt/drones/docker-compose.yml exec -T db mysqldump \
    -u root -p${DB_ROOT_PASSWORD} \
    --single-transaction \
    --quick \
    --lock-tables=false \
    drones_db > "$BACKUP_FILE"

# Compress backup
gzip "$BACKUP_FILE"

# Delete backups older than 30 days
find "$BACKUP_DIR" -name "drones_*.sql.gz" -mtime +30 -delete

echo "Backup completed: ${BACKUP_FILE}.gz"

Make executable and schedule:

# Make executable
sudo chmod +x /opt/backups/backup_database.sh

# Add to crontab
sudo crontab -e

# Add line (runs daily at 2 AM):
0 2 * * * /opt/backups/backup_database.sh >> /var/log/database_backup.log 2>&1

2. File Uploads Backup¶

# Create backup directory
sudo mkdir -p /opt/backups/uploads

# Backup uploads directory
sudo rsync -av /opt/drones/uploads/ /opt/backups/uploads/

# Or create tarball
sudo tar -czf /opt/backups/uploads/uploads_$(date +%Y%m%d).tar.gz /opt/drones/uploads/

3. Application Code Backup¶

# Backup entire application
sudo tar -czf /opt/backups/app_$(date +%Y%m%d).tar.gz \
    --exclude=/opt/drones/.venv \
    --exclude=/opt/drones/__pycache__ \
    --exclude=/opt/drones/*.db \
    /opt/drones/

4. Restore from Backup¶

Database restore:

# Stop web container
sudo docker compose stop web

# Restore database
gunzip < /opt/backups/database/drones_YYYYMMDD_HHMMSS.sql.gz | \
sudo docker compose exec -T db mysql -u root -p${DB_ROOT_PASSWORD} drones_db

# Start web container
sudo docker compose start web

# Verify
curl http://localhost:8000/health

Troubleshooting¶

System and Authentication Issues¶

Sudo PAM authentication errors:

If you encounter errors like:

sudo: PAM account management error: Authentication service cannot retrieve authentication info
sudo: a password is required

Possible causes and solutions:

Network authentication timeout (LDAP/Active Directory):

# Check if system is using network authentication
grep -E "pam_ldap|pam_sss|pam_winbind" /etc/pam.d/common-auth

# If using network auth, verify network connectivity
ping -c 3 <ldap-server-ip>

# Contact infrastructure team if LDAP/AD is unreachable

Sudo session expired - re-authenticate:

# Clear sudo timestamp
sudo -k

# Re-authenticate with password
sudo -v

# Try your command again
sudo <your-command>

Workaround: Download files without sudo, then move:

# For Docker GPG key download (example):
# Download as regular user
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /tmp/docker.asc

# Copy with sudo
sudo cp /tmp/docker.asc /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Clean up
rm /tmp/docker.asc

Check sudo configuration:

# Verify you're in sudoers (ask infrastructure team if not)
sudo -l

# Check PAM configuration (infrastructure team)
cat /etc/pam.d/sudo

# Expected to see lines like:
# @include common-auth
# @include common-account
# @include common-session-noninteractive

Alternative: Use root shell if available:

# Switch to root shell (if permitted)
sudo su -

# Or use sudo -i
sudo -i

# Run commands directly without sudo prefix
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

If issue persists: Contact your infrastructure team - they may need to check: - PAM configuration for LDAP/AD authentication - Network connectivity to authentication servers - Your user's sudo permissions and group memberships

File Permission Issues¶

Airspace cache permission errors:

When creating new projects, you may see errors like:

PermissionError: [Errno 13] Permission denied: 'airspace_cache/downloads'

Cause: The airspace_cache directory needs to be writable by the container user (UID 1000).

Solution:

cd /opt/drones

# Set correct ownership for airspace cache
sudo chown -R 1000:1000 airspace_cache/

# Verify permissions
ls -la airspace_cache/

# Expected output:
# drwxr-xr-x  3 1000 1000 4096 ... airspace_cache

Also fix uploads directory if needed:

# Set ownership for uploads
sudo chown -R 1000:1000 uploads/

# Verify
ls -la uploads/

Initialise airspace cache if missing:

# Create cache directories and download initial data
sudo docker compose exec web python scripts/init_airspace_cache.py

# Verify cache created successfully
sudo docker compose exec web ls -la airspace_cache/

Note: These directories must be owned by UID 1000 (the container's appuser). After pulling new code or rebuilding containers, check permissions if you encounter similar errors.

Container Issues¶

Docker build fails with exit code 100:

If you see errors like:

failed to solve: process "/bin/sh -c apt-get update && apt-get install..."
did not complete successfully: exit code: 100

This indicates package installation failure during Docker build. Try these solutions in order:

Solution 1: Build with no cache and verbose output

# Clean build without cache
cd /opt/drones
sudo docker compose build --no-cache --progress=plain

# Check which package is failing in the output

Solution 2: Check network connectivity from container

# Test if container can reach Debian repositories
sudo docker run --rm python:3.12-slim apt-get update

# If this fails, check DNS resolution
sudo docker run --rm python:3.12-slim ping -c 3 deb.debian.org

# If DNS fails, configure Docker DNS
sudo nano /etc/docker/daemon.json

Add DNS servers:

{
  "dns": ["8.8.8.8", "8.8.4.4", "1.1.1.1"]
}

Then restart Docker:

sudo systemctl restart docker

Solution 3: Update Dockerfile to handle repository issues

Edit Dockerfile to add retry logic and fix-missing flag:

# Replace the apt-get install line in runtime stage (around line 25):
RUN apt-get update && \
    apt-get install -y --no-install-recommends --fix-missing \
    default-mysql-client \
    curl \
    libcairo2 \
    libpango-1.0-0 \
    libpangocairo-1.0-0 \
    libgdk-pixbuf2.0-0 \
    libffi-dev \
    shared-mime-info \
    || (apt-get update && apt-get install -y --no-install-recommends --fix-missing \
    default-mysql-client \
    curl \
    libcairo2 \
    libpango-1.0-0 \
    libpangocairo-1.0-0 \
    libgdk-pixbuf2.0-0 \
    libffi-dev \
    shared-mime-info) \
    && rm -rf /var/lib/apt/lists/*

Solution 4: Fix package name errors

If you see "Package 'xxx' has no installation candidate", the package name is incorrect for the Debian version:

Common issue - libgdk-pixbuf package name:

Error: Package 'libgdk-pixbuf2.0-0' has no installation candidate
Fix: Use 'libgdk-pixbuf-2.0-0' (hyphens instead of dots)

To find correct package names:

# Check Python base image Debian version
sudo docker run --rm python:3.12-slim cat /etc/os-release
# Expected: Debian GNU/Linux 12 (bookworm)

# Search for correct package name
sudo docker run --rm python:3.12-slim bash -c "apt-get update && apt-cache search libgdk-pixbuf"

# Example output:
# libgdk-pixbuf-2.0-0 - GDK Pixbuf library (correct name)
# libgdk-pixbuf2.0-dev - GDK Pixbuf library (development files)

# Test specific package
sudo docker run --rm python:3.12-slim bash -c "apt-get update && apt-cache show libgdk-pixbuf-2.0-0"

Common WeasyPrint dependency corrections for Debian Bookworm: - ✅ libcairo2 (correct) - ✅ libpango-1.0-0 (correct) - ✅ libpangocairo-1.0-0 (correct) - ❌ libgdk-pixbuf2.0-0 → ✅ libgdk-pixbuf-2.0-0 (use hyphens) - ✅ libffi-dev (correct) - ✅ shared-mime-info (correct)

Solution 5: Check for architecture mismatches

# Check your system architecture
uname -m
# Expected: x86_64 or aarch64

# Ensure Docker is building for correct architecture
sudo docker compose build --platform linux/amd64

# If on ARM server, use ARM-compatible base images

Solution 6: Check Docker Hub for base image issues

# Pull latest Python 3.12 slim image
sudo docker pull python:3.12-slim

# If pull fails, Docker Hub may be having issues
# Check: https://status.docker.com/

# Use alternative registry mirror if needed

Quick Fix: Use different Python base image

If all else fails, try a different base image version in Dockerfile:

# Try latest stable
FROM python:3.12-slim as builder

# Or try specific Debian version
FROM python:3.12-slim-bookworm as builder

# Or try Ubuntu-based
FROM python:3.12-slim-bullseye as builder

Container won't start:

# Check logs
sudo docker compose logs db
sudo docker compose logs web

# Check container status
sudo docker compose ps

# Restart containers
sudo docker compose restart

# Rebuild if needed
sudo docker compose down
sudo docker compose build --no-cache
sudo docker compose up -d

Database connection errors:

# Verify database is running
sudo docker compose ps db

# Check database logs
sudo docker compose logs db | grep ERROR

# Verify connection string in .env
cat /opt/drones/.env | grep DATABASE_URL

# Test connection from web container
sudo docker compose exec web python -c "from app import db; db.session.execute(db.text('SELECT 1')); print('Connected')"

Nginx Issues¶

502 Bad Gateway:

# Check if Gunicorn is running
sudo docker compose ps web

# Check if port 8000 is accessible
curl http://localhost:8000/health

# Check Nginx logs
sudo tail -f /var/log/nginx/error.log

# Restart services
sudo docker compose restart web
sudo systemctl restart nginx

SSL Certificate Issues:

# Check certificate status
sudo certbot certificates

# Renew certificate
sudo certbot renew

# Check Nginx SSL configuration
sudo nginx -t

Performance Issues¶

High memory usage:

# Check Docker stats
sudo docker stats

# Check system resources
htop

# Restart containers to clear memory
sudo docker compose restart

Slow database queries:

# Access database
sudo docker compose exec db mysql -u root -p drones_db

# Enable slow query log
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 2;

# Check slow queries
SHOW VARIABLES LIKE 'slow_query%';

Security Hardening¶

1. System Security¶

# Install fail2ban
sudo apt install -y fail2ban

# Configure fail2ban for SSH
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

# Install unattended-upgrades
sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

2. Docker Security¶

# Run containers as non-root user (already configured in Dockerfile)

# Scan images for vulnerabilities
sudo docker scan drones-web

# Keep images updated
sudo docker compose pull
sudo docker compose up -d

3. Database Security¶

# Use strong passwords (already done)
# Limit network access (database only accessible to app container)

# Regular updates
sudo docker compose pull mariadb
sudo docker compose up -d db

4. Nginx Security Headers¶

Add to Nginx configuration:

# Security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Content-Security-Policy "default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; font-src 'self' data:;" always;

5. Regular Security Updates¶

# Update system packages
sudo apt update && sudo apt upgrade -y

# Update Docker images
cd /opt/drones
sudo docker compose pull
sudo docker compose up -d

# Update Python packages (rebuild container instead)
# Note: Container packages are baked into the image, not upgraded in running container
sudo docker compose build --no-cache web
sudo docker compose up -d web

Maintenance Schedule¶

Daily¶

✅ Check application health: curl http://localhost:8000/health
✅ Monitor logs: sudo docker compose logs --tail=50
✅ Automated database backup (via cron)

Weekly¶

✅ Review Nginx access logs for unusual activity
✅ Check disk space: df -h
✅ Review Docker resource usage: sudo docker stats
✅ Test backup restoration process

Monthly¶

✅ Update system packages: sudo apt update && sudo apt upgrade
✅ Update Docker images: sudo docker compose pull && sudo docker compose up -d
✅ Review and rotate logs
✅ Security audit

Quick Reference¶

Essential Commands¶

# Start application
sudo systemctl start drones

# Stop application
sudo systemctl stop drones

# Restart application
sudo systemctl restart drones

# Check service status
sudo systemctl status drones

# View logs (all containers)
sudo docker compose logs -f

# View web container logs only
sudo docker compose logs -f web

# Access web container shell
sudo docker compose exec web /bin/bash

# Access database
sudo docker compose exec db mysql -u root -p drones_db

# Backup database
sudo /opt/backups/backup_database.sh

# Check health
curl http://localhost:8000/health

# Reload Nginx
sudo systemctl reload nginx

Important Files and Directories¶

Application: /opt/drones/
Environment: /opt/drones/.env
Uploads: /opt/drones/uploads/
Backups: /opt/backups/
Nginx Config: /etc/nginx/sites-available/drones
Nginx Logs: /var/log/nginx/
SSL Certs: /etc/letsencrypt/live/yourdomain.com/

Support Resources¶

Flask: https://flask.palletsprojects.com/
Gunicorn: https://docs.gunicorn.org/
Docker: https://docs.docker.com/
Nginx: https://nginx.org/en/docs/
MariaDB: https://mariadb.com/kb/en/documentation/
Let's Encrypt: https://letsencrypt.org/docs/

Post-Deployment Checklist¶

[ ] Application accessible via domain/IP
[ ] Health check endpoint returns healthy status
[ ] Can login with default admin credentials
[ ] Can create and view projects
[ ] File uploads working
[ ] SSL/TLS configured and working
[ ] Automated backups scheduled
[ ] Firewall configured correctly
[ ] Monitoring in place
[ ] Documentation updated with production details
[ ] Changed all default passwords
[ ] Configured domain DNS
[ ] Tested disaster recovery procedure

Congratulations! Your Drone Operations application is now deployed to production.

For development environment setup, see DEVELOPMENT_SETUP.md.