Flight Log Import System - Design Document¶

Executive Summary¶

Overview¶

The Flight Log Import System processes DJI flight log files (.txt format) uploaded by users, extracting telemetry data, calculating flight statistics, and storing structured metadata in the database. The system provides a complete post-flight analysis workflow with batch upload support, automatic drone matching, duplicate detection, and comprehensive error handling.

Business Value¶

Compliance & Record-Keeping: - Maintains complete flight history for regulatory compliance (CAA requirements) - Generates auditable records with timestamps, locations, and flight parameters - Links flight logs to specific projects for operational documentation - Provides exportable telemetry data (CSV format) for detailed analysis

Safety Analysis: - Identifies maximum altitude violations (regulatory limit: 120m/400ft) - Tracks maximum speeds and unusual flight patterns - Validates flight duration against battery capacity - Flags anomalous statistics for review

Operational Insights: - Tracks total flight time and distance per drone - Monitors equipment usage and maintenance needs - Provides historical flight data for site assessments - Supports data-driven decision making for operations

Key Capabilities¶

Batch Upload Processing
Upload up to 25 flight logs simultaneously
Sequential processing with individual success/fail reporting
Duplicate detection across batch (SHA256 hash comparison)
Atomic transactions per file (rollback on failure)
Automatic Data Extraction
Parses proprietary DJI flight log format via external parser
Extracts 1000+ telemetry frames per flight
Calculates accurate statistics using Haversine distance formula
Generates CSV export with timestamp, GPS, altitude, speed, battery data
Intelligent Timestamp Handling
Primary: Extract from telemetry data (ISO 8601 format)
Validation: Reject timestamps before 2013 (pre-consumer drones era)
Fallback: Extract date/time from filename pattern (DJIFlightRecord_YYYY-MM-DD_HH-MM-SS.txt)
Last resort: Use upload timestamp
Drone Matching by Serial Number
Automatic matching against drone database
Fuzzy matching (uses startswith() for truncated serials)
Links flight logs to drone maintenance records
Tracks per-drone flight hours and usage
Statistics Validation
Validates against physical constraints (duration < 24hrs, altitude < 500m, speed < 30m/s)
Cross-checks calculated values against parser summary
Generates warnings for unusual values
Prevents database corruption from malformed data

Architecture Overview¶

System Components¶

┌─────────────────────────────────────────────────────────────────┐
│                      User Upload Interface                       │
│              (dashboard/flight_logs_upload.html)                 │
└───────────────────────────────┬─────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                   Upload Route Handler                           │
│            /dashboard/flight-logs/upload-batch [POST]            │
│                  (routes_flight_logs.py:592)                     │
└───────────────────────────────┬─────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                   Processing Pipeline                            │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────────────┐     │
│  │  Validation │→ │  File Save   │→ │  DB Record Create  │     │
│  └─────────────┘  └──────────────┘  └────────────────────┘     │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────────────┐     │
│  │  Parse Log  │→ │  Validate    │→ │  Update Record     │     │
│  │  (DJI API)  │  │  Statistics  │  │  (with metadata)   │     │
│  └─────────────┘  └──────────────┘  └────────────────────┘     │
└───────────────────────────────┬─────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Storage Layer                               │
│  ┌──────────────────────┐  ┌──────────────────────────────┐    │
│  │  Filesystem          │  │  Database (MariaDB)          │    │
│  │  - Raw logs (.txt)   │  │  - FlightLog table           │    │
│  │  - CSV exports       │  │  - Metadata & statistics     │    │
│  └──────────────────────┘  └──────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────┘

File: `app/routes_flight_logs.py`¶

Primary route handler: Contains all upload, processing, and display logic.

Key functions: - upload_flight_logs_batch() - Main upload handler - parse_and_extract_flight_data() - DJI parser wrapper - calculate_flight_statistics() - Telemetry analysis - validate_flight_statistics() - Bounds checking - match_drone_by_serial() - Drone database lookup

File: `app/models.py`¶

FlightLog model (lines 498-578): Database schema and helper methods.

External Dependency: DJI Parser¶

Location: app/utils/dji_parser/

Purpose: Proprietary DJI flight log format parser (provided by third-party)

API: - parser.get_flight_summary(filepath) - Extract summary metadata - parser.parse(filepath) - Full telemetry frame extraction - parser.parse_to_csv(input_path, output_path) - CSV generation

Requires: DJI API key (environment variable DJI_API_KEY)

Processing Pipeline¶

Phase 1: Upload & Validation (Lines 592-636)¶

Endpoint: POST /dashboard/flight-logs/upload-batch

Input: - files[]: Array of .txt files (max 25) - project_id: Optional project association (int) - description: Required if no project_id

Validation Steps: 1. File count check: Reject if > 25 files 2. Extension check: Only .txt files allowed 3. File size check: Max 50MB per file 4. Empty file check: Reject 0-byte files 5. Metadata validation: Require description if standalone (no project) 6. Project ownership: Verify user has edit access to project

Code Reference:

# app/routes_flight_logs.py:117-153
def validate_flight_log_file(file, check_size=True):
    """Validate uploaded flight log file."""
    if not file or not file.filename:
        return False, "No file selected", 0

    # Extension check
    if not allowed_log_file(filename):
        return False, "Only .txt files are allowed", 0

    # Size check
    if check_size and file_size > MAX_LOG_FILE_SIZE:
        return False, f"File too large. Maximum size is 50MB", file_size

Phase 2: Duplicate Detection (Lines 656-668)¶

Strategy: Content-based deduplication using SHA256 hash

Process: 1. Calculate SHA256 hash of uploaded file content 2. Query database for existing logs with same hash and user_id 3. If duplicate found, return error with original upload details

Why SHA256? - Fast computation (8KB chunks) - Collision-resistant (2^256 keyspace) - Detects identical content regardless of filename changes

Code Reference:

# app/routes_flight_logs.py:156-181
def calculate_file_hash(file):
    """Calculate SHA256 hash of uploaded file."""
    sha256 = hashlib.sha256()
    file.seek(0)
    while chunk := file.read(8192):
        sha256.update(chunk)
    file.seek(0)
    return sha256.hexdigest()

# app/routes_flight_logs.py:183-201
def check_duplicate_log(user_id, file_hash):
    """Check if a flight log with the same hash already exists."""
    return FlightLog.query.filter_by(
        user_id=user_id,
        file_hash=file_hash
    ).first()

Phase 3: File Storage (Lines 670-673)¶

Storage Strategy: - Raw logs: uploads/flight_logs/raw/{user_id}_{uuid}_{original_filename}.txt - CSV exports: uploads/flight_logs/csv/{user_id}_{uuid}_{original_filename}.csv

Naming Convention: - Prefix with user_id for easy filtering - UUID ensures uniqueness - Preserve original filename for user reference

Code Reference:

# app/routes_flight_logs.py:203-248
def save_flight_log_file(file, user_id, original_filename):
    """Save uploaded file to disk with unique naming."""
    unique_id = str(uuid.uuid4())
    raw_filename = f"{user_id}_{unique_id}_{original_filename}"
    csv_filename = f"{user_id}_{unique_id}_{original_filename.rsplit('.', 1)[0]}.csv"

    # Save file
    file.save(raw_filepath)

    return {
        'raw_filename': raw_filename,
        'csv_filename': csv_filename,
        'raw_filepath': raw_filepath,
        'csv_filepath': csv_filepath,
        'unique_id': unique_id
    }

Phase 4: Database Record Creation (Lines 674-683)¶

Initial Record: - Status: parsing_status='pending' - Temporary timestamps (updated after parsing) - File metadata (size, hash, paths) - User association

Code Reference:

# app/routes_flight_logs.py:251-291
def create_flight_log_record(user_id, project_id, description, file_info, file_size, file_hash):
    """Create initial FlightLog database entry with pending status."""
    now = datetime.now(timezone.utc)

    flight_log = FlightLog(
        user_id=user_id,
        project_id=project_id,
        description=description,
        original_filename=file_info['raw_filename'].split('_', 2)[2],
        raw_log_filename=file_info['raw_filename'],
        raw_log_filepath=file_info['raw_filepath'],
        csv_filename=file_info['csv_filename'],
        csv_filepath=file_info['csv_filepath'],
        file_size=file_size,
        file_hash=file_hash,
        uploaded_at=now,
        flight_start_time=now,  # Temporary, updated after parsing
        flight_end_time=now,    # Temporary, updated after parsing
        parsing_status='pending'
    )

    db.session.add(flight_log)
    return flight_log

Phase 5: Log Parsing & Data Extraction (Lines 686-693)¶

DJI Parser Integration: 1. Get summary: Basic metadata (duration, distance, drone serial) 2. Parse full data: Extract all telemetry frames 3. Generate CSV: Export to human-readable format

Data Structure:

# Telemetry frame structure (from DJI parser)
{
    "custom": {
        "dateTime": "2025-06-17T13:44:17.568Z"  # ISO 8601 timestamp
    },
    "osd": {
        "latitude": 55.9533,
        "longitude": -3.1883,
        "height": 10.2,          # Altitude in meters
        "flyTime": 42,           # Seconds since takeoff
        "xSpeed": 2.1,           # X velocity (m/s)
        "ySpeed": 1.5            # Y velocity (m/s)
    },
    "battery": {
        "percent": 85,
        "voltage": 12.4
    },
    # ... additional fields (gimbal, camera, rc, home, recover, app)
}

Code Reference:

# app/routes_flight_logs.py:294-396
def parse_and_extract_flight_data(raw_filepath, csv_filepath):
    """Parse DJI flight log and extract all relevant data."""
    from app.utils.dji_parser.dji_parser import DJIParser

    parser = DJIParser(api_key=os.getenv('DJI_API_KEY'))

    # Get flight summary
    summary = parser.get_flight_summary(raw_filepath)
    if not summary or 'details' not in summary:
        return {'success': False, 'error': 'Failed to parse flight summary'}

    # Parse full data and generate CSV
    data = parser.parse(raw_filepath)
    if not data or 'frames' not in data:
        return {'success': False, 'error': 'Failed to parse flight data'}

    parser.parse_to_csv(raw_filepath, csv_filepath)

    # Calculate statistics
    frames = data.get('frames', [])
    statistics = calculate_flight_statistics(frames)

    # Extract timestamps
    flight_start, flight_end = extract_flight_timestamps(frames, statistics, raw_filepath)

    return {
        'success': True,
        'summary': summary,
        'data': data,
        'statistics': statistics,
        'flight_start': flight_start,
        'flight_end': flight_end
    }

Phase 6: Statistics Calculation (Lines 53-114)¶

Calculated Metrics: 1. Duration: last_frame.flyTime - first_frame.flyTime 2. Total Distance: Sum of Haversine distances between consecutive GPS points 3. Max Altitude: Maximum height value across all frames 4. Max Speed: Maximum horizontal speed (calculated from x/y velocity components)

Haversine Formula:

# app/routes_flight_logs.py:25-50
def haversine_distance(lat1, lon1, lat2, lon2):
    """Calculate distance between two GPS coordinates."""
    R = 6371000  # Earth radius in meters

    lat1_rad = math.radians(lat1)
    lat2_rad = math.radians(lat2)
    dlat = math.radians(lat2 - lat1)
    dlon = math.radians(lon2 - lon1)

    a = math.sin(dlat/2)**2 + math.cos(lat1_rad) * math.cos(lat2_rad) * math.sin(dlon/2)**2
    c = 2 * math.atan2(math.sqrt(a), math.sqrt(1-a))
    distance = R * c

    return distance

Statistics Calculation:

# app/routes_flight_logs.py:53-114
def calculate_flight_statistics(frames):
    """Calculate accurate flight statistics from telemetry frames."""
    if not frames:
        return {
            'duration_seconds': 0,
            'total_distance_meters': 0,
            'max_altitude_meters': 0,
            'max_speed_mps': 0
        }

    # Duration from first and last frame
    first_osd = frames[0].get('osd', {})
    last_osd = frames[-1].get('osd', {})
    duration = last_osd.get('flyTime', 0) - first_osd.get('flyTime', 0)

    # Iterate through frames
    total_distance = 0
    max_altitude = 0
    max_speed = 0

    for i in range(len(frames)):
        osd = frames[i].get('osd', {})

        # Track max altitude
        height = osd.get('height', 0)
        if height > max_altitude:
            max_altitude = height

        # Track max horizontal speed
        x_speed = osd.get('xSpeed', 0)
        y_speed = osd.get('ySpeed', 0)
        h_speed = math.sqrt(x_speed**2 + y_speed**2)
        if h_speed > max_speed:
            max_speed = h_speed

        # Calculate distance between consecutive points
        if i > 0:
            prev_osd = frames[i-1].get('osd', {})
            lat1, lon1 = prev_osd.get('latitude'), prev_osd.get('longitude')
            lat2, lon2 = osd.get('latitude'), osd.get('longitude')

            if lat1 and lon1 and lat2 and lon2:
                distance = haversine_distance(lat1, lon1, lat2, lon2)
                total_distance += distance

    return {
        'duration_seconds': round(duration),
        'total_distance_meters': round(total_distance, 2),
        'max_altitude_meters': round(max_altitude, 2),
        'max_speed_mps': round(max_speed, 2)
    }

Phase 7: Timestamp Extraction (Lines 354-401)¶

Critical Issue: Some DJI logs have invalid telemetry timestamps (Unix epoch: 1970-01-01)

Extraction Strategy (with fallback chain):

Primary: Extract from telemetry custom.dateTime field
Format: ISO 8601 (2025-06-17T13:44:17.568Z)
Validation: Reject dates before 2013 (pre-consumer drones era)
Fallback: Extract from filename pattern
Pattern: DJIFlightRecord_YYYY-MM-DD_HH-MM-SS.txt
Regex: (\d{4})-(\d{2})-(\d{2})_(\d{2})-(\d{2})-(\d{2})
End time: Start time + duration from statistics
Last Resort: Use upload timestamp

Code Reference:

# app/routes_flight_logs.py:354-401
def extract_flight_timestamps(frames, statistics, raw_filepath):
    """Extract flight start and end times with fallback chain."""
    try:
        # Get timestamps from first and last frames
        first_frame = frames[0] if frames else {}
        last_frame = frames[-1] if frames else {}

        first_datetime_str = first_frame.get('custom', {}).get('dateTime', '')
        last_datetime_str = last_frame.get('custom', {}).get('dateTime', '')

        flight_start = None
        flight_end = None

        if first_datetime_str and last_datetime_str:
            # Parse ISO 8601 format timestamps
            flight_start = datetime.fromisoformat(first_datetime_str.replace('Z', '+00:00'))
            flight_end = datetime.fromisoformat(last_datetime_str.replace('Z', '+00:00'))

            # Validate timestamps - reject Unix epoch (1970) or unreasonably old dates
            cutoff_date = datetime(2013, 1, 1, tzinfo=timezone.utc)
            if flight_start < cutoff_date or flight_end < cutoff_date:
                app.logger.warning(f"Invalid timestamps in telemetry (before 2013), falling back to filename")
                flight_start = None
                flight_end = None

        # Fallback: Extract date from filename (format: DJIFlightRecord_2025-06-17_14-40-28.txt)
        if not flight_start or not flight_end:
            import re
            filename = os.path.basename(raw_filepath)
            match = re.search(r'(\d{4})-(\d{2})-(\d{2})_(\d{2})-(\d{2})-(\d{2})', filename)
            if match:
                year, month, day, hour, minute, second = map(int, match.groups())
                flight_start = datetime(year, month, day, hour, minute, second, tzinfo=timezone.utc)
                # Estimate end time using duration from statistics
                duration_seconds = statistics.get('duration_seconds', 0)
                flight_end = flight_start + timedelta(seconds=duration_seconds)
                app.logger.info(f"Extracted flight time from filename: {flight_start}")
            else:
                # Last resort: use current time
                flight_end = datetime.now(timezone.utc)
                flight_start = flight_end

    except (ValueError, TypeError, IndexError) as e:
        app.logger.error(f"Error parsing flight timestamps: {e}")
        flight_end = datetime.now(timezone.utc)
        flight_start = flight_end

    return flight_start, flight_end

Why This Matters: - Ensures accurate flight history chronology - Enables time-based filtering and sorting - Required for compliance reporting - Prevents "year 1970" display issues in UI

Phase 8: Statistics Validation (Lines 695-706)¶

Validation Bounds: - Duration: 0 < x < 24 hours - Distance: 0 ≤ x < 500 km - Altitude: 0 ≤ x < 500 m - Speed: 0 ≤ x < 30 m/s (~108 km/h)

Validation Types: - Errors: Hard limits that prevent database save - Warnings: Unusual values logged but allowed

Code Reference:

# app/routes_flight_logs.py:430-488
def validate_flight_statistics(stats, summary_details):
    """Validate calculated statistics against reasonable bounds."""
    warnings = []
    errors = []

    duration = stats.get('duration_seconds', 0)
    distance = stats.get('total_distance_meters', 0)
    altitude = stats.get('max_altitude_meters', 0)
    speed = stats.get('max_speed_mps', 0)

    # Validate duration (0 < x < 24 hours)
    if duration <= 0:
        errors.append("Duration must be greater than 0")
    elif duration > 86400:
        errors.append(f"Duration {duration}s exceeds 24 hours")

    # Validate altitude (x < 500m regulatory limit)
    if altitude < 0:
        errors.append("Altitude cannot be negative")
    elif altitude > 500:
        warnings.append(f"Altitude {altitude:.1f}m exceeds typical regulatory limit (500m)")

    # Validate speed (x < 30 m/s)
    if speed < 0:
        errors.append("Speed cannot be negative")
    elif speed > 30:
        warnings.append(f"Speed {speed:.1f}m/s exceeds typical drone maximum (30m/s)")

    # Cross-check with summary if available
    if summary_details:
        summary_duration = summary_details.get('totalTime')
        if summary_duration and abs(duration - summary_duration) / max(duration, 1) > 0.1:
            warnings.append(f"Calculated duration ({duration}s) differs from summary ({summary_duration}s) by >10%")

    return {
        'valid': len(errors) == 0,
        'warnings': warnings,
        'errors': errors
    }

Phase 9: Drone Matching (Lines 707-710)¶

Matching Strategy: 1. Extract aircraft serial number from parser summary 2. Query Drone table for matching serial 3. Use startswith() for fuzzy matching (logs may truncate serials) 4. Link first match only

Code Reference:

# app/routes_flight_logs.py:399-427
def match_drone_by_serial(aircraft_sn):
    """Match drone in database by serial number."""
    if not aircraft_sn:
        return None

    aircraft_sn_upper = aircraft_sn.upper()

    # Try to find drone by serial number
    # Note: Using startswith because the log serial may be truncated
    drones = Drone.query.all()
    for drone in drones:
        if drone.serial_number and drone.serial_number.upper().startswith(aircraft_sn_upper):
            return drone

    return None

Why Fuzzy Matching? - DJI logs sometimes truncate serial numbers - Database may have full serial while log has partial - First-match strategy prevents ambiguity

Phase 10: Database Update (Lines 711-713)¶

Final Update: - Set all calculated statistics - Update flight timestamps - Link to matched drone - Set parsing_status='success' - Set parsed_at timestamp

Code Reference:

# app/routes_flight_logs.py:491-541
def update_flight_log_with_parsed_data(flight_log, parse_result, drone):
    """Update FlightLog database record with parsed data."""
    stats = parse_result['statistics']

    # Update statistics
    flight_log.duration_seconds = stats['duration_seconds']
    flight_log.total_distance_meters = stats['total_distance_meters']
    flight_log.max_altitude_meters = stats['max_altitude_meters']
    flight_log.max_speed_mps = stats['max_speed_mps']

    # Update timestamps
    flight_log.flight_start_time = parse_result['flight_start']
    flight_log.flight_end_time = parse_result['flight_end']

    # Update photo and video counts
    summary_details = parse_result['summary'].get('details', {})
    flight_log.photo_count = summary_details.get('photoNum', 0)
    flight_log.video_duration_seconds = summary_details.get('videoTime', 0)

    # Set drone if matched
    if drone:
        flight_log.drone_id = drone.droneID

    # Set parser version
    flight_log.parser_version = parse_result['summary'].get('version', 'unknown')

    # Mark as successfully parsed
    flight_log.parsing_status = 'success'
    flight_log.parsed_at = datetime.now(timezone.utc)

Phase 11: Error Handling (Lines 720-743)¶

Parsing Errors: - Status set to parsing_status='failed' - Error message stored in parsing_error field - Database record retained (for debugging) - Files retained (for manual reprocessing)

File Processing Errors: - Database transaction rolled back - Files deleted from disk - Error returned to user

Code Reference:

# app/routes_flight_logs.py:720-743
except Exception as parse_error:
    app.logger.error(f"Parse error for {original_filename}: {str(parse_error)}")
    flight_log.parsing_status = 'failed'
    flight_log.parsing_error = str(parse_error)
    flight_log.parsed_at = datetime.now(timezone.utc)
    db.session.commit()

    result["error"] = f"Parsing failed: {str(parse_error)}"
    failed += 1

except Exception as e:
    db.session.rollback()
    app.logger.error(f"Error processing {file.filename}: {str(e)}")

    # Clean up file if it was saved
    if 'file_info' in locals() and os.path.exists(file_info['raw_filepath']):
        try:
            os.remove(file_info['raw_filepath'])
        except:
            pass

    result["error"] = str(e)
    failed += 1

Database Schema¶

FlightLog Table¶

File Information: - id (PK): Auto-increment primary key - original_filename: User's filename (e.g., DJIFlightRecord_2025-06-17_14-40-28.txt) - raw_log_filename: Stored filename with UUID (1_ec285d53-3115-4c7d-a962-0f02fe71c0ca_DJIFlightRecord_2025-06-17_14-40-28.txt) - raw_log_filepath: Full path to raw log (uploads/flight_logs/raw/...) - csv_filename: Generated CSV filename - csv_filepath: Full path to CSV (uploads/flight_logs/csv/...) - file_size: Size in bytes - file_hash: SHA256 hash (indexed for duplicate detection)

User-Provided Metadata: - description: Optional text description (required if no project association)

Flight Metadata (extracted from parser): - drone_id (FK → Drone): Matched drone (nullable) - flight_start_time: Flight start datetime (UTC) - flight_end_time: Flight end datetime (UTC) - duration_seconds: Total flight time - total_distance_meters: Cumulative horizontal distance traveled - max_altitude_meters: Maximum height above takeoff point - max_speed_mps: Maximum horizontal speed (meters per second) - photo_count: Number of photos taken - video_duration_seconds: Total video recording time

Parsing Metadata: - parser_version: DJI log format version (from parser) - parsing_status: 'pending', 'success', or 'failed' - parsing_error: Error message if parsing failed

Timestamps: - uploaded_at: When user uploaded file - parsed_at: When parsing completed

Relationships: - user_id (FK → User, required): Owner - project_id (FK → Project, optional): Associated project

Indexes¶

file_hash: For fast duplicate detection
user_id: For filtering user's logs
project_id: For filtering project logs
flight_start_time: For chronological sorting

Helper Functions Reference¶

Validation Functions¶

allowed_log_file(filename) - Check file extension (only .txt allowed) - Returns: bool

validate_flight_log_file(file, check_size=True) - Comprehensive file validation - Returns: (is_valid: bool, error_message: str | None, file_size: int)

validate_flight_statistics(stats, summary_details) - Validate against physical constraints - Returns: {'valid': bool, 'warnings': list, 'errors': list}

File Operations¶

calculate_file_hash(file) - Compute SHA256 hash of file content - Resets file pointer after hashing - Returns: str (hex digest)

check_duplicate_log(user_id, file_hash) - Query for existing log with same hash - Returns: FlightLog | None

save_flight_log_file(file, user_id, original_filename) - Save file to disk with unique naming - Create directories if needed - Returns: dict with filenames and paths

Database Operations¶

create_flight_log_record(user_id, project_id, description, file_info, file_size, file_hash) - Create pending FlightLog record - Add to session (does NOT commit) - Returns: FlightLog instance

update_flight_log_with_parsed_data(flight_log, parse_result, drone) - Update record with parsed data - Set status to 'success' - Does NOT commit

Parsing Functions¶

parse_and_extract_flight_data(raw_filepath, csv_filepath) - Wrapper around DJI parser - Extract timestamps, calculate statistics - Returns: dict with success flag and all extracted data

calculate_flight_statistics(frames) - Calculate duration, distance, max altitude, max speed - Uses Haversine formula for GPS distances - Returns: dict with statistics

haversine_distance(lat1, lon1, lat2, lon2) - Calculate great-circle distance between GPS coordinates - Returns: float (meters)

Matching Functions¶

match_drone_by_serial(aircraft_sn) - Fuzzy match drone by serial number - Uses startswith() for partial matches - Returns: Drone | None

Error Handling & Edge Cases¶

Known Edge Cases¶

1. Invalid Telemetry Timestamps (Unix Epoch)¶

Problem: Some DJI logs have dateTime: "1970-01-01T00:00:00Z" in telemetry

Solution: Validate timestamps against cutoff date (2013-01-01), fall back to filename extraction

Code Location: routes_flight_logs.py:372-391

Example:

Filename: DJIFlightRecord_2025-06-17_14-40-28.txt
Telemetry dateTime: 1970-01-01T00:00:00Z (INVALID)
Fallback: Extract 2025-06-17 14:40:28 from filename

2. Truncated Serial Numbers¶

Problem: DJI logs may truncate aircraft serial numbers

Solution: Use startswith() matching instead of exact match

Code Location: routes_flight_logs.py:424

Example:

Database: 1581F6GKB2396004K7L
Log SN:   1581F6GKB2396004
Match:    True (using startswith)

3. Missing GPS Coordinates¶

Problem: Some telemetry frames may lack GPS data (indoor flight, GPS loss)

Solution: Skip frames with None values in distance calculation

Code Location: routes_flight_logs.py:105

Example:

if lat1 and lon1 and lat2 and lon2:
    distance = haversine_distance(lat1, lon1, lat2, lon2)
    total_distance += distance

4. Duplicate Uploads¶

Problem: Users may accidentally upload same file multiple times

Solution: SHA256 hash-based duplicate detection before processing

Code Location: routes_flight_logs.py:661-668

User Feedback:

"Duplicate file (already uploaded as 'DJIFlightRecord_2025-06-17_14-40-28.txt' on 2025-06-17 13:45)"

5. Parsing Failures¶

Problem: Corrupted logs, unsupported format versions, DJI API errors

Solution: - Set parsing_status='failed' - Store error message in parsing_error field - Retain files and database record for debugging - Continue processing remaining files in batch

Code Location: routes_flight_logs.py:720-729

6. Large File Processing¶

Problem: Large logs (50MB) may timeout during parsing

Solution: - Set upload timeout to 30 seconds (configurable) - Process files sequentially (not parallel) to avoid memory issues - Chunk-based file reading for hash calculation

Configuration: MAX_LOG_FILE_SIZE = 50 * 1024 * 1024 (routes_flight_logs.py:17)

Security Considerations¶

File Upload Security¶

Extension Whitelist: Only .txt files accepted
Size Limit: 50MB maximum per file
Unique Naming: UUID-based names prevent overwriting
User Isolation: Files prefixed with user_id

Access Control¶

Authorisation Checks (FlightLog model methods): - can_access(): View log (owner, admin, or project member) - can_delete(): Delete log (owner, admin, or project editor)

Code Location: app/models.py:560-578

Path Traversal Prevention¶

Use secure_filename() from Werkzeug
Store files in dedicated directory tree
No user-controlled path components

Duplicate Detection¶

Hash-based deduplication per user
Prevents storage exhaustion attacks
Early rejection (before file save)

Performance Considerations¶

Batch Processing¶

Sequential Processing: - Process one file at a time - Prevents memory exhaustion - Allows partial success (some files succeed, others fail)

Transaction Strategy: - Atomic per file (commit after each success) - Rollback on individual file failure - Continue processing remaining files

Database Optimization¶

Indexes: - file_hash (duplicate lookup) - user_id (user filtering) - flight_start_time (chronological sorting)

Eager Loading (prevent N+1 queries):

FlightLog.query.options(
    db.joinedload(FlightLog.user),
    db.joinedload(FlightLog.drone),
    db.joinedload(FlightLog.project)
).all()

File Storage¶

Directory Structure:

uploads/
└── flight_logs/
    ├── raw/          # Original .txt files
    │   ├── 1_uuid_file1.txt
    │   ├── 1_uuid_file2.txt
    │   └── 2_uuid_file3.txt
    └── csv/          # Generated CSV exports
        ├── 1_uuid_file1.csv
        ├── 1_uuid_file2.csv
        └── 2_uuid_file3.csv

Benefits: - Fast lookup by filename - Easy backup/restore - Scalable (no directory size limits)

Testing Strategy¶

Unit Tests¶

Test Targets: - calculate_flight_statistics() - Various telemetry inputs - haversine_distance() - Known GPS coordinate pairs - validate_flight_statistics() - Boundary conditions - match_drone_by_serial() - Exact and partial matches - calculate_file_hash() - Hash consistency

Test File: tests/unit/test_flight_log_helpers.py

Integration Tests¶

Test Scenarios: 1. Successful upload - Valid log, all processing succeeds 2. Duplicate detection - Upload same file twice 3. Invalid file - Wrong extension, too large, empty 4. Parsing failure - Corrupted log format 5. Timestamp fallback - Invalid telemetry, extract from filename 6. Drone matching - Serial number matching (exact and fuzzy) 7. Project association - Upload to project vs standalone

Test File: tests/integration/test_flight_logs.py

Test Fixtures¶

Factory Pattern (using Factory Boy):

# tests/fixtures/factories.py:199-260
class FlightLogFactory(BaseFactory):
    class Meta:
        model = FlightLog

    original_filename = factory.Sequence(lambda n: f'DJIFlightRecord_{n}.txt')
    raw_log_filename = factory.LazyAttribute(lambda obj: f'{uuid.uuid4()}_{obj.original_filename}')
    flight_start_time = factory.Faker('date_time_between', start_date='-30d', end_date='now', tzinfo=timezone.utc)
    duration_seconds = fuzzy.FuzzyInteger(60, 3600)
    # ... additional fields

Sample Data¶

Real DJI Logs (for integration testing): - Place in tests/fixtures/flight_logs/ - Include both valid and edge-case logs (invalid timestamps, truncated serials, missing GPS)

Audit Logging¶

Logged Events: - upload_flight_log: Batch upload with success/fail counts - delete_flight_log: File deletion with filename

Code Locations: - Upload: routes_flight_logs.py:747-751 - Delete: routes_flight_logs.py:833-837

Audit Entry Example:

action: 'upload_flight_log'
message: 'Batch uploaded 3/5 flight logs'
user_id: 1
timestamp: 2025-06-17 13:45:22 UTC

Future Enhancements¶

Potential Improvements¶

Async Processing
Use Celery/Redis for background parsing
Immediate upload confirmation, parsing runs asynchronously
Progress tracking via WebSocket
Advanced Telemetry Analysis
3D flight path visualisation
Battery consumption modeling
Wind speed estimation from GPS track deviations
Compliance Reporting
Automated generation of CAA-compliant flight logs
PDF export with flight path maps
Integration with digital logbook services
Predictive Maintenance
Track flight hours per drone
Alert when maintenance intervals approach
Battery health monitoring (voltage trends)
Geofence Violation Detection
Check flight path against airspace restrictions
Flag flights in controlled airspace
Integration with NOTAM system
Batch Reprocessing
Admin tool to reprocess failed logs
Bulk re-matching after drone serial updates
Statistics recalculation after algorithm improvements

API Reference¶

Endpoints¶

`POST /dashboard/flight-logs/upload-batch`¶

Description: Upload multiple DJI flight log files

Authentication: Required (@login_required)

Request: - Content-Type: multipart/form-data - Fields: - files[]: Array of .txt files (max 25) - project_id: Integer (optional) - description: String (required if no project_id)

Response:

{
  "results": [
    {
      "filename": "DJIFlightRecord_2025-06-17_14-40-28.txt",
      "status": "success",
      "log_id": 123,
      "message": "Parsed successfully",
      "error": null
    },
    {
      "filename": "DJIFlightRecord_2025-06-17_14-44-17.txt",
      "status": "failed",
      "log_id": null,
      "message": null,
      "error": "Parsing failed: Invalid log format"
    }
  ],
  "summary": {
    "total": 2,
    "succeeded": 1,
    "failed": 1
  }
}

Status Codes: - 200 OK: All files processed (check individual statuses) - 400 Bad Request: Invalid request (no files, metadata missing, file count exceeded) - 403 Forbidden: Project access denied - 404 Not Found: Project not found

`GET /dashboard/flight-logs`¶

Description: List user's flight logs (paginated)

Query Parameters: - page: Page number (default: 1) - project_id: Filter by project (optional)

Response: HTML page with paginated log list

`GET /dashboard/flight-logs/<int:log_id>`¶

Description: View detailed log information

Response: HTML page with: - Flight statistics - Telemetry summary - Download links (raw .txt, CSV) - Project association

`GET /dashboard/flight-logs/<int:log_id>/download-csv`¶

Description: Download CSV telemetry export

Response: CSV file

`GET /dashboard/flight-logs/<int:log_id>/download-raw`¶

Description: Download original raw log file

Response: .txt file

`POST /dashboard/flight-logs/<int:log_id>/delete`¶

Description: Delete flight log and associated files

Response: Redirect to flight logs list

Troubleshooting Guide¶

Common Issues¶

Issue: "Parsing failed: Failed to parse flight summary"¶

Cause: DJI parser cannot read log file

Solutions: 1. Check DJI_API_KEY environment variable is set 2. Verify file is valid DJI format (not corrupted) 3. Check parser version compatibility 4. Review application logs for detailed error

Issue: "Invalid timestamps in telemetry (before 2013), falling back to filename"¶

Cause: DJI log has Unix epoch timestamps (1970-01-01)

Impact: Timestamps extracted from filename instead (less precise)

Solution: This is expected behaviour for certain log formats. No action needed.

Issue: Flight log shows "1970-01-01" start time¶

Cause: Telemetry timestamp extraction failed AND filename doesn't match pattern

Solution: 1. Verify filename follows pattern: DJIFlightRecord_YYYY-MM-DD_HH-MM-SS.txt 2. Check application logs for timestamp extraction errors 3. Manually correct timestamp in database if needed

Issue: Drone not matched to log¶

Cause: Serial number mismatch between log and database

Solutions: 1. Check drone serial number in database (Admin → Drones) 2. Verify log contains aircraft serial (parse summary) 3. Add/update drone serial in database 4. Reprocess log (admin feature needed)

Issue: "Duplicate file" error but file was deleted¶

Cause: Database record exists but file was manually deleted

Solution: Delete FlightLog database record via admin panel

Issue: Upload fails with timeout¶

Cause: Large file (>50MB) or slow parsing

Solutions: 1. Check file size (must be < 50MB) 2. Increase upload timeout in Nginx/Apache config 3. Consider splitting large logs

Maintenance Tasks¶

Periodic Cleanup¶

Orphaned Files (files without database records):

# Find files older than 30 days not in database
find uploads/flight_logs/raw -type f -mtime +30 -name "*.txt"

Failed Logs (parsing_status='failed'):

-- Delete logs failed >90 days ago
DELETE FROM flight_log
WHERE parsing_status = 'failed'
AND parsed_at < DATE_SUB(NOW(), INTERVAL 90 DAY);

Database Maintenance¶

Rebuild Statistics (if calculation algorithm changes):

# Script: scripts/rebuild_flight_stats.py
from app import app, db
from app.models import FlightLog
from app.routes_flight_logs import parse_and_extract_flight_data, update_flight_log_with_parsed_data

with app.app_context():
    logs = FlightLog.query.filter_by(parsing_status='success').all()
    for log in logs:
        parse_result = parse_and_extract_flight_data(log.raw_log_filepath, log.csv_filepath)
        if parse_result['success']:
            update_flight_log_with_parsed_data(log, parse_result, log.drone)
            db.session.commit()

Monitoring¶

Key Metrics: - Upload success rate (succeeded / total) - Parsing failure rate (parsing_status='failed' / total) - Average processing time per file - Storage usage (uploads/flight_logs/ directory size)

Log Monitoring:

# Check for parsing errors
grep "Parse error" logs/application.log

# Check for timestamp fallback warnings
grep "Invalid timestamps in telemetry" logs/application.log

Configuration¶

Environment Variables¶

Required: - DJI_API_KEY: API key for DJI parser (obtain from DJI developer portal)

Optional: - MAX_CONTENT_LENGTH: Max upload size in bytes (default: Flask default)

Application Settings¶

File: app/routes_flight_logs.py:14-18

FLIGHT_LOG_RAW_FOLDER = 'uploads/flight_logs/raw/'
FLIGHT_LOG_CSV_FOLDER = 'uploads/flight_logs/csv/'
ALLOWED_LOG_EXTENSIONS = {'txt'}
MAX_LOG_FILE_SIZE = 50 * 1024 * 1024  # 50MB
LOGS_PER_PAGE = 25

Database Configuration¶

Connection Pool (for concurrent uploads):

# app/__init__.py
'pool_size': 10,
'pool_recycle': 3600,
'pool_pre_ping': True,
'max_overflow': 20,

References¶

External Documentation¶

DJI Parser: app/utils/dji_parser/README.md
DJI Parser Quickstart: app/utils/dji_parser/QUICKSTART.md
Flask-Login: https://flask-login.readthedocs.io/
SQLAlchemy: https://docs.sqlalchemy.org/

Project Design: docs/design/
API Documentation: docs/api/
CLAUDE.md: Developer reference for codebase patterns

Code References¶

Primary Files: - app/routes_flight_logs.py (847 lines) - Main implementation - app/models.py:498-578 - FlightLog model - app/utils/dji_parser/ - External parser library

Test Files: - tests/unit/test_flight_log_helpers.py - tests/integration/test_flight_logs.py - tests/fixtures/factories.py:199-260 - FlightLogFactory

Document History¶

Version	Date	Author	Changes
1.0	2026-01-08	System	Initial documentation created

Last Updated: 2026-01-08 Maintained By: Development Team Status: Production

Flight Log Import System - Design Document¶

Executive Summary¶

Overview¶

Business Value¶

Key Capabilities¶

Architecture Overview¶

System Components¶

File: app/routes_flight_logs.py¶

File: app/models.py¶

External Dependency: DJI Parser¶

Processing Pipeline¶

Phase 1: Upload & Validation (Lines 592-636)¶

Phase 2: Duplicate Detection (Lines 656-668)¶

Phase 3: File Storage (Lines 670-673)¶

Phase 4: Database Record Creation (Lines 674-683)¶

Phase 5: Log Parsing & Data Extraction (Lines 686-693)¶

Phase 6: Statistics Calculation (Lines 53-114)¶

Phase 7: Timestamp Extraction (Lines 354-401)¶

Phase 8: Statistics Validation (Lines 695-706)¶

Phase 9: Drone Matching (Lines 707-710)¶

Phase 10: Database Update (Lines 711-713)¶

Phase 11: Error Handling (Lines 720-743)¶

Database Schema¶

FlightLog Table¶

Indexes¶

Helper Functions Reference¶

Validation Functions¶

File Operations¶

Database Operations¶

Parsing Functions¶

Matching Functions¶

Error Handling & Edge Cases¶

Known Edge Cases¶

1. Invalid Telemetry Timestamps (Unix Epoch)¶

2. Truncated Serial Numbers¶

3. Missing GPS Coordinates¶

4. Duplicate Uploads¶

5. Parsing Failures¶

6. Large File Processing¶

Security Considerations¶

File Upload Security¶

Access Control¶

Path Traversal Prevention¶

Duplicate Detection¶

Performance Considerations¶

Batch Processing¶

Database Optimization¶

File Storage¶

Testing Strategy¶

Unit Tests¶

Integration Tests¶

Test Fixtures¶

Sample Data¶

Audit Logging¶

Future Enhancements¶

Potential Improvements¶

API Reference¶

Endpoints¶

POST /dashboard/flight-logs/upload-batch¶

GET /dashboard/flight-logs¶

GET /dashboard/flight-logs/<int:log_id>¶

GET /dashboard/flight-logs/<int:log_id>/download-csv¶

GET /dashboard/flight-logs/<int:log_id>/download-raw¶

POST /dashboard/flight-logs/<int:log_id>/delete¶

Troubleshooting Guide¶

Common Issues¶

Issue: "Parsing failed: Failed to parse flight summary"¶

Issue: "Invalid timestamps in telemetry (before 2013), falling back to filename"¶

Issue: Flight log shows "1970-01-01" start time¶

Issue: Drone not matched to log¶

Issue: "Duplicate file" error but file was deleted¶

Issue: Upload fails with timeout¶

Maintenance Tasks¶

Periodic Cleanup¶

Database Maintenance¶

Monitoring¶

Configuration¶

Environment Variables¶

Application Settings¶

Database Configuration¶

File: `app/routes_flight_logs.py`¶

File: `app/models.py`¶

`POST /dashboard/flight-logs/upload-batch`¶

`GET /dashboard/flight-logs`¶

`GET /dashboard/flight-logs/<int:log_id>`¶

`GET /dashboard/flight-logs/<int:log_id>/download-csv`¶

`GET /dashboard/flight-logs/<int:log_id>/download-raw`¶

`POST /dashboard/flight-logs/<int:log_id>/delete`¶