Enhanced File Streaming (Large 2GB) & MIME Type Management

[mdaneri]

Overview

This PR introduces significant improvements to Pode's file handling capabilities and adds a comprehensive MIME type management system. The changes enable support for large file streaming (>2GB), implement HTTP range requests, and provide a robust MIME type registry with PowerShell cmdlets.

🔧 Major Changes

1. Enhanced PodeResponse.cs - Large File Streaming Support

Breaking the 2GB Barrier:

• Added MAX_IN_MEMORY_FILE_SIZE constant (64 MiB) to intelligently handle file buffering
• Implemented WriteStreamAsync() method for efficient streaming of large files
• Added WriteLargeStream() method with HTTP range request support
• Introduced StreamSectionAsync() for chunked streaming using ArrayPool for memory efficiency

New File Handling Methods:

• WriteFileAsync(string path, long[] ranges, PodeCompressionType compression) - Async file streaming with range support
• WriteFileAsync(FileSystemInfo file, long[] ranges, PodeCompressionType compression) - Enhanced file streaming
• WriteByteAsync(byte[] bytes, long[] ranges, PodeCompressionType compression) - Byte array streaming
• WriteStringAsync(string content, Encoding encoding, long[] ranges, PodeCompressionType compression) - String content streaming

HTTP Range Request Support:

• Single range requests (206 Partial Content)
• Multi-range requests (206 multipart/byteranges)
• Invalid range handling (416 Range Not Satisfiable)
• Proper boundary generation for multipart responses

Performance Improvements:

• Uses ArrayPool<byte> for efficient memory management
• 64 KiB buffer sizes for optimal streaming performance
• Automatic timeout cancellation for long-running transfers
• Smart buffering strategy (small files in memory, large files streamed)

2. New PodeMimeTypes.cs - Comprehensive MIME Type Registry

Core Features:

• Case-insensitive file extension to MIME type mapping
• Extensive built-in MIME type database (400+ types)
• Thread-safe operations using Dictionary<string, string>
• Support for modern file types (WebP, AVIF, WASM, etc.)

Key Methods:

• Get(string extension) - Retrieve MIME type with fallback to "application/octet-stream"
• TryGet(string extension, out string contentType) - Safe MIME type retrieval
• Add(string extension, string contentType) - Add new mappings (throws if exists)
• AddOrUpdate(string extension, string contentType) - Upsert operation
• Remove(string extension) - Remove mappings
• Contains(string extension) - Check if mapping exists
• LoadFromFile(string path) - Bulk load from Apache-style mime.types files
• IsTextualMimeType(string type) - Identify text-based content types

3. PowerShell MIME Type Management (Mime.ps1)

User-Friendly Cmdlets:

# Add new MIME type mapping
Add-PodeMimeType -Extension '.json' -MimeType 'application/json'

# Update existing mapping
Set-PodeMimeType -Extension '.json' -MimeType 'application/vnd.api+json'

# Get MIME type for extension
$mimeType = Get-PodeMimeType -Extension '.pdf'

# Test if mapping exists
if (Test-PodeMimeType -Extension '.myext') { ... }

# Remove mapping
Remove-PodeMimeType -Extension '.oldext'

# Bulk import from file
Import-PodeMimeTypeFromFile -Path './custom-mime-types.txt'

🔄 Enhanced Route Pipeline Functions

Add-PodeRouteCache - Advanced HTTP Caching Configuration

This function provides comprehensive HTTP caching control for Pode routes, supporting modern caching strategies with ETag generation and fine-grained Cache-Control directives.

Key Parameters:

• -Enable: Activates caching for the route(s)
• -Disable: Completely disables caching, overriding any existing settings
• -Visibility: Controls cache visibility ('public', 'private', 'no-cache', 'no-store')
• -MaxAge: Sets max-age directive in seconds for client-side caching
• -SharedMaxAge: Sets s-maxage directive for shared/proxy caches
• -MustRevalidate: Forces revalidation when cache expires
• -Immutable: Indicates resource will never change (useful for versioned assets)
• -ETagMode: Controls ETag generation strategy:
  • 'None': Disables ETag generation
  • 'Auto': Intelligent selection ('mtime' for static files, 'hash' for dynamic content)
  • 'Hash': Content-based hash ETags (CPU intensive but accurate)
  • 'Mtime': File modification time ETags (fast but less accurate)
  • 'Manual': Allows manual ETag setting via response cmdlets
• -WeakValidation: Generates weak ETags (prefixed with W/)
• -PassThru: Returns modified routes for pipeline chaining

Practical Examples:

# Basic caching with 1-hour expiration
$routes = Get-PodeStaticRoute -Path '/assets/*' -Source './public/assets'
$routes | Add-PodeRouteCache -Enable -MaxAge 3600 -Visibility public -ETagMode auto

# Aggressive caching for versioned assets (never changes)
Get-PodeStaticRoute -Path '/assets/v*/js/*' -Source './public/assets' |
    Add-PodeRouteCache -Enable -MaxAge 31536000 -Immutable -ETagMode hash -Visibility public

# API responses with validation requirements
Add-PodeRoute -Method Get -Path '/api/data' -ScriptBlock { ... } -PassThru |
    Add-PodeRouteCache -Enable -MaxAge 300 -MustRevalidate -ETagMode manual -Visibility private

# Different caching for CDN vs browsers
$apiRoutes = @(
    (Add-PodeRoute -Method Get -Path '/api/public/*' -ScriptBlock { ... } -PassThru)
)
$apiRoutes | Add-PodeRouteCache -Enable -MaxAge 600 -SharedMaxAge 3600 -ETagMode auto

# Disable caching for sensitive content
Get-PodeRoute -Method Get -Path '/admin/*' |
    Add-PodeRouteCache -Disable

# Conditional caching based on content type
$imageRoutes = Get-PodeStaticRoute -Path '/images/*' -Source './images'
$imageRoutes | Add-PodeRouteCache -Enable -MaxAge 86400 -ETagMode mtime -Visibility public -WeakValidation

# Chain multiple route modifications
Add-PodeStaticRoute -Method Get -Path '/downloads/*' -Source './files' -PassThru |
    Add-PodeRouteCache -Enable -MaxAge 1800 -ETagMode hash -PassThru |
    Add-PodeRouteCompression -Enable -Encoding gzip,br

Cache-Control Header Examples:

# Basic public caching
Cache-Control: public, max-age=3600
ETag: "abc123def456"

# Immutable versioned assets
Cache-Control: public, max-age=31536000, immutable
ETag: "hash-of-content-xyz789"

# Private API with revalidation
Cache-Control: private, max-age=300, must-revalidate
ETag: W/"manual-etag-value"

# Different TTLs for proxies vs clients
Cache-Control: public, max-age=600, s-maxage=3600
ETag: "auto-generated-etag"

Add-PodeRouteCompression - Advanced Response Compression

This function configures response compression for Pode routes, supporting modern compression algorithms and directional control.

Key Parameters:

• -Enable: Activates compression for the route(s)
• -Disable: Completely disables compression
• -Encoding: Specifies compression algorithms ('gzip', 'deflate', 'br' for Brotli)
• -Direction: Controls compression direction:
  • 'Response' (default): Compress outgoing responses
  • 'Request': Decompress incoming requests
  • 'Both': Compress responses and decompress requests
• -PassThru: Returns modified routes for pipeline chaining

Practical Examples:

# Basic gzip compression for API responses
Add-PodeRoute -Method Get -Path '/api/*' -ScriptBlock { ... } -PassThru |
    Add-PodeRouteCompression -Enable -Encoding gzip

# Modern compression with Brotli fallback
$staticRoutes = Get-PodeStaticRoute -Path '/assets/*' -Source './public'
$staticRoutes | Add-PodeRouteCompression -Enable -Encoding br,gzip,deflate

# Request/Response compression for upload endpoints
Add-PodeRoute -Method Post -Path '/api/upload' -ScriptBlock {
    # Handle compressed uploads and return compressed responses
} -PassThru | Add-PodeRouteCompression -Enable -Direction Both -Encoding gzip

# Selective compression based on content type
$jsonRoutes = Get-PodeRoute -Method Get -Path '/api/data/*'
$jsonRoutes | Add-PodeRouteCompression -Enable -Encoding br,gzip

# Disable compression for already-compressed content
Get-PodeStaticRoute -Path '/images/*' -Source './images' |
    Add-PodeRouteCompression -Disable

# High-performance text content compression
$textRoutes = @(
    (Add-PodeRoute -Method Get -Path '/reports/*' -ScriptBlock { ... } -PassThru),
    (Get-PodeStaticRoute -Path '/docs/*' -Source './documentation')
)
$textRoutes | Add-PodeRouteCompression -Enable -Encoding br,gzip -Direction Response

# Chain with caching for optimal performance
Add-PodeStaticRoute -Method Get -Path '/dist/*' -Source './build' -PassThru |
    Add-PodeRouteCache -Enable -MaxAge 86400 -ETagMode hash -Immutable -PassThru |
    Add-PodeRouteCompression -Enable -Encoding br,gzip,deflate

Compression Priority & Browser Support:

# Brotli (best compression) -> Gzip (universal) -> Deflate (fallback)
$routes | Add-PodeRouteCompression -Enable -Encoding br,gzip,deflate

# Request headers: Accept-Encoding: br, gzip, deflate
# Response headers: Content-Encoding: br (if supported)

Advanced Pipeline Combinations:

# Complete static asset optimization pipeline
function Optimize-StaticAssets {
    param($Path, $Source, $MaxAge = 86400)
    
    Add-PodeStaticRoute -Method Get -Path $Path -Source $Source -PassThru |
        Add-PodeRouteCache -Enable -MaxAge $MaxAge -ETagMode auto -Visibility public -PassThru |
        Add-PodeRouteCompression -Enable -Encoding br,gzip -PassThru
}

# Usage
Optimize-StaticAssets -Path '/css/*' -Source './public/css' -MaxAge 604800
Optimize-StaticAssets -Path '/js/*' -Source './public/js' -MaxAge 604800
Optimize-StaticAssets -Path '/fonts/*' -Source './public/fonts' -MaxAge 2592000

# API optimization with different strategies
$publicApiRoutes = Get-PodeRoute -Method Get -Path '/api/public/*'
$publicApiRoutes |
    Add-PodeRouteCache -Enable -MaxAge 300 -ETagMode auto -Visibility public -PassThru |
    Add-PodeRouteCompression -Enable -Encoding gzip,deflate

$privateApiRoutes = Get-PodeRoute -Method Get -Path '/api/private/*'
$privateApiRoutes |
    Add-PodeRouteCache -Enable -MaxAge 60 -ETagMode manual -Visibility private -PassThru |
    Add-PodeRouteCompression -Enable -Encoding gzip

# Upload endpoints with request decompression
Add-PodeRoute -Method Post -Path '/api/bulk-upload' -ScriptBlock {
    # Automatically handles gzip-compressed request bodies
    $data = $WebEvent.Data
    # Process large datasets efficiently
} -PassThru | Add-PodeRouteCompression -Enable -Direction Both -Encoding gzip

Performance Considerations:

# For high-traffic sites: Pre-compress static assets
$preCompressedRoutes = Get-PodeStaticRoute -Path '/static/*' -Source './precompressed'
$preCompressedRoutes | Add-PodeRouteCompression -Disable  # Already compressed

# For dynamic content: Use appropriate compression levels
$heavyApiRoutes = Get-PodeRoute -Method Get -Path '/api/reports/*'
$heavyApiRoutes | Add-PodeRouteCompression -Enable -Encoding gzip  # Faster than Brotli

# For real-time APIs: Minimal or no compression
$realtimeRoutes = Get-PodeRoute -Method Get -Path '/api/realtime/*'
$realtimeRoutes | Add-PodeRouteCompression -Disable  # Latency over bandwidth

💡 Technical Highlights

Memory Efficiency:

• ArrayPool usage for reduced GC pressure
• Streaming approach for large files prevents memory exhaustion
• Smart buffering thresholds (64 MiB boundary)

HTTP Compliance:

• Proper Content-Range headers for partial content
• Multipart/byteranges support for multiple ranges
• Correct status codes (200, 206, 416)
• Chunked transfer encoding when appropriate

Backwards Compatibility:

• All existing APIs remain functional
• New methods provide enhanced functionality without breaking changes
• Graceful fallbacks for unsupported scenarios

🧪 Usage Examples

Large File Streaming:

// Stream a large video file with range support
await response.WriteFileAsync("/path/to/large-video.mp4", ranges);

// Stream with compression
await response.WriteStreamAsync(fileStream, fileSize, compression: PodeCompressionType.Gzip);

MIME Type Management:

# Configure custom MIME types during server startup
Add-PodeMimeType -Extension '.wasm' -MimeType 'application/wasm'
Set-PodeMimeType -Extension '.json' -MimeType 'application/vnd.api+json'

🔧 Files Changed

• src/Listener/PodeResponse.cs - Enhanced with large file streaming capabilities
• src/Listener/PodeMimeTypes.cs - New comprehensive MIME type registry
• src/Public/Mime.ps1 - New PowerShell cmdlets for MIME type management
• src/Public/Routes.ps1 - Enhanced route pipeline functions

🎯 Benefits

• Performance: Efficient handling of files >2GB without memory issues
• Standards Compliance: Full HTTP range request support
• Developer Experience: Intuitive PowerShell cmdlets for MIME type management
• Flexibility: Comprehensive MIME type database with extensibility
• Reliability: Proper error handling and resource cleanup

This enhancement positions Pode as a robust solution for serving large media files, supporting modern web applications, and providing enterprise-grade file streaming capabilities.