Skip to content

Conversation

@AnirudhBharatiya
Copy link

Description

This pull request introduces centralized, structured, rotating file-based logging for both the Flask web application and the Celery worker. It ensures that logs are persisted across restarts, consistently formatted, and protected from unbounded growth, significantly improving observability and debuggability.


Changes

1. Introduce Centralized Logging Configuration

Issue:
Logging was configured inconsistently across components, with no shared setup between the web application and the Celery worker. Log files could grow indefinitely and were not guaranteed to persist outside containers.

Fix:
Added a new logging_config.py module that provides a reusable setup_logging() function. This function:

  • Configures a RotatingFileHandler (10 MB max size, 5 backups)
  • Applies a structured log format (timestamp level logger module message)
  • Attaches both file and stdout handlers
  • Sets appropriate log levels for common third-party libraries

2. Enable File-Based Logging for the Web Application

Issue:
Flask and Gunicorn logs were not consistently written to a persistent file, making debugging difficult after container restarts.

Fix:
Updated app.py to initialize logging via setup_logging(app) after loading configuration.
This ensures all application-level logs are written to the rotating log file.


3. Enable File-Based Logging for Celery Workers

Issue:
Asynchronous task execution and failures were not reliably logged to disk, limiting visibility into background processing errors.

Fix:
Updated celeryWorker.py to apply the same logging configuration during worker startup.
Celery task logs now use the shared rotating log file and consistent formatting.


4. Persist Logs via Docker Volumes

Issue:
Logs generated inside containers were not persisted on the host filesystem.

Fix:
Updated docker-compose.yml to mount a shared ./logs directory into both the web and worker containers (/app/logs), ensuring logs survive container restarts.


Impact

Observability

All application and background task logs are now persisted, structured, and centrally accessible, making debugging and monitoring significantly easier.

Reliability

Log rotation prevents uncontrolled log growth, reducing the risk of disk exhaustion in long-running deployments.

Maintainability

A single, shared logging configuration ensures consistent behavior across components and simplifies future logging-related changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants