Logging in Django

Logging is essential for monitoring, debugging, and maintaining Django applications. Django's logging framework, built on Python's logging module, provides flexible and powerful tools for capturing and managing application logs across development and production environments.

Logging in Django

Understanding Django Logging

Django uses Python's standard logging module with pre-configured loggers for different components. Understanding the logging architecture helps you effectively capture and manage application events.

Logging Components

Loggers: Entry points for log messages Handlers: Determine where log messages go (file, console, email, etc.) Filters: Provide fine-grained control over which log records are processed Formatters: Specify the layout of log messages

Default Django Loggers

Django provides several built-in loggers:

django: Root logger for all Django messages
django.request: HTTP request/response cycle
django.server: Development server messages
django.template: Template rendering issues
django.db.backends: Database query execution
django.security: Security-related messages

Basic Logging Configuration

Simple Configuration

# settings.py
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
            'style': '{',
        },
        'simple': {
            'format': '{levelname} {message}',
            'style': '{',
        },
    },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'simple',
        },
        'file': {
            'class': 'logging.FileHandler',
            'filename': 'django.log',
            'formatter': 'verbose',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['console', 'file'],
            'level': 'INFO',
            'propagate': True,
        },
    },
}

Production-Ready Configuration

# settings.py
import os

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '[{levelname}] {asctime} {name} {module}.{funcName}:{lineno} - {message}',
            'style': '{',
            'datefmt': '%Y-%m-%d %H:%M:%S',
        },
        'simple': {
            'format': '[{levelname}] {message}',
            'style': '{',
        },
        'json': {
            '()': 'pythonjsonlogger.jsonlogger.JsonFormatter',
            'format': '%(asctime)s %(name)s %(levelname)s %(message)s',
        },
    },
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse',
        },
        'require_debug_true': {
            '()': 'django.utils.log.RequireDebugTrue',
        },
    },
    'handlers': {
        'console': {
            'level': 'INFO',
            'filters': ['require_debug_true'],
            'class': 'logging.StreamHandler',
            'formatter': 'simple',
        },
        'file': {
            'level': 'INFO',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': os.path.join(BASE_DIR, 'logs', 'django.log'),
            'maxBytes': 1024 * 1024 * 15,  # 15MB
            'backupCount': 10,
            'formatter': 'verbose',
        },
        'error_file': {
            'level': 'ERROR',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': os.path.join(BASE_DIR, 'logs', 'django_errors.log'),
            'maxBytes': 1024 * 1024 * 15,  # 15MB
            'backupCount': 10,
            'formatter': 'verbose',
        },
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler',
            'include_html': True,
        },
    },
    'loggers': {
        'django': {
            'handlers': ['console', 'file'],
            'level': 'INFO',
            'propagate': False,
        },
        'django.request': {
            'handlers': ['error_file', 'mail_admins'],
            'level': 'ERROR',
            'propagate': False,
        },
        'django.db.backends': {
            'handlers': ['console'],
            'level': 'DEBUG' if DEBUG else 'INFO',
            'propagate': False,
        },
        'myapp': {
            'handlers': ['console', 'file', 'error_file'],
            'level': 'DEBUG' if DEBUG else 'INFO',
            'propagate': False,
        },
    },
    'root': {
        'handlers': ['console', 'file'],
        'level': 'INFO',
    },
}

Using Loggers in Code

Basic Logging

# views.py
import logging

logger = logging.getLogger(__name__)

def my_view(request):
    logger.debug('Debug message - detailed information')
    logger.info('Info message - general information')
    logger.warning('Warning message - something unexpected')
    logger.error('Error message - something failed')
    logger.critical('Critical message - serious problem')
    
    try:
        # Your code here
        result = perform_operation()
        logger.info(f'Operation completed successfully: {result}')
        return JsonResponse({'status': 'success', 'result': result})
    except Exception as e:
        logger.exception('Operation failed with exception')
        return JsonResponse({'status': 'error', 'message': str(e)}, status=500)

Structured Logging

# utils/logging.py
import logging
import json
from datetime import datetime

class StructuredLogger:
    def __init__(self, name):
        self.logger = logging.getLogger(name)
    
    def log_event(self, event_type, level='info', **kwargs):
        """Log structured event with metadata"""
        log_data = {
            'timestamp': datetime.utcnow().isoformat(),
            'event_type': event_type,
            **kwargs
        }
        
        log_message = json.dumps(log_data)
        
        if level == 'debug':
            self.logger.debug(log_message)
        elif level == 'info':
            self.logger.info(log_message)
        elif level == 'warning':
            self.logger.warning(log_message)
        elif level == 'error':
            self.logger.error(log_message)
        elif level == 'critical':
            self.logger.critical(log_message)
    
    def log_request(self, request, response_time=None, status_code=None):
        """Log HTTP request details"""
        self.log_event(
            'http_request',
            method=request.method,
            path=request.path,
            user=str(request.user),
            ip_address=self.get_client_ip(request),
            response_time=response_time,
            status_code=status_code
        )
    
    @staticmethod
    def get_client_ip(request):
        x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR')
        if x_forwarded_for:
            ip = x_forwarded_for.split(',')[0]
        else:
            ip = request.META.get('REMOTE_ADDR')
        return ip

# Usage
logger = StructuredLogger(__name__)

def my_view(request):
    start_time = time.time()
    
    try:
        # Process request
        result = process_request(request)
        
        response_time = time.time() - start_time
        logger.log_request(request, response_time=response_time, status_code=200)
        
        return JsonResponse(result)
    except Exception as e:
        response_time = time.time() - start_time
        logger.log_event('request_error', level='error',
                        error=str(e),
                        path=request.path,
                        response_time=response_time)
        return JsonResponse({'error': str(e)}, status=500)

Context Logging

# middleware.py
import logging
import uuid
from django.utils.deprecation import MiddlewareMixin

logger = logging.getLogger(__name__)

class RequestLoggingMiddleware(MiddlewareMixin):
    def process_request(self, request):
        # Generate unique request ID
        request.id = str(uuid.uuid4())
        request.start_time = time.time()
        
        logger.info(
            f'Request started',
            extra={
                'request_id': request.id,
                'method': request.method,
                'path': request.path,
                'user': str(request.user),
                'ip': self.get_client_ip(request)
            }
        )
    
    def process_response(self, request, response):
        if hasattr(request, 'start_time'):
            duration = time.time() - request.start_time
            
            logger.info(
                f'Request completed',
                extra={
                    'request_id': getattr(request, 'id', 'unknown'),
                    'method': request.method,
                    'path': request.path,
                    'status_code': response.status_code,
                    'duration': f'{duration:.3f}s'
                }
            )
        
        return response
    
    def process_exception(self, request, exception):
        logger.error(
            f'Request failed with exception',
            extra={
                'request_id': getattr(request, 'id', 'unknown'),
                'method': request.method,
                'path': request.path,
                'exception': str(exception)
            },
            exc_info=True
        )
    
    @staticmethod
    def get_client_ip(request):
        x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR')
        if x_forwarded_for:
            ip = x_forwarded_for.split(',')[0]
        else:
            ip = request.META.get('REMOTE_ADDR')
        return ip

Advanced Logging Patterns

Database Query Logging

# settings.py - Log all SQL queries in development
if DEBUG:
    LOGGING['loggers']['django.db.backends'] = {
        'handlers': ['console'],
        'level': 'DEBUG',
        'propagate': False,
    }

# Custom query logger
import logging
from django.db import connection
from django.utils.deprecation import MiddlewareMixin

logger = logging.getLogger('sql_queries')

class QueryLoggingMiddleware(MiddlewareMixin):
    def process_response(self, request, response):
        if settings.DEBUG:
            queries = connection.queries
            total_time = sum(float(q['time']) for q in queries)
            
            logger.info(
                f'SQL Queries: {len(queries)} queries, Total time: {total_time:.3f}s',
                extra={
                    'path': request.path,
                    'query_count': len(queries),
                    'total_time': total_time
                }
            )
            
            # Log slow queries
            for query in queries:
                if float(query['time']) > 0.1:  # Queries slower than 100ms
                    logger.warning(
                        f'Slow query detected: {query["time"]}s',
                        extra={'sql': query['sql'][:200]}
                    )
        
        return response

Performance Logging

# decorators.py
import logging
import time
from functools import wraps

logger = logging.getLogger(__name__)

def log_performance(func):
    """Decorator to log function performance"""
    @wraps(func)
    def wrapper(*args, **kwargs):
        start_time = time.time()
        
        try:
            result = func(*args, **kwargs)
            duration = time.time() - start_time
            
            logger.info(
                f'{func.__name__} completed',
                extra={
                    'function': func.__name__,
                    'duration': f'{duration:.3f}s',
                    'status': 'success'
                }
            )
            
            return result
        except Exception as e:
            duration = time.time() - start_time
            
            logger.error(
                f'{func.__name__} failed',
                extra={
                    'function': func.__name__,
                    'duration': f'{duration:.3f}s',
                    'status': 'error',
                    'error': str(e)
                },
                exc_info=True
            )
            raise
    
    return wrapper

# Usage
@log_performance
def expensive_operation():
    # Your code here
    pass

Security Event Logging

# security/logging.py
import logging

security_logger = logging.getLogger('security')

def log_security_event(event_type, request, **kwargs):
    """Log security-related events"""
    security_logger.warning(
        f'Security event: {event_type}',
        extra={
            'event_type': event_type,
            'user': str(request.user),
            'ip_address': get_client_ip(request),
            'path': request.path,
            'user_agent': request.META.get('HTTP_USER_AGENT', ''),
            **kwargs
        }
    )

# Usage in views
def login_view(request):
    if request.method == 'POST':
        username = request.POST.get('username')
        password = request.POST.get('password')
        
        user = authenticate(request, username=username, password=password)
        
        if user is not None:
            login(request, user)
            security_logger.info(
                'Successful login',
                extra={'user': username, 'ip': get_client_ip(request)}
            )
            return redirect('dashboard')
        else:
            log_security_event('failed_login', request, username=username)
            return render(request, 'login.html', {'error': 'Invalid credentials'})

External Logging Services

Sentry Integration

# settings.py
import sentry_sdk
from sentry_sdk.integrations.django import DjangoIntegration
from sentry_sdk.integrations.logging import LoggingIntegration

sentry_logging = LoggingIntegration(
    level=logging.INFO,
    event_level=logging.ERROR
)

sentry_sdk.init(
    dsn="your-sentry-dsn",
    integrations=[
        DjangoIntegration(),
        sentry_logging,
    ],
    traces_sample_rate=0.1,
    send_default_pii=True,
    environment=os.environ.get('ENVIRONMENT', 'development'),
)

# Custom Sentry logging
import sentry_sdk

def my_view(request):
    try:
        # Your code
        pass
    except Exception as e:
        sentry_sdk.capture_exception(e)
        sentry_sdk.capture_message('Custom error message', level='error')
        
        # Add context
        with sentry_sdk.configure_scope() as scope:
            scope.set_tag('custom_tag', 'value')
            scope.set_context('custom_context', {
                'key': 'value'
            })

CloudWatch Logs Integration

# settings.py
import boto3
from watchtower import CloudWatchLogHandler

cloudwatch_handler = CloudWatchLogHandler(
    log_group='django-app',
    stream_name='production',
    boto3_client=boto3.client('logs', region_name='us-east-1')
)

LOGGING['handlers']['cloudwatch'] = {
    'class': 'watchtower.CloudWatchLogHandler',
    'log_group': 'django-app',
    'stream_name': 'production',
    'level': 'INFO',
    'formatter': 'json',
}

LOGGING['loggers']['django']['handlers'].append('cloudwatch')

Testing Logging

# tests.py
from django.test import TestCase
import logging

class LoggingTests(TestCase):
    def test_logging_output(self):
        """Test that logging works correctly"""
        with self.assertLogs('myapp', level='INFO') as cm:
            logger = logging.getLogger('myapp')
            logger.info('Test message')
            
            self.assertEqual(len(cm.output), 1)
            self.assertIn('Test message', cm.output[0])
    
    def test_error_logging(self):
        """Test error logging"""
        with self.assertLogs('myapp', level='ERROR') as cm:
            logger = logging.getLogger('myapp')
            
            try:
                raise ValueError('Test error')
            except ValueError:
                logger.exception('An error occurred')
            
            self.assertEqual(len(cm.output), 1)
            self.assertIn('An error occurred', cm.output[0])

Best Practices

Use Appropriate Log Levels: DEBUG for development, INFO for production events, ERROR for failures
Include Context: Add relevant information (user, request ID, etc.) to log messages
Rotate Log Files: Use RotatingFileHandler to prevent disk space issues
Structured Logging: Use JSON format for easier parsing and analysis
Security: Never log sensitive data (passwords, tokens, personal information)
Performance: Be mindful of logging overhead in high-traffic applications
Centralize Logs: Use external services for production log aggregation
Monitor Logs: Set up alerts for critical errors and unusual patterns

Effective logging is crucial for maintaining healthy Django applications, enabling quick debugging, performance monitoring, and security auditing across all environments.

Advanced Security Hardening

Advanced security hardening goes beyond Django's built-in security features to implement enterprise-grade security measures. This comprehensive guide covers advanced authentication, authorization patterns, security monitoring, compliance frameworks, and defense-in-depth strategies for production Django applications.

FAQ and Troubleshooting

This comprehensive guide addresses the most common questions, issues, and problems Django developers encounter. From setup and configuration to deployment and performance, find solutions to typical Django challenges.