Django Docs
Microservices with Django

Setting Up the Development and Runtime Environment

Setting up a proper development and runtime environment is crucial for successful microservices development with Django. This section covers everything from local development setup to production-ready deployment configurations.

Setting Up the Development and Runtime Environment

Setting up a proper development and runtime environment is crucial for successful microservices development with Django. This section covers everything from local development setup to production-ready deployment configurations.

Development Environment Setup

1. Prerequisites Installation

First, ensure you have the necessary tools installed:

# Install Python 3.11+
brew install python@3.11  # macOS
# or
sudo apt-get install python3.11 python3.11-venv  # Ubuntu

# Install Docker and Docker Compose
brew install docker docker-compose  # macOS
# or follow Docker's official installation guide

# Install Git
brew install git  # macOS
# or
sudo apt-get install git  # Ubuntu

# Install additional tools
brew install httpie jq  # API testing and JSON processing

2. Project Structure Setup

Create a well-organized project structure for multiple microservices:

# Create main project directory
mkdir django-microservices
cd django-microservices

# Create individual service directories
mkdir -p services/{user-service,product-service,order-service,payment-service}
mkdir -p infrastructure/{docker,kubernetes,monitoring}
mkdir -p shared/{libraries,configs}
mkdir -p scripts
mkdir -p docs

# Create environment files
touch .env.development
touch .env.production
touch docker-compose.yml
touch docker-compose.prod.yml

3. Virtual Environment Setup

# Create virtual environment for each service
cd services/user-service
python3.11 -m venv venv
source venv/bin/activate

# Install base requirements
pip install --upgrade pip
pip install django djangorestframework python-decouple
pip freeze > requirements.txt

# Create Django project
django-admin startproject user_service .
cd user_service
python manage.py startapp users

4. Environment Configuration

# .env.development
DEBUG=True
SECRET_KEY=your-secret-key-here
DATABASE_URL=postgresql://postgres:password@localhost:5432/user_service_db
REDIS_URL=redis://localhost:6379/0
RABBITMQ_URL=amqp://admin:password@localhost:5672/
CONSUL_HOST=localhost
CONSUL_PORT=8500
SERVICE_SECRET_TOKEN=your-service-secret-token
ALLOWED_HOSTS=localhost,127.0.0.1,0.0.0.0
CORS_ALLOWED_ORIGINS=http://localhost:3000,http://127.0.0.1:3000

# services/user-service/user_service/settings.py
import os
from decouple import config
from pathlib import Path

BASE_DIR = Path(__file__).resolve().parent.parent

# Security
SECRET_KEY = config('SECRET_KEY')
DEBUG = config('DEBUG', default=False, cast=bool)
ALLOWED_HOSTS = config('ALLOWED_HOSTS', default='').split(',')

# Application definition
DJANGO_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

THIRD_PARTY_APPS = [
    'rest_framework',
    'rest_framework.authtoken',
    'corsheaders',
]

LOCAL_APPS = [
    'users',
]

INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + LOCAL_APPS

MIDDLEWARE = [
    'corsheaders.middleware.CorsMiddleware',
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'users.middleware.RequestTrackingMiddleware',
    'users.middleware.ServiceAuthMiddleware',
]

ROOT_URLCONF = 'user_service.urls'

# Database
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': config('DB_NAME', default='user_service_db'),
        'USER': config('DB_USER', default='postgres'),
        'PASSWORD': config('DB_PASSWORD', default='password'),
        'HOST': config('DB_HOST', default='localhost'),
        'PORT': config('DB_PORT', default='5432'),
        'OPTIONS': {
            'MAX_CONNS': 20,
            'CONN_MAX_AGE': 600,
        }
    }
}

# Cache
CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': config('REDIS_URL', default='redis://localhost:6379/0'),
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
        }
    }
}

# REST Framework
REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': [
        'rest_framework.authentication.TokenAuthentication',
    ],
    'DEFAULT_PERMISSION_CLASSES': [
        'rest_framework.permissions.IsAuthenticated',
    ],
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.PageNumberPagination',
    'PAGE_SIZE': 20,
    'DEFAULT_THROTTLE_CLASSES': [
        'rest_framework.throttling.AnonRateThrottle',
        'rest_framework.throttling.UserRateThrottle'
    ],
    'DEFAULT_THROTTLE_RATES': {
        'anon': '100/hour',
        'user': '1000/hour'
    }
}

# CORS
CORS_ALLOWED_ORIGINS = config('CORS_ALLOWED_ORIGINS', default='').split(',')
CORS_ALLOW_CREDENTIALS = True

# Logging
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
            'style': '{',
        },
        'simple': {
            'format': '{levelname} {message}',
            'style': '{',
        },
    },
    'handlers': {
        'file': {
            'level': 'INFO',
            'class': 'logging.FileHandler',
            'filename': 'user_service.log',
            'formatter': 'verbose',
        },
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'simple',
        },
    },
    'root': {
        'handlers': ['console', 'file'],
        'level': 'INFO',
    },
}

# Service-specific settings
SERVICE_NAME = 'user-service'
SERVICE_VERSION = '1.0.0'
SERVICE_SECRET_TOKEN = config('SERVICE_SECRET_TOKEN')
CONSUL_HOST = config('CONSUL_HOST', default='localhost')
CONSUL_PORT = config('CONSUL_PORT', default=8500, cast=int)
RABBITMQ_URL = config('RABBITMQ_URL', default='amqp://localhost:5672/')

Docker Development Environment

1. Service Dockerfile

# services/user-service/Dockerfile
FROM python:3.11-slim

# Set environment variables
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

# Set work directory
WORKDIR /app

# Install system dependencies
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        postgresql-client \
        gcc \
        python3-dev \
        libpq-dev \
    && rm -rf /var/lib/apt/lists/*

# Install Python dependencies
COPY requirements.txt /app/
RUN pip install --no-cache-dir -r requirements.txt

# Copy project
COPY . /app/

# Create non-root user
RUN adduser --disabled-password --gecos '' appuser
RUN chown -R appuser:appuser /app
USER appuser

# Expose port
EXPOSE 8000

# Health check
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8000/health/ || exit 1

# Run the application
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "--workers", "3", "user_service.wsgi:application"]

2. Development Docker Compose

# docker-compose.yml
version: '3.8'

services:
  # Infrastructure Services
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: microservices_db
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./infrastructure/docker/postgres/init.sql:/docker-entrypoint-initdb.d/init.sql
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    command: redis-server --appendonly yes
    volumes:
      - redis_data:/data
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

  rabbitmq:
    image: rabbitmq:3-management-alpine
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: password
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq
    ports:
      - "5672:5672"
      - "15672:15672"
    healthcheck:
      test: ["CMD", "rabbitmq-diagnostics", "ping"]
      interval: 30s
      timeout: 10s
      retries: 5

  consul:
    image: consul:1.15
    command: consul agent -dev -client=0.0.0.0 -ui
    ports:
      - "8500:8500"
    volumes:
      - consul_data:/consul/data

  # Microservices
  user-service:
    build: ./services/user-service
    environment:
      - DEBUG=1
      - DB_HOST=postgres
      - DB_NAME=user_service_db
      - DB_USER=postgres
      - DB_PASSWORD=password
      - REDIS_URL=redis://redis:6379/0
      - RABBITMQ_URL=amqp://admin:password@rabbitmq:5672/
      - CONSUL_HOST=consul
    ports:
      - "8001:8000"
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy
    volumes:
      - ./services/user-service:/app
    command: >
      sh -c "python manage.py migrate &&
             python manage.py runserver 0.0.0.0:8000"

  product-service:
    build: ./services/product-service
    environment:
      - DEBUG=1
      - DB_HOST=postgres
      - DB_NAME=product_service_db
      - DB_USER=postgres
      - DB_PASSWORD=password
      - REDIS_URL=redis://redis:6379/1
      - RABBITMQ_URL=amqp://admin:password@rabbitmq:5672/
      - CONSUL_HOST=consul
    ports:
      - "8002:8000"
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy
    volumes:
      - ./services/product-service:/app

  order-service:
    build: ./services/order-service
    environment:
      - DEBUG=1
      - DB_HOST=postgres
      - DB_NAME=order_service_db
      - DB_USER=postgres
      - DB_PASSWORD=password
      - REDIS_URL=redis://redis:6379/2
      - RABBITMQ_URL=amqp://admin:password@rabbitmq:5672/
      - CONSUL_HOST=consul
    ports:
      - "8003:8000"
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy
    volumes:
      - ./services/order-service:/app

  # API Gateway
  kong:
    image: kong:3.4
    environment:
      KONG_DATABASE: "off"
      KONG_DECLARATIVE_CONFIG: /kong/declarative/kong.yml
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_ADMIN_LISTEN: 0.0.0.0:8001
    volumes:
      - ./infrastructure/docker/kong/kong.yml:/kong/declarative/kong.yml
    ports:
      - "8000:8000"
      - "8444:8444"
    depends_on:
      - user-service
      - product-service
      - order-service

volumes:
  postgres_data:
  redis_data:
  rabbitmq_data:
  consul_data:

3. Database Initialization

-- infrastructure/docker/postgres/init.sql
CREATE DATABASE user_service_db;
CREATE DATABASE product_service_db;
CREATE DATABASE order_service_db;
CREATE DATABASE payment_service_db;

-- Create users for each service
CREATE USER user_service_user WITH PASSWORD 'user_service_pass';
CREATE USER product_service_user WITH PASSWORD 'product_service_pass';
CREATE USER order_service_user WITH PASSWORD 'order_service_pass';
CREATE USER payment_service_user WITH PASSWORD 'payment_service_pass';

-- Grant permissions
GRANT ALL PRIVILEGES ON DATABASE user_service_db TO user_service_user;
GRANT ALL PRIVILEGES ON DATABASE product_service_db TO product_service_user;
GRANT ALL PRIVILEGES ON DATABASE order_service_db TO order_service_user;
GRANT ALL PRIVILEGES ON DATABASE payment_service_db TO payment_service_user;

4. Kong API Gateway Configuration

# infrastructure/docker/kong/kong.yml
_format_version: "3.0"
_transform: true

services:
  - name: user-service
    url: http://user-service:8000
    routes:
      - name: user-routes
        paths:
          - /api/v1/users
        strip_path: false
    plugins:
      - name: rate-limiting
        config:
          minute: 100
          hour: 1000
      - name: cors
        config:
          origins:
            - "*"
          methods:
            - GET
            - POST
            - PUT
            - DELETE
            - OPTIONS
          headers:
            - Accept
            - Accept-Version
            - Content-Length
            - Content-MD5
            - Content-Type
            - Date
            - X-Auth-Token
            - Authorization

  - name: product-service
    url: http://product-service:8000
    routes:
      - name: product-routes
        paths:
          - /api/v1/products
        strip_path: false
    plugins:
      - name: rate-limiting
        config:
          minute: 200
          hour: 2000

  - name: order-service
    url: http://order-service:8000
    routes:
      - name: order-routes
        paths:
          - /api/v1/orders
        strip_path: false
    plugins:
      - name: rate-limiting
        config:
          minute: 150
          hour: 1500

Development Scripts

1. Setup Script

#!/bin/bash
# scripts/setup-dev.sh

set -e

echo "Setting up Django Microservices Development Environment..."

# Create virtual environments for each service
services=("user-service" "product-service" "order-service" "payment-service")

for service in "${services[@]}"; do
    echo "Setting up $service..."
    
    cd "services/$service"
    
    # Create virtual environment
    python3.11 -m venv venv
    source venv/bin/activate
    
    # Install dependencies
    pip install --upgrade pip
    if [ -f "requirements.txt" ]; then
        pip install -r requirements.txt
    fi
    
    # Run migrations
    if [ -f "manage.py" ]; then
        python manage.py migrate
    fi
    
    deactivate
    cd ../..
done

echo "Development environment setup complete!"
echo "Run 'docker-compose up' to start all services"

2. Service Management Script

#!/bin/bash
# scripts/manage-services.sh

COMMAND=$1
SERVICE=$2

case $COMMAND in
    "start")
        if [ -z "$SERVICE" ]; then
            echo "Starting all services..."
            docker-compose up -d
        else
            echo "Starting $SERVICE..."
            docker-compose up -d $SERVICE
        fi
        ;;
    "stop")
        if [ -z "$SERVICE" ]; then
            echo "Stopping all services..."
            docker-compose down
        else
            echo "Stopping $SERVICE..."
            docker-compose stop $SERVICE
        fi
        ;;
    "restart")
        if [ -z "$SERVICE" ]; then
            echo "Restarting all services..."
            docker-compose restart
        else
            echo "Restarting $SERVICE..."
            docker-compose restart $SERVICE
        fi
        ;;
    "logs")
        if [ -z "$SERVICE" ]; then
            docker-compose logs -f
        else
            docker-compose logs -f $SERVICE
        fi
        ;;
    "shell")
        if [ -z "$SERVICE" ]; then
            echo "Please specify a service name"
            exit 1
        fi
        docker-compose exec $SERVICE /bin/bash
        ;;
    "migrate")
        if [ -z "$SERVICE" ]; then
            echo "Please specify a service name"
            exit 1
        fi
        docker-compose exec $SERVICE python manage.py migrate
        ;;
    *)
        echo "Usage: $0 {start|stop|restart|logs|shell|migrate} [service-name]"
        exit 1
        ;;
esac

3. Testing Script

#!/bin/bash
# scripts/run-tests.sh

set -e

echo "Running tests for all microservices..."

services=("user-service" "product-service" "order-service" "payment-service")

for service in "${services[@]}"; do
    echo "Testing $service..."
    
    if [ -d "services/$service" ]; then
        cd "services/$service"
        
        # Activate virtual environment
        source venv/bin/activate
        
        # Run tests
        if [ -f "manage.py" ]; then
            python manage.py test
        fi
        
        deactivate
        cd ../..
    fi
done

echo "All tests completed!"

Runtime Environment Configuration

1. Production Docker Compose

# docker-compose.prod.yml
version: '3.8'

services:
  # Production PostgreSQL with replication
  postgres-master:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: microservices_db
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_REPLICATION_USER: replicator
      POSTGRES_REPLICATION_PASSWORD: ${POSTGRES_REPLICATION_PASSWORD}
    volumes:
      - postgres_master_data:/var/lib/postgresql/data
      - ./infrastructure/docker/postgres/postgresql.conf:/etc/postgresql/postgresql.conf
      - ./infrastructure/docker/postgres/pg_hba.conf:/etc/postgresql/pg_hba.conf
    command: postgres -c config_file=/etc/postgresql/postgresql.conf
    networks:
      - microservices-network

  postgres-slave:
    image: postgres:15-alpine
    environment:
      POSTGRES_MASTER_SERVICE: postgres-master
      POSTGRES_SLAVE_USER: replicator
      POSTGRES_SLAVE_PASSWORD: ${POSTGRES_REPLICATION_PASSWORD}
    volumes:
      - postgres_slave_data:/var/lib/postgresql/data
    depends_on:
      - postgres-master
    networks:
      - microservices-network

  # Redis Cluster
  redis-master:
    image: redis:7-alpine
    command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD}
    volumes:
      - redis_master_data:/data
    networks:
      - microservices-network

  redis-slave:
    image: redis:7-alpine
    command: redis-server --slaveof redis-master 6379 --requirepass ${REDIS_PASSWORD}
    volumes:
      - redis_slave_data:/data
    depends_on:
      - redis-master
    networks:
      - microservices-network

  # Production services with scaling
  user-service:
    image: user-service:${VERSION}
    environment:
      - DEBUG=0
      - DB_HOST=postgres-master
      - DB_NAME=user_service_db
      - DB_USER=user_service_user
      - DB_PASSWORD=${USER_SERVICE_DB_PASSWORD}
      - REDIS_URL=redis://redis-master:6379/0
      - REDIS_PASSWORD=${REDIS_PASSWORD}
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M
    networks:
      - microservices-network
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health/"]
      interval: 30s
      timeout: 10s
      retries: 3

networks:
  microservices-network:
    driver: bridge

volumes:
  postgres_master_data:
  postgres_slave_data:
  redis_master_data:
  redis_slave_data:

2. Environment Variables Management

# .env.production
DEBUG=False
SECRET_KEY=your-super-secret-production-key
POSTGRES_PASSWORD=secure-postgres-password
POSTGRES_REPLICATION_PASSWORD=secure-replication-password
REDIS_PASSWORD=secure-redis-password
USER_SERVICE_DB_PASSWORD=secure-user-service-password
PRODUCT_SERVICE_DB_PASSWORD=secure-product-service-password
ORDER_SERVICE_DB_PASSWORD=secure-order-service-password
PAYMENT_SERVICE_DB_PASSWORD=secure-payment-service-password
SERVICE_SECRET_TOKEN=secure-service-communication-token
VERSION=1.0.0

3. Health Check Implementation

# shared/health_check.py
from django.http import JsonResponse
from django.db import connection
from django.core.cache import cache
import redis
import pika
from django.conf import settings

def comprehensive_health_check(request):
    """Comprehensive health check for microservices"""
    health_status = {
        'status': 'healthy',
        'service': settings.SERVICE_NAME,
        'version': settings.SERVICE_VERSION,
        'checks': {}
    }
    
    # Database check
    try:
        with connection.cursor() as cursor:
            cursor.execute("SELECT 1")
        health_status['checks']['database'] = 'healthy'
    except Exception as e:
        health_status['checks']['database'] = f'unhealthy: {str(e)}'
        health_status['status'] = 'unhealthy'
    
    # Redis check
    try:
        cache.set('health_check', 'ok', 10)
        cache.get('health_check')
        health_status['checks']['redis'] = 'healthy'
    except Exception as e:
        health_status['checks']['redis'] = f'unhealthy: {str(e)}'
        health_status['status'] = 'unhealthy'
    
    # RabbitMQ check
    try:
        connection_params = pika.URLParameters(settings.RABBITMQ_URL)
        connection = pika.BlockingConnection(connection_params)
        connection.close()
        health_status['checks']['rabbitmq'] = 'healthy'
    except Exception as e:
        health_status['checks']['rabbitmq'] = f'unhealthy: {str(e)}'
        health_status['status'] = 'unhealthy'
    
    status_code = 200 if health_status['status'] == 'healthy' else 503
    return JsonResponse(health_status, status=status_code)

4. Monitoring and Logging

# shared/monitoring.py
import logging
import time
from django.utils.deprecation import MiddlewareMixin
from prometheus_client import Counter, Histogram, generate_latest

# Metrics
REQUEST_COUNT = Counter('django_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
REQUEST_LATENCY = Histogram('django_request_duration_seconds', 'Request latency')

class MetricsMiddleware(MiddlewareMixin):
    def process_request(self, request):
        request.start_time = time.time()
    
    def process_response(self, request, response):
        if hasattr(request, 'start_time'):
            duration = time.time() - request.start_time
            REQUEST_LATENCY.observe(duration)
            REQUEST_COUNT.labels(
                method=request.method,
                endpoint=request.path,
                status=response.status_code
            ).inc()
        
        return response

def metrics_view(request):
    """Prometheus metrics endpoint"""
    from django.http import HttpResponse
    return HttpResponse(generate_latest(), content_type='text/plain')

Development Workflow

1. Daily Development Commands

# Start development environment
docker-compose up -d

# View logs
docker-compose logs -f user-service

# Run migrations
docker-compose exec user-service python manage.py migrate

# Create superuser
docker-compose exec user-service python manage.py createsuperuser

# Run tests
docker-compose exec user-service python manage.py test

# Access service shell
docker-compose exec user-service python manage.py shell

# Stop environment
docker-compose down

2. Service Development Cycle

# 1. Make code changes
# 2. Restart specific service
docker-compose restart user-service

# 3. Check logs
docker-compose logs -f user-service

# 4. Run tests
docker-compose exec user-service python manage.py test

# 5. Test API endpoints
http GET localhost:8000/api/v1/users/ Authorization:"Token your-token"

Summary

A well-configured development and runtime environment is essential for microservices success. Key components include:

  • Containerized services with Docker
  • Infrastructure services (PostgreSQL, Redis, RabbitMQ)
  • API Gateway for routing
  • Service discovery with Consul
  • Comprehensive health checks
  • Monitoring and metrics
  • Automated setup scripts

This foundation enables efficient development, testing, and deployment of Django microservices. In the next section, we'll explore cloud-native data processing with MongoDB.

Introducing the Django Microservices Architecture

Django, traditionally known for building monolithic web applications, can be effectively adapted for microservices architecture. In this section, we'll explore Django's native components that support microservices development and the external tools that complement Django in a distributed environment.

Cloud-native Data Processing with MongoDB

MongoDB is an excellent choice for microservices due to its flexible schema, horizontal scaling capabilities, and cloud-native features. This section explores how to integrate MongoDB with Django microservices for efficient data processing and storage.