In this article, we are going to build a dockerized Django application with Redis, celery, and Postgres to handle asynchronous tasks. Basically, the main idea here is to configure Django with docker containers, especially with Redis and celery. At the end of this tutorial, you will able to create your own asynchronous apps easily by using the initial configuration of this project. Before starting, you’ll need a basic understanding of Django, Docker, and Celery to run some important commands.
Step 1: Add a Dockerfile
Start by creating an empty directory named docker-django-redis-celery and create another new directory named app inside it. Next, we will add required dependencies and let’s include them inside requirements.txt as shown below:
Django>=2.2 celery==4.4.1 redis==3.4.1 psycopg2>=2.7.5,<2.8.0
Then, create a new file named Dockerfile inside the project. We are going to add basic configuration here and especially for Postgres because it requires additional dependencies for a successful build.
FROM python:3.7-alpine ENV PYTHONUNBUFFERED 1 COPY ./requirements.txt /requirements.txt
Initially, we are using python:3.7-alpine which is a very lightweight image and highly recommended to keep the size as small as possible and usually, it’s safe in terms of compatibility. Setting the non-empty value of PYTHONUNBUFFERED means that the python output is transmitted directly to the terminal without being buffered and that allows displaying the application’s output in real-time. It also means that there is no partial output in a buffer in case the python program fails.
Next, we are going to install our dependencies by copying our requirements.txt into the docker image then we will run the pip command to start the installation.
In order to interact with Postgres we need to install postgresql-client and to achieve that we can use the package manager that comes with alpine. We’re going add a package by using —update option which means update the registry before adding it and –no-cache option is not storing the registry index on our docker file. The reason for including these options is to minimize the number of extra files and packages that are added to our docker container. This is the best practice for keeping the smallest footprints possible in our application and it also doesn’t include any extra dependencies in the system which may cause unexpected side effects or it may even create security vulnerabilities.
RUN apk add --update --no-cache postgresql-client jpeg-dev
Now that we have this package we are going to install some temporary packages that need to be installed on the system while we run our requirements and then we can remove them after the installation finished. This is all part of making sure that our docker file has the absolute minimal footprint possible and we won’t keep extra dependencies in our docker file unless they are absolutely necessary. The –virtual option here sets up an alias for our dependencies that we can use to easily remove all those dependencies later. Then we list out all of the temporary dependencies that are required for installing the Postgres client.
RUN apk add --update --no-cache --virtual .tmp-build-deps \ gcc libc-dev linux-headers postgresql-dev musl-dev zlib zlib-dev RUN pip install -r /requirements.txt RUN apk del .tmp-build-deps
Next we are going to make a directory within our Docker image that we can use to store our application source code. It creates an empty folder on our docker image and then it switches to that as the default directory. So any application we run using our docker container will run from this location. Basically, it copies the app directory from our local machine to the docker image.
RUN mkdir /app COPY ./app /app WORKDIR /app
Step 2: Initial Docker-Compose File
Next, we will create a Docker compose configuration for our project. Docker compose is a tool that allows us to run our Docker image easily from our project location. Basically, it allows us to easily manage the different services that combine with our project. This configuration locates in a file within the root of level our project called docker-compose.yml and contains the configuration for all of the services that make up our project. The first line of our Docker compose configuration file is the version of Docker compose that we’re going to be writing our file for and then we define the services for our application below. At this point, we only need one service for our Django application.
version: '3' services: app: build: context: . ports: - "8000:8000" volumes: - ./app:/app command: > sh -c "python3 manage.py migrate && python3 manage.py runserver 0.0.0.0:8000"
As you see above, we named the service app, and then we are going to build the section of the configuration by setting the context to dot symbol which represents our current directory that running docker compose from. The port configuration maps our project from port 8000 on our host to port 8000 on our image. Below the ports, we added a volumes which allows us to get the updates in real-time that we make to our project into our Docker image. Finally, define the commands that is used to run the application inside the Docker container. Once you all set, build the compose file by the following command:
docker-compose build
Step 3: Create a Django Project
We used Docker compose to run a command on our image that contains the Django dependency and that will create the project files that we need for our app. Basically, we will run the command on app and then anything you pass in after is going to be treated as a command.
docker-compose run app sh -c "django-admin startproject app ."
The reason we added sh -c option is it to make it clear to see the command that you’re running on versus all the Docker compose command. It’s also possible to run it without this option but it’s a good practice to separate the actual command. Once you run it a new Django project will be created inside the app directory.
Step 4: Setting up Database
We will add a new service in docker-compose file called db which represents the Postgres database and pulls Postgres image from docker hub with the alpine version tag. Then, we are going to set up environment variables for the database name, username, and password that is created, once our database service starts. So we will define the list of environment variables under a block called environment .
db: image: postgres:10-alpine environment: - POSTGRES_DB=app - POSTGRES_USER=postgres - POSTGRES_PASSWORD=supersecretpassword ports: - "5432:5432"
Now that we’ve created our DB service we can modify our app service to set some environment variables and also depend it on our DB service. Let’s start by adding some environment variables to our app.
- DB_HOST – name of the service that runs our database and that’s going to be db.
- DB_NAME– Postgres DB name so that’s going to be app.
- DB_USER – username which is Postgres.
- DB_PASS – password that we created which supersecretpassword.
The full configuration so far will look like below:
version: '3' services: app: build: context: . ports: - "8000:8000" volumes: - ./app:/app command: > sh -c "python3 manage.py migrate && python3 manage.py runserver 0.0.0.0:8000" environment: - DB_HOST=db - DB_NAME=app - DB_USER=postgres - DB_PASS=supersecretpassword depends_on: - db db: image: postgres:10-alpine environment: - POSTGRES_DB=app - POSTGRES_USER=postgres - POSTGRES_PASSWORD=supersecretpassword ports: - "5432:5432"
Now we have docker all set up, so we can continue by configuring the Django project to use it with the Postgres database. Replace the SQLite configuration with Postgres as shown below:
DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql', 'HOST': os.environ.get('DB_HOST'), 'NAME': os.environ.get('DB_NAME'), 'USER': os.environ.get('DB_USER'), 'PASSWORD': os.environ.get('DB_PASS'), } }
We are getting all variables directly from the environment to make our settings more secure. Let’s run the docker and see if our app is running properly:
docker-compose up --build
Step 5: Waiting for Postgres
In this part, we’re going add a management command to the core app of our Django project. The management command is going to be a helper command
that allows us to wait for the database to be available before continuing and running other commands. This command will be used in our docker compose file when starting our Django app. The reason that we need this command because sometimes when using Postgres with docker compose in a Django app it fails to start because of a database error. It turns out that, once the Postgres service has started there are a few extra setup tasks that need to be done on the Postgres before it is ready to accept connections. Basically, the Django app will try and connect to database before the database is ready and therefore it will fail with an exception. First, let’s create a new app named core and add it into INSTALLED_APPS configuration in settings.py:
docker-compose run app sh -c "django-admin startapp core"
settings.py
INSTALLED_APPS = [ 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'core', ]
To create the command we’re going to start by creating a new directory in our core app that we are going to store our management commands.So this is the Django convention and it’s recommended on the Django website to put all of your commands in a directory called management and then forward slash
commands. So we’re going to start by creating a folder called management and make sure that it locates in the actual core app folder. In each of these folders create a new __init__.py file to make sure that this is picked up as a Python module. The project structure so far will look like below:
\---app | manage.py | +---app | | asgi.py | | settings.py | | urls.py | | wsgi.py | | __init__.py | \---core | admin.py | apps.py | models.py | tests.py | views.py | __init__.py | +---management | | __init__.py | | | \---commands | __init__.py | wait_for_db.py | \---migrations __init__.py
We can add the custom name of the command that we want to create. For now, the command is going to be called wait_for_db.py and we’re going to start by importing the time module which comes up with a standard Python library that we can use to make our applications sleep for a few seconds in between each database check. Simply, add the following code snippet into the command file:
import time from django.db import connections from django.db.utils import OperationalError from django.core.management import BaseCommand class Command(BaseCommand): """Django command to pause execution until db is available""" def handle(self, *args, **options): self.stdout.write('Waiting for database...') db_conn = None while not db_conn: try: db_conn = connections['default'] except OperationalError: self.stdout.write('Database unavailable, waititng 1 second...') time.sleep(1) self.stdout.write(self.style.SUCCESS('Database available!'))
Now, each time we start our project this command needs to be executed so let’s add this in our compose file under the commands block:
version: '3' services: app: build: context: . ports: - "8000:8000" volumes: - ./app:/app command: > sh -c "python3 manage.py migrate && python3 manage.py wait_for_db && python3 manage.py runserver 0.0.0.0:8000" environment: - DB_HOST=db - DB_NAME=app - DB_USER=postgres - DB_PASS=supersecretpassword depends_on: - db db: image: postgres:10-alpine environment: - POSTGRES_DB=app - POSTGRES_USER=postgres - POSTGRES_PASSWORD=supersecretpassword
Great! Now if you run the containers the command will force the app to wait until the database becomes active. We covered all initial configurations so far, and you can use the compose file above for further projects as a starting point if your project will include this tech stack.
Step 6: Configuring Redis and Celery Service
In this part, we will add Redis and celery services to our compose file. There are few tricky points while configuring celery due to the database connection. However, before we add the services let’s add a new file named celery.py into our Django project directory:
celery.py
import os from celery import Celery os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'app.settings') app = Celery('app') app.config_from_object('django.conf:settings', namespace='CELERY') app.autodiscover_tasks()
Basically, it will discover all tasks alongside the project and will pass them to the queue. Next, we also need update __init__.py file inside the current directory, which is our Django project:
__init__.py
from .celery import app as celery_app __all__ = ['celery_app']
Celery requires broker URL for tasks so in this case, we will use Redis as a message broker. Open your settings file and add the following configurations:
CELERY_BROKER_URL = "redis://redis:6379" CELERY_RESULT_BACKEND = "redis://redis:6379"
Great! Now, let’s update the compose file by adding the Redis and celery services:
version: '3' services: app: build: context: . ports: - "8000:8000" volumes: - ./app:/app command: > sh -c "python3 manage.py migrate && python3 manage.py wait_for_db && python3 manage.py runserver 0.0.0.0:8000" environment: - DB_HOST=db - DB_NAME=app - DB_USER=postgres - DB_PASS=supersecretpassword depends_on: - db db: image: postgres:10-alpine environment: - POSTGRES_DB=app - POSTGRES_USER=postgres - POSTGRES_PASSWORD=supersecretpassword redis: image: redis:alpine celery: restart: always build: context: . command: celery -A app worker -l info volumes: - ./app:/app environment: - DB_HOST=db - DB_NAME=app - DB_USER=postgres - DB_PASS=supersecretpassword depends_on: - db - redis - app
As you noticed, we added an environment block under the celery service and the reason is celery must access the database while running. If we don’t provide the database credentials then it will throw a connection exception. At the end of the compose file, you will see depends_on block which means the celery will wait for the services below in order to become active.
As you see, we are frequently using environment variables. At this point, we can make our configuration file more secure by adding environment files. The main objective is here to keep the code clean and managed for the further development process. To achieve that, create a new file in the root level of the project (not inside Django project) named .env.dev and simply copy the variables there:
.env.dev
DB_HOST=db DB_NAME=app DB_USER=postgres DB_PASS=supersecretpassword POSTGRES_DB=app POSTGRES_USER=postgres POSTGRES_PASSWORD=supersecretpassword
Once you copied, then it’s possible to get variables directly from the environment file by using env_file block in compose file. The final configuration of docker-compose file will look like below:
docker-compose.yml
version: '3' services: app: build: context: . ports: - "8000:8000" volumes: - ./app:/app command: > sh -c "python3 manage.py migrate && python3 manage.py wait_for_db && python3 manage.py runserver 0.0.0.0:8000" env_file: - ./.env.dev depends_on: - db db: image: postgres:10-alpine env_file: - ./.env.dev volumes: - pgdata:/var/lib/postgresql/data redis: image: redis:alpine celery: restart: always build: context: . command: celery -A app worker -l info volumes: - ./app:/app env_file: - ./.env.dev depends_on: - db - redis - app volumes: pgdata:
Now if you run the docker-compose again everything is going to be the same but our variables will more secure inside the environment file. For more information about environment variables visit the medium article Using Variables in Docker-Compose.
Conclusion
In this tutorial, you learned how to combine Redis and celery with a dockerized Django web application. Additionally, you can configure the celery beat schedule to perform certain tasks periodically. You will find the source code in my GitHub account and you can use it as an initial state of your own application.
hi why you first added
environment:
– DB_HOST=db
– DB_NAME=app
– DB_USER=postgres
– DB_PASS=supersecretpassword to celery and at the end deleted from docker-compose celery service ?
Hello, because I replaced it with environment file by adding all env variables into it.
got it , do you have idea how then can I use file on gitlab-CI ?