Django-Celery

I hate vegetables, but this is different.

Hungry people usually do not want to be kept waiting for food. People hate waiting, especially when they do not know the process of what they are waiting for. Same goes for website users, say, there is a very time-consuming task, i.e.  data-mining or video processing. Users will be irritated when their browsers hang up for a very long time. Tasks such as those usually run indefinitely. Of course, as developers of these sites, we do not want users spending more time waiting for these tasks to finish than exploring the beauty of our site. Moreover, it would be best to run this type of tasks in background and have a separate process working on it while still having full control. Well, no more worries, Celery is here!

Celery is an asynchronous task/job queue based on distributed message passing. It is used in production systems to process millions of tasks a day.

It gives developers more options to execute and control long processes. It can also be used as a scheduler that is more powerful than a cron job.

Installation

For the installation part, I am assuming that you have pip installed.

1. Install celery:

2. Choose a message-passing backend. There are many brokers you can choose from, but for simplicity of setting up, I will only give two: Kombu and RabbitMQ. For high-traffic production, use RabbitMQ, otherwise, Kombu will be enough.

If you chose Kombu:

Or, if you chose RabbitMQ:

3. Install django-celery:

4. Edit settings.py, and add the following lines:

If using django-kombu, you have to add djkombu to the INSTALLED_APPS and add this additional line to your settings:

Creating Tasks

To create a task, just define it in a file app/tasks.py, where app is a name of an app registered in your INSTALLED_APPS.

If you want a cron job type of task:

By default, django-celery will search all the tasks.py inside your apps and register them to celery. If you named it differently, you might want to tell celery where your tasks are defined:

Running Tasks

Now we are ready to run our very long task. Usually, you call this type of tasks in your views:

This will not be executed until you start Celery in your project:

If you are running a periodic task, set it to beat mode:

Some interesting functions you might want to know:

And that’s that. Everything here is a summary of all the django-celery tutorials that I’ve read. Hope you learned a lot from it.