I hate vegetables, but this is different.
Hungry people usually do not want to be kept waiting for food. People hate waiting, especially when they do not know the process of what they are waiting for. Same goes for website users, say, there is a very time-consuming task, i.e. data-mining or video processing. Users will be irritated when their browsers hang up for a very long time. Tasks such as those usually run indefinitely. Of course, as developers of these sites, we do not want users spending more time waiting for these tasks to finish than exploring the beauty of our site. Moreover, it would be best to run this type of tasks in background and have a separate process working on it while still having full control. Well, no more worries, Celery is here!
Celery is an asynchronous task/job queue based on distributed message passing. It is used in production systems to process millions of tasks a day.
It gives developers more options to execute and control long processes. It can also be used as a scheduler that is more powerful than a cron job.
Installation
For the installation part, I am assuming that you have pip installed.
1. Install celery:
pip install celery
2. Choose a message-passing backend. There are many brokers you can choose from, but for simplicity of setting up, I will only give two: Kombu and RabbitMQ. For high-traffic production, use RabbitMQ, otherwise, Kombu will be enough.
If you chose Kombu:
pip install django-kombu
Or, if you chose RabbitMQ:
sudo apt-get install rabbitmq-server
3. Install django-celery:
pip install django-celery
4. Edit settings.py, and add the following lines:
INSTALLED_APPS = ( …
‘celery’,
‘djcelery’,
)
# celery config info
BROKER_HOST = “localhost”
BROKER_PORT = 5672
BROKER_USER = “guest”
BROKER_PASSWORD = “guest”
BROKER_VHOST = “/”
import djcelery
djcelery.setup_loader()
If using django-kombu, you have to add djkombu to the INSTALLED_APPS and add this additional line to your settings:
BROKER_BACKEND = “djkombu.transport.DatabaseTransport”
then, sync your database:
./manage.py syncdb
Creating Tasks
To create a task, just define it in a file app/tasks.py, where app is a name of an app registered in your INSTALLED_APPS.
# app/tasks.py
from celery.decorators import task
@task
def a_very_long_task(args):
…
return result
If you want a cron job type of task:
from celery.task.schedules import crontab
from celery.decorators import periodic_task
@periodic_task(run_every=crontab(hour=”*”, minute=”*”, day_of_week=”*”))
def a_periodic_task():
print “Hello World!”
By default, django-celery will search all the tasks.py inside your apps and register them to celery. If you named it differently, you might want to tell celery where your tasks are defined:
CELERY_IMPORTS = (“app.jobs”, )
Running Tasks
Now we are ready to run our very long task. Usually, you call this type of tasks in your views:
from app.tasks import a_very_long_task
def view_for_a_very_long_task(request, args):
…
a_very_long_task.delay(args)
This will not be executed until you start Celery in your project:
./manage.py celeryd -l info —settings=settings &
If you are running a periodic task, set it to beat mode:
./manage.py celeryd -B -l info —settings=settings &
Some interesting functions you might want to know:
» result = a_very_long_task.delay(args)
» result.ready() # waits until task is done
» True
» result.state
» u’SUCCESS’
» result.successful()
» True
# execute after a given number of seconds
» result = a_very_long_task.delay(args, countdown=3)
# execute at a given time
» from datetime import datetime, timedelta
» result = a_very_long_task.delay(args, eta=datetime.now() + timedelta(minutes=1))
» result.successful()
And that’s that. Everything here is a summary of all the django-celery tutorials that I’ve read. Hope you learned a lot from it.