I hate vegetables, but this is different.
Hungry people usually do not want to be kept waiting for food. People hate waiting, especially when they do not know the process of what they are waiting for. Same goes for website users, say, there is a very time-consuming task, i.e. data-mining or video processing. Users will be irritated when their browsers hang up for a very long time. Tasks such as those usually run indefinitely. Of course, as developers of these sites, we do not want users spending more time waiting for these tasks to finish than exploring the beauty of our site. Moreover, it would be best to run this type of tasks in background and have a separate process working on it while still having full control. Well, no more worries, Celery is here!
Celery is an asynchronous task/job queue based on distributed message passing. It is used in production systems to process millions of tasks a day.
It gives developers more options to execute and control long processes. It can also be used as a scheduler that is more powerful than a cron job.
For the installation part, I am assuming that you have pip installed.
1. Install celery:
123 pip install celery
2. Choose a message-passing backend. There are many brokers you can choose from, but for simplicity of setting up, I will only give two: Kombu and RabbitMQ. For high-traffic production, use RabbitMQ, otherwise, Kombu will be enough.
If you chose Kombu:
123 pip install django-kombu
Or, if you chose RabbitMQ:
123 sudo apt-get install rabbitmq-server
3. Install django-celery:
123 pip install django-celery
4. Edit settings.py, and add the following lines:
123456789101112 INSTALLED_APPS = ( …‘celery’,‘djcelery’,)# celery config infoBROKER_HOST = “localhost”BROKER_PORT = 5672BROKER_USER = “guest”BROKER_PASSWORD = “guest”BROKER_VHOST = “/”
1234 import djcelerydjcelery.setup_loader()
If using django-kombu, you have to add djkombu to the INSTALLED_APPS and add this additional line to your settings:
123 BROKER_BACKEND = “djkombu.transport.DatabaseTransport”
123 then, sync your database:
123 ./manage.py syncdb
To create a task, just define it in a file app/tasks.py, where app is a name of an app registered in your INSTALLED_APPS.
12345678 # app/tasks.pyfrom celery.decorators import task@taskdef a_very_long_task(args):…return result
If you want a cron job type of task:
1234 from celery.task.schedules import crontabfrom celery.decorators import periodic_task
12345 @periodic_task(run_every=crontab(hour=”*”, minute=”*”, day_of_week=”*”))def a_periodic_task():print “Hello World!”
By default, django-celery will search all the tasks.py inside your apps and register them to celery. If you named it differently, you might want to tell celery where your tasks are defined:
123 CELERY_IMPORTS = (“app.jobs”, )
Now we are ready to run our very long task. Usually, you call this type of tasks in your views:
123456 from app.tasks import a_very_long_taskdef view_for_a_very_long_task(request, args):…a_very_long_task.delay(args)
This will not be executed until you start Celery in your project:
123 ./manage.py celeryd -l info —settings=settings &
If you are running a periodic task, set it to beat mode:
123 ./manage.py celeryd -B -l info —settings=settings &
Some interesting functions you might want to know:
123456789 » result = a_very_long_task.delay(args)» result.ready() # waits until task is done» True» result.state» u’SUCCESS’» result.successful()» True
12345678 # execute after a given number of seconds» result = a_very_long_task.delay(args, countdown=3)# execute at a given time» from datetime import datetime, timedelta» result = a_very_long_task.delay(args, eta=datetime.now() + timedelta(minutes=1))» result.successful()
And that’s that. Everything here is a summary of all the django-celery tutorials that I’ve read. Hope you learned a lot from it.