3

I've been wondering something about how we pass complex ORM objects to background workers — in my case, Django models and Celery, but this could apply to any similar background processing framework.

I usually pass model objects "by reference" using the primary key. In quick Python, this would be how I send an email for an user :

@app.task
def background_task(user_id):
    user = User.objects.get(id=user_id)
    send_email(user.first_name, user.last_name)

background_task.delay(user.id)

A coworker instead prefers to pass values directly to the task :

@app.task
def background_task(user_first_name, user_last_name):
    send_email(user_first_name, user_last_name)

background_task.delay(user.first_name, user.last_name)

I find the first method much easier to maintain :

  • We have tasks using the second paradigm taking 10+ arguments coming from a single object, and these are quite hard to read and reuse because one needs to make sure that all the arguments are accounted for.

  • Adding more information to the task means that you need to add more arguments, and make sure that they are also added to all the places that use this task, which is time-consuming and error-prone.

The second method has the following advantages :

  • You do not run the risk of using the wrong information. For instance, with the first method, a model could be updated twice before the task can run, resulting in the first update not being processed.
  • You also do not run a second query when the task runs since task arguments are stored along with the task, which could lighten the load on the database.

I am rather biased towards the first method, but neither of us were able to reach a conclusion on this.

  • Are there advantages/inconvenients that I did not list?
  • Can you think of some projects using either of these that we could use as examples?
  • How should we handle this?
  • 5
    Your coworker's approach has the advantage of not being tightly-coupled to the User class. – Robert Harvey Mar 3 '17 at 23:46
  • @RobertHarvey That is actually a very good point I hadn't considered! – F.X. Mar 4 '17 at 0:00
  • 1
    If you have some type of object serialization, besides the database, that would be a third approach. Serialize the object containing the parameters. – Frank Hileman Mar 7 '17 at 16:29
  • That would be a good first step, but what if there are many such objects, each responsible for some of the method's arguments? @RobertHarvey's comment made a lot of sense in my mind, and after reading on the subject I like the idea of some clearly defined object exposing features from many low-level models to our controllers. I am unfamiliar with these concepts, so I'd welcome a more complete answer based on that. – F.X. Mar 9 '17 at 0:13
  • The argument that you need 10 parameters from a single object isn't really relevant, even if you pass the whole object and add a new field, you will still need to use that new field in the task. However if it is clear that a specific task is really coupled to an object, i see nothing wrong as using the 1st method. Another method would be similar to how workflow engine usually works : you have a map of attributes to fill, the problem is that you have to know every field to be initialized for each task, unlike having function with parameters. – Walfrat Nov 6 '18 at 8:03
0

To pass model object in celery task you can use django's inbuilt serializer
https://docs.djangoproject.com/en/2.1/topics/serialization/

from django.core import serializers
data = serializers.serialize("json", SomeModel.objects.all())

Pass this serialised data as argument to celery task and there you can deserialise it back

for obj in serializers.deserialize("json", data):
    do_something_with(obj)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.