0

I'm designing a web application for image analysis using AI. My API for CRUD operations is written in Django. Now I want to add the AI functionality which loads the model once on the startup, receives images, and returns the results. In the future, it would also allow for annotating data and fine-tuning the models.

First, I wanted to load the models in the Django application, but then I'd have to store a variable with the loaded models, which is not a good design. Is that correct?

What alternative do I have? The AI server must be stateful, and I need some kind of API. Is there any clean solution to this?

###############################################

The AI models are implemented in Keras and loaded from *.h5 files. I have considered a message queue and a separate AI server but besides image processing, the server would have to expose some API for settings. If I'd go for implementing a separate AI server with API, then why not use the Django server I already have? In such a case where to store the model's object?

An additional problem is that one of the objects I need to store is a thread, which continuously monitors a server checking for new images to be analyzed. If I declare this variable for example in settings.py, it blocks the further execution, because it never leaves the namespace it's declared in.

I'm asking for recommendations regarding the design for this problem.

Improve this question
8
  • Huh? Why does loading models require a global variable?
    – candied_orange
    Commented Dec 18, 2022 at 2:33
  • @KarolBorkowski can we assume that, by global variable you mean caching? I mean, holding data which lifecycle span several requests untill is no longer necessary due to user inactivity or because it expires (for whatever reason)
    – Laiv
    Commented Dec 18, 2022 at 11:22
  • @candied_orange You're right, it doesn't. I need a variable though, which is available in the whole app.
    – Karol Borkowski
    Commented Dec 18, 2022 at 11:42
  • @Laiv Yes, but it's not a simple text or numerical variable, but heavy Keras models.
    – Karol Borkowski
    Commented Dec 18, 2022 at 11:42
  • 1
    Must the image analysis run inside Django? Can it not be a separate process receiving requests via (e.g.) a queue from Django and sending processed images via file system/Dicom/queue?
    – Kasper van den Berg
    Commented Dec 18, 2022 at 22:30

1 Answer 1

Reset to default
5

Here's a rough answer.

  • "Good Design" for web apis is based around the assumption that you are running a web site.

  • Web sites generally want to serve lots of users at the same time

  • To serve many users at the same time you can't hold each users, state in memory because you will run out.

  • To be stateless when we have state, we store state in a db and reload, or store the state client side and send it every time. There are a bunch of tricks based around avoiding state on the server side.

However! If your application requires a large amount of server side state, such as a ML model, or needs super fast responses based on state, such as a multiplayer game. Then you need to hold that state in memory between calls. It's just a technological limitation and those tricks wont work for you.

If you have to do this then your architecture changes to avoid the problem of running out of memory in different ways than being stateless.

  • Limit the number of concurrent users with a queue
  • Use sticky sessions to link a user to a server
  • Aggressive horizontal scaling out of servers
  • Customers pay for computer time
  • Force log off after a timeout

etc

The standard "Good Practice" for a website, doesn't apply to you.

Improve this answer
5
  • Thank you for the answer, but it doesn't solve my problem yet. Where should I store such objects in a Django project? I add that one of the objects is a DICOM server listener, which runs continuously until a user terminates it.
    – Karol Borkowski
    Commented Dec 18, 2022 at 11:46
  • @KarolBorkowski: I think what Ewan is saying is that there is no standard solution for this problem.
    – Greg Burghardt
    Commented Dec 18, 2022 at 12:46
  • @GregBurghardt Then, I'm asking for some recommendations
    – Karol Borkowski
    Commented Dec 18, 2022 at 15:17
  • 1
    I guess im saying "dont worry about being stateful" which was what i interpreted as the core of your problem. as to the django technical problems, no idea.
    – Ewan
    Commented Dec 19, 2022 at 12:01
  • @KarolBorkowski what's wrong with just using a global variable?
    – Criticizing Israel not allowed
    Commented Feb 25, 2023 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.