I have got a native image processing application that can enhance photos(peApp). Now, I want to serve this functionality to the world by having a server application that supports REST. Now while taking the scalability into account, I think I better seperate this project into 3 components:

  • peApp : does only his computationally intensive job (may use GPU)
  • zeroMQ(or another mq) : a message queue application that can organize request
  • node.js & express based web service(or maybe a native app) : provides REST and puts requests into message queue.

Now my questions are:

  • Are those TYPE of components the right choice? (I'm not asking if I should use zeroMQ, I'm asking if using a message queue application is sound)
  • In terms of scalability, will I eventually need a middleware?
  • What are the best practices about such systems? (a heavy duty app and serving it)
  • How can I achieve a bidirectional communication/notification?
  • 1
  • 2
    Let me put this to you another way: how can you re-word your first two bullets so that the answers are not merely "yes" or "no?" – Robert Harvey Aug 26 '17 at 16:51
  • 1
    I'm being persistent about this for a reason: asking good questions is hard. Answering bad (i.e. vague, too broad, unclear) questions is even harder. You're expecting us to give you some assurance that you've made the "right" choices. We can't give you that assurance, because you've given us no insight into your decision-making process. If there is such a thing as a "right" choice, it is a choice made after evaluating all of the costs and benefits of each approach, and deciding on the approach that is the best fit for your specific requirements... – Robert Harvey Aug 26 '17 at 16:54
  • 1
    @RobertHarvey : My criteria is performance and extendibility. Because I can bundle everything(server and the worker) in a single executable and put them in different threads and adjust affinities, or node.js can act as a mq beside being a web service, etc. Problem is that I cannot foresee the possible problems of all those choices. That's why I'm asking for experience rather than a solid technical information which unfortunately makes me ask question including the words "best practices". Thank you – Ridiculous Fish Aug 26 '17 at 16:56
  • 3
    Unfortunately, none of us can foresee all of the possible problems you may have with any technical choices you make. Your best bet is to stand up a small prototype of your proposed architecture, and evaluate it on its merits. Does that take work? Of course it does. But that's what any of us here would do if we were evaluating any technology that is new to us. Your best way to become informed about your technological choices is to be "hands-on." – Robert Harvey Aug 26 '17 at 16:57

In my experience, @RobertHarvey's comment that "none of us can foresee all of the possible problems" is generally correct, and as he suggests you should probably prototype out your solution and setup good metrics to figure out where it needs to scale.

That said, I think based on your description your approach is reasonable. If you want to have a RESTful API, that API will need to accept a POST or PUT with the image(s) that are to be enhanced. You don't want to block the request, so you accept it immediately and drop the file somewhere so that the image processor can process it later (on a different process or even a different machine). That 'somewhere' could be a S3 bucket or a file system location or a message queue or anywhere; I don't think we can evaluate what's best for you in that case.

What I would look for in terms of performance and scalability are:

1) Have I divided the components in such a way that I only need to expand the places that are the bottleneck. For example, if you expect that you'll need to run multiple machines that are using peApp, then that needs to be isolated and only contain code needed to run that process.

2) Given #1, do my other process have the infrastructure they need in order to perform without injecting logic into the other components needlessly. For example, you add a MQ because you don't want your REST app to directly call peApp. this makes sense.

Beyond those two factors, it's a lot of optimizing for your specific problem. If it's me, I'd start off w/o the MQ frankly, and just have peApp ping a well known location (e.g. a bucket on S3) to get the 'next' file in the pile to process. I'd watch lag times CPU(GPU) load and so on to decide if I need to scale out or scale up. Then I'd solve the next problem when those are sorted out.

Not the answer you're looking for? Browse other questions tagged or ask your own question.