5

Basically I am designing a web-enabled application that should have an API in the middle. I found this question on Stack Overflow, which unfortunately has no answers.

Let's say there is a system, similar to this picture: enter image description here

The center "API" is basically an HTTP server written with one of the RESTful frameworks in some language. The "Web App" is basically another "Client", so we should treat it the same way.

Now, let's say my clients (iOS, Android, Web) connect via OAuth2 and communicate with a RESTful JSON-based API with the server. All good and logical, every new 'user' has an API key assigned to them, so when a user performs an operation, i.e. viewOrder(333), the system would log that "user 'xxx' at H:i:s time requested order #333". Still good.

Then we have workers and this is where my main question resides. Say workers do the order processing, for example a worker that checks the payment status of an order. What the worker does is checks for new (related) payments in an external system and pushes them to a queuing server for later processing. Another worker (or the same one, maybe?) constantly checks the queue for newly added payments, pops them from the queue, does all the processing and if all is correct, updates the DB with new order status, money payed, etc, etc...

So my question is: should I treat a worker as another 'user' in my system? Say, 'system_user_1' that is responsible for order processing. Should this system user also authenticate with OAuth2 as the rest of them? Is it even correct to have this 'system user' in the system? That are the security drawbacks? What are any other possible implications of such a system? Or am I doing this all wrong and there is a completely different solution to my problem?

p.s. This system must scale and it potentially could scale very fast, so this is an attempt to create a long-term solution. Ideally there will be multiple instances of the 'API' server with nginx load balancing for high loads.

  • 6
    I guess my question would be "why wouldn't you?" – Robert Harvey Jul 11 '16 at 15:22
  • Thanks everyone, all your answers are helpfull and greatly appreciated! They gave me ideas about the future system, which is what I was looking for. Eveone is pretty much suggesting realitively the same thing. Currently I'm looking into even more of an async aproach with Tornado. I will try to post an update when I have a more solid understanding of the requirements. The latest draft includes splitting the public and private API into two instances and hiding the "private" one in case of any DDoS or other attacks. In the mean time more comments are always welcome! – lime Jul 12 '16 at 14:35
6

The "right" answer would almost always have edges authenticate with some sort of user account or other auth mechanism.

Why? Because you can limit access of that specific edge interface to only the data they need with the right scopes. For example, a worker only updates orders but can't see payment info. If there was a compromise of the worker it can't update orders, marking them all paid. Or overpaid, triggering a refund.

This doesn't always mean actual users. Maybe you issue access keys to the workers - but you can still limit what the access key can do plus revoke one of needed.

Bottom line, if the API is your core, everyone touching it should be touching it in some way that ensures you can

  • limit reach in a breach or exploit and
  • revoke access if the token or endpoint is compromised in any way.
3

Here is how I design similar system :

Client : ios/Android/other mobile clients interact with my API to make some orders or view orders

API : Backend Restful service which handles the requests from clients and UI. If the orders need to be processed asynchronously then we should use Queues, from the question I get that the orders are handled asynchronously.

API should keep the order requests in the DB with the state unprocessed and keep them in the Queue (like Amazon SQS, RabbitMQ etc) and then workers should pick them from the Queue process them and update the DB. We should not expose processing API because that can cause security concerns.

1

I've always had the fleet of workers interact directly with the database (or queue, whatever has the order state), which might simplify access. But in logs, you want to make sure you can name and trace back to workers, for auditing an order flow. Also, you need to synchronize the workers doing work, and clients grabbing state via the API. If you have a single network identity (like an LDAP user) shared among workers, then the question of which worker did what to which order and when becomes reliant on host names or IP addresses to tell the workers apart in logs. That's easier if the logs come from the workers rather than the API. For example, what is the source address of an HTTP API request if a proxy or load balancer sits in front of the API server(s)?

If you give each worker it's own network identity, that changes things. But then you need to worry about making sure nobody crashes your party by pretending to be a worker.

1

The reason it hasn't been answered is that it appears to be mixing different levels of abstraction.

It is common to have different categories of users. In this case I see three categories of users of the API:

  • Web App
  • Client (may use different interfaces than the Web App)
  • Workers (may access the APIs using a Web App or client)

It may be better to conceptualize the model by role. These appear to be at least roles:

  • Order Placers: Web App, Client
  • Order Processors: Workers (There may be different roles for different workers.)
  • Database Client: API

In the diagram the Order Placers have been modeled by Interface rather than role. Workers may actually use a Web App and/or a Client.

1

Most usually yes.

In fact, it's usually the simplest solution, because the authentication and authorization mechanisms need to be built and tested anyway. And, I'd argue for it being the default solution unless compelling reasons surface to bypass the normal auth* rules.

The alternative is to add another set of authentication and authorization modules, increasing the overall complexity of the system and your maintenance burden.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.