1

I have 2 micro-services. One is a Django API, the other is a worker. Each one of these lives in side a docker container.

The API is a public API for the web front end. It handles CRUD operations for users of the web application and it only has an authentication backend for endusers. Some REST actions by the user create SQS messages. The API and worker each have their own AWS IAM roles and permissions. The worker polls the SQS for the messages, and for every message, does some processing.

The issue is, that after the worker is done processing it needs to communicate back to the API with success or failure. The workers are async, and could take a few minutes to finish. I see 2 ways of doing this:

  1. Create an endpoint on the API for the worker to hit. The problem with this is that i think that the django API should only authorize end users of the application. I would have to add another authorization backend just for the one API endpoint or do some sort of token based authentication.

  2. Hit the database directly, but this involves having the database credentials in multiple places.

Is there a better way to go about this? I don't like the idea of having multiple authentication backends in what is meant to be just the end user CRUD api. I also can't have a consumer running in the same docker container as the API because it needs to run as a single process inside of the container.

  • Only one microservice will ever need access to the database? – Robert Harvey Aug 8 '16 at 16:09
  • yes technically. If the worker connects to the db directly it would just have to set a boolean column to true though. – user1152226 Aug 8 '16 at 16:48
1

If I understand correctly, the public API is creating the SQS requests that are being serviced by the worker. Why not use a request-reply approach since you already have connections to the messaging system?

  • SQS requires long polling, which would be another process along side the django API if the django api were to be a consuming of messages. This is something you shouldn't do with docker, as it needs to be a single process. The creation of the SQS message is just a rest API call. – user1152226 Aug 8 '16 at 20:10
  • According to docs.aws.amazon.com/AWSSimpleQueueService/latest/… long polling is an option, not a requirement. – JimmyJames Aug 8 '16 at 20:35
0

Your authorization is for external users. Since the worker is internal and trusted, why should it need authorization? I'm assuming that you are dispatching an asynchronous job from the front end and want to notify the user when it is completed.

In the integration world we would use the request-reply pattern with a correlation identifier to match the result of a job to the original requestor.

If that is too low-level for you, check out Celery which handles the dispatching of operations to async workers for you. I believe it even has a plugin to use with Amazon SQS (although I'd recommend just creating a RabbitMQ backend in a docker container).

  • I guess I am asking specific to django. With what you described above, i would still need to reply to a endpoint or exposed api on the django application. The worker is async. – user1152226 Aug 8 '16 at 20:07
  • I can look at celery though, but i would prefer to use SQS because I can trigger SQS messages directly from s3 events, which is valuable to me right now. – user1152226 Aug 8 '16 at 20:15
  • One downside of SQS is that you have to poll the broker for updates. Also the SQS backend for Celery is experimental right now :( docs.celeryproject.org/en/latest/getting-started/brokers/… – Michael Brown Aug 8 '16 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.