How can I create a Python microservice on AWS that both accepts REST connections and processes SQS messages?

Question

Background: I'm working on a new project at work that will run in AWS. We're trying to use a modern microservice architecture and take advantage of cloud technology, but we don't have much experience with that yet. Due to some library restrictions, we need to run at least some of the services in Python, which my organization has a severe lack of production experience with. Most of our previous applications are in Spring/Java (and some of our other microservices will be running using them).

In our architecture, we want to have a number of microservices running in a sort of processing pipeline that communicates using SQS/SNS for async communication. At the same time, they'll need to be able to accept REST communication from our frontend or other services that need data that the service owns.

My Question: How can I best handle both of these forms of communication in python? Is my solution below viable, or should we be doing something entirely different with our communication strategy?

What I've come up with: My current idea is to have the background/pipeline processing for a microservice happening on an EC2 instance that will listen and process SQS messages, then make a POST request to a storage service specifically for that microservice which sits behind an AWS API Gateway and uses Lambda functions to store the results in a database. In this way, we can have the frontend/other services call to the API Gateway to get data out of the microservice without interrupting the processing of SQS messages. This also lets us take advantage of serverless architecture, making scaling much simpler.

I personally am fairly inexperienced and haven't been able to find satisfactory information or examples of architecture like this through my own searching, so I would appreciate any insights or inputs you can give.

To clarify, what I'm most curious about is the more fine-grained issue of how to open up my microservices to both REST and messaging communication, or if some other pattern would be a better idea.

Answer 1

If you want to have your Lambda-based microservice triggered both via REST and by SQS messages, then you can do that simply by using both API Gateway and also by publishing the SQS messages to an SNS topic. Lambda functions can be triggered from multiple event sources, see here: http://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html

Let me know if this doesn't make sense. If you elect to run your own Python runtime on EC2 anyway, and are looking for a good Python framework for running async code, I'd recommend the Twisted Framework (https://twistedmatrix.com/trac/)

Answer 2

Since you asked about

... or if some other pattern would be a better idea.

The modern way to do this sort of thing is with an entirely hybrid approach. The solution I am going to offer is an AWS specific solution (and, I know it's not trendy, but I seriously dislike AWS and prefer Digital Ocean)

Create, Update, Delete (CUD) -> implement as a containerized service if you like (virts are decade old tech and can't spin up nearly as fast as a container) or better as a set of lambdas behind a gateway (you mentioned gateways). Reading and writing to a data store is so common it's almost not worth talking about. However, you will want to publish all changes to your entity to an SNS topic. This is where other services can tap into the event channels created by SNS (which, for the record, isn't a true topic). If you put data in dynamo it automatically will trigger a lambda for you so you can kill 2 birds with one stone.

READ (R) on the other hand is where things get much more interesting: All reads should be directed to a distributed cache (Like, REDIS which is supported natively by AWS). You use a Lambda to receive your CUD events from your SNS topic which you may then morph and publish to your REDIS cluster (which will handle propagation).

This strategy is know as CQRS

Your question has been signigicantly updated since I originally answered.

To have your service handle REST endpoints and async messaging (inbound is a real problem) it will get really tricky with AWS. SNS (despite what it says and hence why I hate it) is NOT a topic so you can't receive broadcast messages. But that's exactly what you need if your microservice is horizontally scalable and in a cluster (it is/will be, right?) so that all instances can receive the same message.

The only 2 ways ready ways I know of to do this are: 1. Stand up your own AMQP cluster and use a lambda to publish to a REAL TOPIC Then receive from it. But this has numerous operational issues to overcome. 2. publish incoming messages to a Redis cluster (from AWS) and make your app be a part of the cluster so messages get propagated to it.