0

Background: I'm working on a new project at work that will run in AWS. We're trying to use a modern microservice architecture and take advantage of cloud technology, but we don't have much experience with that yet. Due to some library restrictions, we need to run at least some of the services in Python, which my organization has a severe lack of production experience with. Most of our previous applications are in Spring/Java (and some of our other microservices will be running using them).

In our architecture, we want to have a number of microservices running in a sort of processing pipeline that communicates using SQS/SNS for async communication. At the same time, they'll need to be able to accept REST communication from our frontend or other services that need data that the service owns.

My Question: How can I best handle both of these forms of communication in python? Is my solution below viable, or should we be doing something entirely different with our communication strategy?

What I've come up with: My current idea is to have the background/pipeline processing for a microservice happening on an EC2 instance that will listen and process SQS messages, then make a POST request to a storage service specifically for that microservice which sits behind an AWS API Gateway and uses Lambda functions to store the results in a database. In this way, we can have the frontend/other services call to the API Gateway to get data out of the microservice without interrupting the processing of SQS messages. This also lets us take advantage of serverless architecture, making scaling much simpler.

I personally am fairly inexperienced and haven't been able to find satisfactory information or examples of architecture like this through my own searching, so I would appreciate any insights or inputs you can give.

To clarify, what I'm most curious about is the more fine-grained issue of how to open up my microservices to both REST and messaging communication, or if some other pattern would be a better idea.

  • I did microservices in python with the Tornado server. – Niklas Rosencrantz Nov 7 '17 at 17:46
  • @DjDacSaunders I took a quick look, but I didn't see anything that would allow me to handle the async messaging portion more easily with it over something like flask. – SBR Nov 7 '17 at 19:04
0

If you want to have your Lambda-based microservice triggered both via REST and by SQS messages, then you can do that simply by using both API Gateway and also by publishing the SQS messages to an SNS topic. Lambda functions can be triggered from multiple event sources, see here: http://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html

Let me know if this doesn't make sense. If you elect to run your own Python runtime on EC2 anyway, and are looking for a good Python framework for running async code, I'd recommend the Twisted Framework (https://twistedmatrix.com/trac/)

  • I agree with what you're saying. The Lambda only solution triggered from SNS is one we're already planning on using for one service, and from there I think we can just update the DB directly rather than needing to go through the storage API Gateway. However there is at least one service that probably will need its own instance running. Do you think having that instance calling the Gateway for storage would be ok in this case? – SBR Nov 7 '17 at 19:46
  • Sorry I just edited my answer a bit too aggressively-- I think if you want to buy in to the serverless paradigm, you can run the Python module as a Lambda function as well and can dispense with EC2 entirely. If you simply cannot do that and wish to run the Python code on your own, then yes you could either call the Gateway via HTTP or have your python code send a message to an SNS topic and run the microservice that way. – RibaldEddie Nov 7 '17 at 19:49
  • Also very curious as to why you need the separate instance vs separate lambda function. Can you maybe speak more to that? – RibaldEddie Nov 7 '17 at 19:51
  • I think there are some environmental requirements that to my knowledge lambda won't meet, which is why we would use a separate instance in that case. As for your previous comment, I hadn't thought of using an SNS topic to trigger the storage lambda. I'll definitely look into that. – SBR Nov 7 '17 at 19:55
  • I will say that one nice thing about using a POST to the API Gateway for storage is that if some part of the storage functionality or lambda fails, we have instant feedback on that and can either send the offending message to a dead letter queue or try it again later. I'm not sure publishing to an SNS topic would give us the same level of feedback. – SBR Nov 7 '17 at 19:59
-1

Since you asked about

... or if some other pattern would be a better idea.

The modern way to do this sort of thing is with an entirely hybrid approach. The solution I am going to offer is an AWS specific solution (and, I know it's not trendy, but I seriously dislike AWS and prefer Digital Ocean)

Create, Update, Delete (CUD) -> implement as a containerized service if you like (virts are decade old tech and can't spin up nearly as fast as a container) or better as a set of lambdas behind a gateway (you mentioned gateways). Reading and writing to a data store is so common it's almost not worth talking about. However, you will want to publish all changes to your entity to an SNS topic. This is where other services can tap into the event channels created by SNS (which, for the record, isn't a true topic). If you put data in dynamo it automatically will trigger a lambda for you so you can kill 2 birds with one stone.

READ (R) on the other hand is where things get much more interesting: All reads should be directed to a distributed cache (Like, REDIS which is supported natively by AWS). You use a Lambda to receive your CUD events from your SNS topic which you may then morph and publish to your REDIS cluster (which will handle propagation).

This strategy is know as CQRS

Your question has been signigicantly updated since I originally answered.

To have your service handle REST endpoints and async messaging (inbound is a real problem) it will get really tricky with AWS. SNS (despite what it says and hence why I hate it) is NOT a topic so you can't receive broadcast messages. But that's exactly what you need if your microservice is horizontally scalable and in a cluster (it is/will be, right?) so that all instances can receive the same message.

The only 2 ways ready ways I know of to do this are: 1. Stand up your own AMQP cluster and use a lambda to publish to a REAL TOPIC Then receive from it. But this has numerous operational issues to overcome. 2. publish incoming messages to a Redis cluster (from AWS) and make your app be a part of the cluster so messages get propagated to it.

  • This seems to have virtually nothing to do with the question. – Philip Kendall Nov 7 '17 at 18:25
  • We already plan to use an API Composition layer, which is an alternate pattern for accomplishing this. What I'm more curious about is the more fine-grained issue of how to open up my microservices to both REST and messaging communication. – SBR Nov 7 '17 at 19:10
  • @SBR Can you update your question to make that last point more clear? – RibaldEddie Nov 7 '17 at 19:18
  • @PhilipKendall his question says ... or if some other pattern would be a better idea. I assumed people knew that a lambda could be python – Christian Bongiorno Nov 8 '17 at 19:33
  • @SBR my answer more-or-less mirrors ribalds – Christian Bongiorno Nov 8 '17 at 19:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.