30

My company is currently working on adopting a microservice architecture but we are encountering some growing pains (shock!) along the way. One of the key contention points we are facing is how to communicate large quantities of data between our different services.

As a bit of background we have a document store that serves as a repository for any document we might need to handle across the company. Interacting with said store is done via a service which provides a client with a unique ID and a location to stream the document. The document's location can later be accessed via a lookup with the provided ID.

The problem is this - Does it make sense for all our microservices to be accepting this unique ID as part of their API for the purposes of interacting with documents or not? To me this feels inherently wrong - the services are no longer independent and rely upon the document store's service. While I do acknowledge this might simplify API design and perhaps even have some performance gains the resulting coupling more than counterbalances the benefits.

Does anyone know how the rainbow unicorns (Netflix, Amazon, Google, etc.) handle large files / data exchange between their services?

Improve this question
2
  • What are you using for a highly available document/file store?
    – Terence Johnson
    Sep 25, 2015 at 21:37
  • @TerenceJohnson We are using a home-grown solution for now. We are migrating towards a solution that leverages a RESTful Api that only persists a unique document id and its location (which is provided to the client rather than a stream to prevent unnecessary internal network burden). The actual persistence will be done via AWS.
    – PremiumTier
    Sep 28, 2015 at 23:49

5 Answers 5

Reset to default
9

Does anyone know how the rainbow unicorns (Netflix, Amazon, Google, etc.) handle large files / data exchange between their services?

Unfortunately I do not know how they deal with such problems.

The problem is this - Does it make sense for all our microservices to be accepting this unique ID as part of their API for the purposes of interacting with documents or not?

It violates the Single Responsibility Principle, which should be inherently in your microservice's architecture. One microservice - logically one, physically many instances representing one - should be dealing with one topic.

In the case of your document store, you have one point, where all queries for documents go (of course you could split this logical unit up into multiple document stores for several kinds of documents).

  • If your "application" needs to work on a document, it asks the respective microservice and processes its result(s).

  • If another service needs an actual document or parts of it, it has to ask the document service.

One of the key contention points we are facing is how to communicate large quantities of data between our different services.

This is an architectural problem:

  1. Decrease the need to transfer big amounts of data

    Ideally, each service has all of it's data and needs no transfer to simply serve requests. As an extension of this idea - if you have the need to transfer data, think of redundancy (*in a positive way_): Does it make sense to have the data redundant in many places (where they are needed)? Think of how possible inconsistencies might harm your processes. There is no transfer faster as actually none.

  2. Decrease the size of the data itself

    Think of how you could compress your data: Starting with actual compression algortihms up to smart data structures. The less goes over the wire, the faster you are.

Improve this answer
4

If the ID returned by your document store is the way to reference documents throughout the system, then it makes sense for all services to accept that 'Document ID' on their API when the service needs to know which document it needs to work with.

This does not necessarily create a tighter coupling between the services than needed. Services that need to access documents need to access the document-store service anyway and they need that ID to tell the store which document to access.
Services that don't access documents directly might need to pass the Document ID along, but to those services it would be just an arbitrary string that does not create a dependency.

Improve this answer
3
  • Thank you for your reply. I should add that we could potentially benefit from exposing our microservices to external consumers who may not want to also leverage our internal document store. With that in mind do you still feel this is the best approach?
    – PremiumTier
    May 12, 2015 at 19:43
  • @PremiumTier: Yes. But those external customers would have to provide a store of their own that supports the same API as your internal store, so that your services can cooperate with it.
    – Bart van Ingen Schenau
    May 12, 2015 at 20:13
  • That makes sense but it still feels more cumbersome than having services accept streams, byte arrays, or json blobs instead of document references. In that case an 'adapter' service could easily be invoked first to get the file stream if needed before calling any subsequent services. I'm not trying to be argumentative by the way but rather just trying to understand the merits of this approach :)
    – PremiumTier
    May 13, 2015 at 15:25
2

Personally, I'd rather not use a separate document store service and document id, but a URL to access the documents (with proper header authentication). With this approach you won't need other services to rely on the document service rather it could just use the full URL to access the document.And also it makes sense when it comes to scaling as well, you could use multiple document stores as and when the storage grows and provide the URL.

However you might need a service(s) to upload a document and to obtain it's URL.

Improve this answer
2

Does anyone know how the rainbow unicorns (Netflix, Amazon, Google, etc.) handle large files / data exchange between their services?

Checkout Amazon S3 REST API specs, seemingly they return full object in bytes. Seems not many options if you are designing a microservice. Amazon S3 response format link

Improve this answer
0

This is very late, But will leave the answer to others people who encounter the same problem.

Does it make sense for all our microservices to be accepting this unique ID as part of their API for the purposes of interacting with documents or not?

Yes, the only way to reduce data transfer is to store it in the right place and deal with it using ref ID.

I recommend you to have a microservice that works specifically for file sharing. (I'll call this file sharing microservice)

Every other microservices could add the file sharing microservice as a dependency, But the file sharing microservice can't have a dependency on others.

You could use S3 or whatever technology behind the file sharing microservice but this microservice provides your best-fit abstraction. And later you could migrate the underlying tech to fit your app scale.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.