3

Problem overview:
I'm creating a spring application just for learning purposes. I would like to create a microservice just for files, which:

  • at the beginning would have only two basic endpoints upload/download a file (used by other microservices, not by a user directly)
  • create db only for that microservice with thefile table. At least at the beginning.
  • store files in a cloud (f.e. Amazon S3)
  • store meta-information about the file in the file table. By meta I mean:
    • Url to file in the cloud
    • Name
    • Size
    • Type (a directory or a file)
    • Extension

Example flow:

  • a microservice A sends an image: user_attachment.jpg to the files microservice upload endpoint.
  • server does validation, secuirty check etc.
  • sends the file to a cloud storage f.e. Amazon S3
  • saves meta-info about that file in a local table called file. F.e. it adds: INSERT INTO file (url, name, size, type, extension) VALUES ("http://aws.amazon.com/testapp/pictures/2018-09/picutre.jpg", "picture", 34233, "FILE", "JPG")
  • send meta-info back to the microservice A that initialized upload operation.

Uploading to cloud would be in a different thread, asynchronously and maybe even with a messagebroker help.


Questions:

  1. Is it a clean/good-design-patterns-compliant approach ?
  2. What is good/bad about that approach?
  3. How could I do that in a better way ? (don't want to store files (content) on the machine where application is running [locally] )

Misc:
This example is simple. I believe in commercial/big projects design might have similar assumptions/foundations. Thus let's assume that there is multiple tables in the file microservice, and both their structure and microservice logic are advanced.

How does file-management-microservice in commercial application might look like according to good design-pattern rules ?

  • 3
    Why would you do that? It doesn't seem like you're adding any value compared to using S3 directly, while the cost of doing such... proxy is rather high. So what's the point? – Arseni Mourzenko Sep 9 '18 at 19:38
  • I wasn't precise enough, sorry for that. file microservice wouldn't be used by users directly, but indirectly via another microservices (let's call them A, B and C). Yes, I could use S3 directly in all of them. In my question I just wanted to know how it's done in commercial/big projects. I believe the foundation of files-managment-microservice is designated in a similar way (but logic and db tables are obviously more advanced). By my question I just wanted someone to show me how file-management should look like according to good design-patterns. – user3529850 Sep 9 '18 at 20:13
  • If it’s just for educational purposes, why not just store files locally? – RubberDuck Sep 11 '18 at 9:33
5

A goal of a service in SOA is to perform some logic which, otherwise, would have to be duplicated among other services. That logic is, in turn, the reason for the service to exist.

In your case, you add no value through the service. Any other service could easily access S3 directly, given that S3 is itself a service (it is not your service, but that is irrelevant here).

Not only there is no added value to the intermediary service, but its cost is rather prohibitive. You may imagine that you can draft one in a few hours, which is true. However, in order to have something that works reliably and handles all the edge cases, you have to deal with caching, redundancy, interrupted uploads and recovery, large files, timeouts, brute force and DDOS, special characters in parameters, permissions and IAM, and dozens of other pretty things.

Therefore, you're about to spend two or three weeks of development and possibly one week of debugging (YMMV; I don't know your experience in HTTP, SOA and the technology you want to use to create your service, but even if you're highly experienced, you shouldn't expect spending less than two weeks on that, which is a lot).

I just wanted to know how it's done in commercial/big projects.

Companies pay on average $700/day for a highly experienced developer, so given my estimate of two weeks, you're at $7,000 for a task. Commercial projects are about money; when there is an opportunity to save an amount even as small as $7,000, they'll do it.

Additional to this direct cost, there is a cost in terms of hosting and code maintenance. This service will have to be maintained for years, adding to the bill much more than the original price. Again, all this money wasted without the clear understanding of what could such service save to the company. It doesn't save bandwidth, nor other resources. It doesn't reduce the amount one will pay to Amazon, so...

This is not all. The cost of maintenance of all the projects which depend on the intermediary service will also increase. If a service:

  • Has to be patched and the patch requires an interface change,
  • Has to be moved to another location, with a change in its URL,
  • Is down, requiring to have a circuit breaker to ensure the client service doesn't go down in turn,
  • Is deprecated, requiring to migrate the client to something else,

the immediate and unforeseen maintenance is required and is usually costly as well. Now, it is much more likely that those four things happen to your service than to Amazon's S3. Not because you are a bad developer, no. But because Amazon's scale is slightly different than the scale of your service, which means that they have much more workforce to pay to ensure the clients can rely on S3.

Finally, many developers have prior experience with Amazon AWS (and possibly S3). This means that when you hire a new guy, he can easily understand how a service is storing files if it uses S3 directly. If you add a level of indirection, this benefit is lost. And, more importantly, every time someone has a problem with the storage, he would need to ask himself if the problem comes from the client service, from your intermediary or from S3. This adds to the debugging time.

So:

Is it a clean/good-design-patterns-compliant approach ?

No. Add services when they add value. Don't make things more complex than they need to be (KISS) and, specifically, don't add layers which bring no benefits.

What is good/bad about that approach?

What is good: the fact that you provide an interface which is much simpler compared to S3. For developers unfamiliar with AWS, S3 can be quite complex.

What is bad: already told it above.

How could I do that in a better way ? (don't want to store files (content) on the machine where application is running [locally] )

By calling directly S3.

  • 1
    Would the value of abstracting away the service provider be enough added value to justify the service? While cloud file services offer about the same features, it is an outside service that can creep through your entire infrastructure if you aren't careful. And if you don't centralize this within your ecosystem, changing cloud file providers can be a traumatic change, can't it? – Greg Burghardt Sep 10 '18 at 17:27
  • Given the requirements, such migration won't be very difficult to perform. Consider the other scenario. Services A and B use the intermediary service which abstracts S3. One day, team A decides that it needs a feature which is only available in Azure Storage. Team B, however, doesn't want to pay more for the storage. So now, the intermediary has to support S3 and Azure Storage at the same time, and somehow provide features of Azure Storage selectively for clients who have chosen it. – Arseni Mourzenko Sep 10 '18 at 18:03
0

Amazing explanation by Arseni.

One advantage I see of this intermediary service is if it provides the caller (micro service or end user) the ability to choose the storage to be used for their use. I don't know how much of a practical suggestion that is, but I often save my important document across multiple cloud storage vendors for fear of losing it on one. So, this intermediate service could provide the ability to choose which storage the caller wants to use (optionally) or else default it to the cheapest that suits the caller's purpose.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.