I'm trying to implement a system that allows a user to upload files over HTTP, saves the file to object storage as well as any metadata surrounding the file that already exists to a NoSQL database (i.e. the groups who have access to it, it's name, size, etc.). At that point it will be sent to a document parser will will perform some operations on it to glean more metadata, which will then be saved back to the NoSQL database and also put into ElasticSearch.

I thought initially that this was not an uncommon architecture. However, the tools that I'm finding for object storage seem to not very easily fit the technical capabilities I have at the file's entrypoint into the backend (I can go into this in more detail if needed, but it's lower-level).

That leads me to think that maybe I'm not taking the best approach. Here, you can find the general flow of the file and it's metadata inwards.

My question is, is this a "bad" architecture? Is it uncommon? If so, what are some alternatives? If not, what are some tools for object storage that might better suit this architecture?

File Ingestion Architecture

I would like this system to be able to be hosted offline using docker containers (each box in the architecture being a container, aside from the web UI). This obviously means that using a cloud-based service like S3 is out of the question.

I would also like to avoid storing files on the API servers themselves, and avoid the need to load entire files into memory on these servers (for obvious reasons).

  • tbh I would also consider it a pretty common architecture type (in a good sense). Curious what other folks can say about it though. Dec 13, 2020 at 20:15
  • @AndreyRusanov Yeah, that's what I thought. I didn't want to directly expose my object storage to the client. However, it seems I cannot find any good examples of how to proxy object storage access through my application. Dec 13, 2020 at 20:19
  • I guess you have two ways: either stream a file through you application back to user (you could add another service for it or simply reuse your REST API) or return a URL from object storage. It really depends, obviously, on your object storage. As I am aware, MinIO (widely used self-hosted S3 replacement) can do it - it has support of presigned URLs and share URLs for upload/download files. Dec 14, 2020 at 8:09
  • @AndreyRusanov MinIO is actually what I'm using, but I wanted to avoid making it internet-facing as much as possible. But, even then, how will the user be able to view the content? I'd have to stream it to the API and then from the API to the web client. Kinda defeats the purpose of S3-like storage. Dec 14, 2020 at 17:03


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.