1

Folks, I am in middle of writing a web application (Python/Flask) where home page has user profile image in the navbar which is coming from a database (blob), I am wondering if this is a good practice. Here I have following choices:

  1. Currently it is stored image in the backend db in a table in a blob field.
  2. Store the image disk location in the table instead of storing the image itself in the blob field, this will refer to the disk file where to load the image from instead of transferring the image file from a blob field.
  3. Avoid hitting the db altogether when accessing the images and use a local disk file location instead, this location can be mapped to nginx directly which means web caching will ensure fast image retrieval.

Wondering if this is a big issue and how you guys are handing it anyway, what is the industry best practices here? Please note the database will be hit anyway to load various other items like user first_name and possibly other bits too but I don't want to lose the choice of storing images which can be cached.

Any help here will be highly apprecaited.

5
  • I presume your concern is that this will be frequently retrieved and rarely changed. It would be good to at least insure the browser can cache this resource so it's not re-retrieved all the way back to the database. There are several good ways to do that without missing updates to the image. Going farther than that would depend on your workload and priorities.
    – joshp
    Commented Apr 25 at 0:51
  • You seem to be more concerned with where to store the image, than on whether to cache it. There's not enough information here to say whether storing in a db blob is a good practice or not. It can be fine for a wide range of scales and workloads. The question to ask is whether it's worth your time to change it. Is there sufficient benefit to justify the time and making your project a little more complicated. Is it burning significant time and resources with your current or expected workload to justify a change?
    – joshp
    Commented Apr 25 at 0:57
  • And how is retrieving the image from the database any more of a concern than retrieving the other data necessary to render the front page? Commented Apr 25 at 14:14
  • Thanks for the input. I appreciate the time and effort taken.
    – afsar
    Commented Apr 28 at 6:42
  • How many requests per minute?
    – JacquesB
    Commented Apr 29 at 11:28

2 Answers 2

2

The industry best practice is to never store images (and other big blobs) in database directly. And only store "references" to those images. The reason for that is that such images overload database heavily for no reason. And database will often be the most important resource for you. It is the biggest bottleneck most of the time.

Databases, especially ACID databases are notoriously hard to scale. But images and other static blobs, are not hard to scale. In fact this problem is already solved through Content Delivery Networks, which also handle other issues, like load balancing, geospatial latency, caching, etc. While you might not need full CDN power from the beginning, it is a good idea to be prepared for it.

Consistency is rarely an issue. You just need a background worker that will periodically scan for orphaned files and remove them. That's likely to be the only consistency issue.

  1. Currently it is stored image in the backend db in a table in a blob field.

Bad. Scales badly, overloads database.

  1. Store the image disk location in the table instead of storing the image itself in the blob field, this will refer to the disk file where to load the image from instead of transferring the image file from a blob field.
  2. Avoid hitting the db altogether when accessing the images and use a local disk file location instead, this location can be mapped to nginx directly which means web caching will ensure fast image retrieval.

These two sound very similar. I assume that the difference is that in 3 you use nginx and in 2 you use the web application and serve the file manually? That is secondary to be honest. Nginx will do that in a more efficient way, so sure, why not. But yes, storing files locally, and serving through a separate mechanism is a good start.

There are few improvements that you may want to consider.

  1. You can store full URLs in the database, and serve them to the client. That way it will be possible for the client to load the image from anywhere, even outside your application. This will be especially useful once you want to switch to CDN (which is a separate service, with its own domain).
  2. Security is a concern. Should those images be publicly accessible? If not, then you will have to put some authorization layer. Most CDNs support some form of URL signing, which is an efficient form of authorization, and often good enough. But with this concern I don't think nginx is an option (unless there's some plugin that supports it, which I'm not aware of). And also signed URLs means that you can't store URLs in the database, they have to be generated on the fly. You will have to store some kind of identifier/path, but not a static URL. You could make it very flexible and store (kind, reference) pairs in the database, where kind can be URL or signed URL, or something else.
  3. Uploading is a separate issue. Going through the web server is an option, and the only reasonable at the beginning. But there's a risk involved, what if too many uploads happen? What if a malicious user tries to DDOS you? Many people don't bother with those questions, and that is fine most of the time. If you ever need to deal with it, you will need a proper upload service, like AWS S3. Let maintainers of those services deal with those issues. Most of those services support notifications after the upload, so that your web application can react once it finishes (to do some post processing for example, or database update, or whatever).
0

Since images are mostly static content, usually a CDN (Content Delivery Network) is used to deliver them as mentioned by @freakish.

Consider a global website. One user is in California, the other in Germany. The big advantage with a CDN is that content can be delivered and cached from an edge location, so that users in California get images from an edge location on the west coast of the United States and users in Germany get images from an edge location in Europe. This maximizes network utilization.

If your application is a single set of users in just one location, maybe a CDN is overkill, but any sort of global web site can benefit greatly from delivering static content in this fashion.

I once worked on an application that served images from a database and many customers called in complained how much bandwidth the application consumed on their network. So, it wasn't the greatest design since the application served lots of images. We offloaded the images to a CDN and customers were much happier after that as network utilization dropped considerably.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.