2

I am building an application where I need to store millions of images and later tag them. The tags attributed to the images could change over time as the tagging system evolves. Images will then be searched for by tags.

In terms of storing the files, I have eliminated the option of storing them in a RDBMS; I have tried this in the past and ran into scaling and performance problems and likewise I have eliminated the option of storing them on a file system as that too has given me performance, scalability and backup issues. I am now considering using a NOSQL key-value data store or something like Amazon S3. Is a key-value store an appropriate choice for this type of data?

In terms of storing the tag data for each image, since the tag types are unknown upfront I am looking to leverage the schemaless nature of NOSQL and use either and document data store or perhaps a column profile one. What would be the key factors in deciding which type of store to use? Are there other options that I should consider?

Lastly, does it make sense to split the image data and meta data into separate stores or is there a technology that can do both? Perhaps something like a key value store that also allows the addition of Metadata and querying against the Metadata?

Update: I have seen the previous answers but they are a few years old and do not seem to be leveraging contemporary technologies. Can someone please comment if RDBMS + Filesystem is still the best way to do this or are their newer and improved solutions.

  • 2
    Possible duplicate of Advice on which storage technology for an Image repository – gnat Oct 31 '17 at 5:38
  • What kinds of queries you want to perform? – rwong Oct 31 '17 at 6:09
  • I want to query against the tags. So for example if an image had tags summer, green and August and I did a search on images in August, the image would be found. – ssc327 Oct 31 '17 at 6:35
  • 1
    Why not S3 and Elasticsearch? S3 for the blob store and Elasticsearch for very fast and powerful faceted searches on the tags. Both are very scalable, and Amazon provides a hosting solution for Elasticsearch now. An RDBMS can handle millions of images, but the queries can be complicated for multi-tag searches. S3/Elasticsearch can easily scale to trillions of records and more. – Berin Loritsch Oct 31 '17 at 13:12
  • @gnat, the world has changed since 2014, the answer might be different now. – Berin Loritsch Oct 31 '17 at 13:13
5

The question is one of scale, where it will be hosted, cost and management. If you know you are going to host in AWS, then you can take advantage of the distributed nature that makes the cloud more scalable.

First Decision: Self Hosted vs the Cloud

The old answers (circa 2014) reflect the mindset when self hosting was still predominant. However, there are reasons to look outside of an RDBMS for tag related queries.

Filesystem hosting requires that you manage your NAS or SAN yourselves and ensure you have enough provisioning and the expertise to improve performance and capacity as necessary. It can be very expensive if the costs are not amortized across several applications.

The cloud allows you to use AWS S3 or whatever equivalent blob storage for your cloud provider. This solution only charges you for the storage you use, and cloud blob storage provides both the scale and performance needed to scale as your application grows.

Second Decision: RDBMS or Search

The way you have to store tags in a relational database vs. a document store makes the queries to get records related to those tags more difficult. This is even more so when you are looking for intersections between tags (i.e. documents that have 2 or more identical tags). The queries will slow down the more complicated it gets.

ElasticSearch, SOLR, and similar search servers that can double as a document store provide an ideal middle ground. Many cloud providers have hosting solutions for these types of problems. They are designed to scale to very large sizes and perform searches very quickly. In fact this site (softwareengineering.stackexchange.com) uses ElasticSearch to do queries like this. NOTE: ElasticSearch is also a NoSQL DB in addition to being a search server.

I will say that you can't think in relational terms when you are doing document searches so there is a learning curve.

Added bonus is that at least with AWS, ElasticSearch costs less than an RDBMS for the same size tier.

Bottom Line

Millions of records is not astronomical for today's RDBMS's. However, you will reach a saturation point. Many websites still use an RDBMS for the data storage of record and then synchronize that with a search server for the heavy lifting. That decision really depends on things outside the scope of this question.

The ElasticSearch/S3 route will scale well beyond that. However, do your research. There are tradeoffs that you have to weigh. In my case this choice was the right one.

  • Those who use RBDMS with search engine usually use the database for : storage (obviously), integrity, no duplications of data, transactions. Depending of the noSQL solution you use for storage and search (or two different for each) you might have to ensure the integrity yourself, to duplicate some data to be able to search, have to handle operations without the supports of transactions. They're still tons of reason to keep RDBMS around, this is all dependant of the exact requirments. "Store and tag millions of images" seems a bit short to me to be able to choose. – Walfrat Oct 31 '17 at 15:33
  • @Walfrat, I agree that in several cases that an RDBMS can still have relevance. I think for the case of doing the tag searches a search engine like ElasticSearch and SOLR is superior when performing those searches. I am not suggesting an all or nothing approach. A current application I am supporting has a traditional RDBMS and will be adding ElasticSearch to perform the faceted searches we need, and I don't see that changing any time soon. – Berin Loritsch Oct 31 '17 at 15:50
  • @borin loritsch even if you go down the self-hosted path, are there not better options than file servers? Would not standing up a self hosted nosql key value store be superior? Or do you feel that the complexity of doing do yourself is far greater than managing SANs and File Servers? – ssc327 Nov 1 '17 at 1:30
  • @ssc327, You might look at Hadoop and it's file system since it is designed for this purpose. I would never host a binary in a document NoSQL server. A key/value NoSQL store might be good. Having a proper blob store could also be a good way to go. There are libraries that let you relocate your implementation to a cloud instance when you are ready to go that way. There's a lot of factors in the decision process. – Berin Loritsch Nov 1 '17 at 12:44
  • 1
    @ssc327 The storage of the actual image files, and the indexing of the tags, can be handled separately. Berin's answer focuses on the queries by tags, such as boolean expressions on tags, or returning items which match any 4 out of 5 tags, or to automatically include statistically related tags, etc. – rwong Nov 1 '17 at 18:10
2

Storing the files should be the least painful option. However if you need the scalability you need to put it on a distributed file system such as GFS or HDFS. When you are storing them you can prescan them so they are

  1. valid image files
  2. get the sha256sum or 512 may be overkill for each and use that as the file name.
  3. (optionally) strip off non-image data which may be appended after the image file.
  4. (optionally) reencode lossless images into a new format.

When storing the files do not store them all in one directory instead group them by 2 character hex paths that would improve the directory scanning rate.

By doing the sha256sum of the file you can quickly eliminate exact file duplicates. By doing #3, #4 you can further eliminate duplicates.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.