1

I am running a personal project of offering 3D model generation as a service (in this case CAD model). Since most of the generation of 3D models take a long time, I thought about using cache to improve performance for the users. The 3D models are generated based on well-known parameters, and those would be used to access the data from the cache.

Generated CAD models are saved on a file system. The service would return those files as a byte array. It is hard do specify the frequency in which these models will be generated, since the service is not online yet.

I have been searching for a cache solution for a project of mine that has the following spec:

  1. Database storage of cached data. In memory caching would not be good here since CAD models can go all the way up to 40 MB each. Also, I would need to distribute the cached data acess accross multiple servers, with all of them caching data to the same storage.
  2. Upfront + lazy population, along with monitoring of hits on the cache.
  3. Explicit eviction via command. I want every CAD model to be cached, but, I also want to be able to explicitly kill the entries that match a certain key (via request, for example).

I have taken a look into some solutions:

  1. Redis: As I mentioned earlier, I think it is not a good idea to store this type of data in-memory, since the models will be cached without a time/access based eviction policy (unless this isnt really a problem... correct me if I am wrong).
  2. CouchBase and MongoDB (well-known NoSQL database solutions): Those would provide a database to store the models, sometimes with a query cache layer (in case of CouchBase). Although, it seems to me like this is more of selecting a database, instead of selecting a cache solution. Even if I go for a NoSQL database for caching, I will still have to develop most of the logic behind eviction and cache access/storage.

I have been looking for a cache solution that fits my spec with standard functionalities. I am trying to avoid developing any of that myself, but it seems to me right now that using a NoSQL database with some development to store/access data is my best bet.

Can anyone more experienced with caching give some insights on that matter? Am I going right about it?

  • Alright, let me complement the question... @DocBrown, is it answerable now? Let me know if you need more details... I think the bottleneck is already described there. – Gabriel Robaina Dec 12 '19 at 14:31
  • Think about a parallelepiped. It has a height, length and a width. Each of those is a parameter for the service to generate a parallelepiped. Since it is generating solids, it would not go all the way to "a few thousand values". Certainly less than a hundred. – Gabriel Robaina Dec 12 '19 at 14:46
  • I was thinking about using a hash made out of the input parameters as the key on the table, and the BLOB as the value. This way, I would not necessarily need a NoSQL database (unless NoSQL is faster... I suppose it is, but I am not entirely sure). If I stored all parameters into attributes of a stored object in NoSQL, would it make my life harder? Would I still be able to query that efficiently? – Gabriel Robaina Dec 12 '19 at 16:39
  • I see... maybe I went wrong about it. I can probably use both... I have been going about the database as if it was a cache, but in fact it is just a generation history storage. I gave it a little more thought yesterday, and realized that I need to have a database storing data just like you mentioned, but also, I should have a Redis (or some other in-memory solution) as a cache layer for queries to that database. What do you think? – Gabriel Robaina Dec 13 '19 at 11:17
  • Thank you for your support @DocBrown. Would you mind writing down an aswer so that I can give you points for it? – Gabriel Robaina Jan 8 at 12:15
1

I think implementing this using something like SqLite looks pretty straightforward, with

  • just one table containing the parameters (as primary key) and
  • one BLOB column for the CAD model, and
  • a few columns for statistics.

Yes, you will have to implement the logic to manage the cache by yourself, but the particular population and eviction requirements you mentioned don't look very complex to me. I think it is unlikely you will find an "out-of-the-box" solution easily which suits your needs. If you expect the structure of the parameters to change often, a No-SQL solution may be suited better, of course.

I doubt SQL or NoSQL willl make a big difference for what you are trying, but I would actually keep things as simple and lightweight as possible at the start. You can always switch to something more complex when it really turns out the simple solution is not sufficient. MongoDB and CouchDB look both larger and more complex than you may need (but I am not an expert on those systems).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.