1

I am building a scientific application in matlab which handles several hundred large matrices (large as in 'very few of these will fit into ram'). Each matrix is contained within a dedicated object to handle its metadata (the property's name is .data).

I use the getter method to load the data from file into ram as soon as it is needed.

How do I decide when to unload? Most access to the object's data will be bundled, thus I don't want to unload every time I am done with getting data. Furthermore, I might need to use one or two more often than others. I do not think I can solve this within each object, so I thought I could implement a public function to clear the matrix from ram, called by an external object that decides when (and possibly, which) object is too much right now.

Does this make sense or is it a recipe for disaster? Does this count as observer, or is there something else I could use? How to identify 'good' decision criteria?

Thanks!

  • If all of your mechanisms are indeed written in MATLAB and not in an MATLAB-External Interface (MEX) with C, C++, Java, C#, or Fortran, then I'm afraid your efforts will be wasted. For 32-bit environments, memory fragmentation will catch you before your mechanism can react to it. My personal experience (back in 2007) is to buy lots of RAM - as much as your stipend can afford. Like tens of GBs. – rwong Jul 4 '14 at 8:21
  • 1
  • Large datasets are often processed in cubes or slices, as described in OLAP cube article. When I worked on face recognition (FRGC), I used a different approach: filtering by queries, followed by random subsampling (partitioning). – rwong Jul 4 '14 at 8:26
  • jaxenter.com/… – rwong Jul 4 '14 at 8:30
2

I can try to answer your question on a generic level.

You can create a centralized repository object which contains all currently loaded matrix objects and also records accesses to them. It loads the matrix objects on first access and stores them internally. All requests for these matrix objects must go through this repository object to record them correctly.

Then, the repository uses an aging algorithm such as FIFO or LRU to decide which matrix objects to unload. Once it reaches some defined threshold (such as max memory size), it finds the next candidate for removal using the selected algorithm and then unloads it.

You can tweak the threshold and the algorithm as per your requirements and the configuration of the system.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.