0

I've built a flatfile db system that will work like mini-noSQL system and it split db to smaller files, and I did a SplFixedArray wrapper for it with a method for searching usage built on preg_grep.

Everything working very good till now, but if the array became more than billions of indexes how to handle it? When searching it, it will iterate over the small files then get the matches and put it in a temporary fixedArray so, what if it found very large matches?

How the noSQL dbms do that?

NOTE: If you wonder why I do that without a real noSQL solution, I'll say because I want to know more.

Improve this question
1

Generally a buffer is used for this sort of operation, often a FIFO (First In, First Out) data structure where when new information comes in that would be beyond the buffer capacity, the oldest (first) info in the buffer is purged out to make room for the new data.

This is sort of like reading a file line by line, outputting the results wherever you want them, and then reading the next line. The same goal is sought - to avoid crashing your script due to excess memory usage.

So, what happens when you have a dataset that is just too big to keep in memory? In short - you can't continue. You either need to stop the work and return something like "result set too large - here are first n occurrences", or you need to output the results to a file or some such that lets you get things out of RAM so you can keep working.

A canonical example has become SAX vs DOM in XML processing. When data sets are large, the downside is that you lose the simplicity and convenience of being able to just shove everything into RAM and not worry about it. It almost always requires a shift in the strategy and coding techniques to handle such problems.

"He who increaseth in raw data, increaseth in coding complexity." - computerized Proverb

Improve this answer
6
  • as i understood, that is why mysql generates a data that we can loop over it with while loop ? while loop get e.g: line by line !, if it is true, how can i generate a data that uses while loop ! i just want to send a buffer then flush it to send the next , is it true ? – Mohammed Al Ashaal Nov 18 '13 at 6:10
  • Yes, with SQL usually in your chosen there is an implementation where it, say, returns the first 10 results (or even just 1 at a time), and you just loop through them. With each execution of the loop the language either gives you the next cached item in the set, and/or it goes to the database to sask for more. But yes, in this instance you will probably want to get 1 result, deal with it, then go onto the next. Minimal RAM usage and it works as well on 1000 or 1 billion results - if your script doesn't time out :) – BrianH Nov 18 '13 at 6:25
  • many thannnnnnnnnnnx to you , you only how answered my Q, any suggestion from you for the best way i do it, or any idea could be good ? – Mohammed Al Ashaal Nov 18 '13 at 6:29
  • could you suggest any way to send one by one result :) – Mohammed Al Ashaal Nov 18 '13 at 14:46
  • I think, based on your explanation of what you have so far, that I would alter your temporary array to output its results to perhaps a temporary file each time a new piece of data is pushed. There's lots of ways to do it, but that's one ways. YMMV, and it depends on more things than I can know about what you want to accomplish. – BrianH Nov 18 '13 at 19:11
0

You basically stumbled upon the whole OS idea of paging/virtual memory...what you are thinking about is really only a little different from the issue that OS developers have been dealing with system memory requirements becoming too large to handle in physical memory.

Lots of really smart people have been working on this idea for a very long time...you could do well by researching the various solutions.

Improve this answer
3
  • now i reached to this point: i should send the result one by one , how really i do that ? – Mohammed Al Ashaal Nov 18 '13 at 15:54
  • For ex: u have 100 files to search on, suppose u found 10 results from searching the first 30 files, u will then save a reference to last file u searched on & return the 10 results to user, the user calling the function again u will start searching from file No 31 and so on... The user will have to keep calling your fetch function until he receive NULL which mean u searched all files u have. that's simple approach u might also think about caching, buffering, indexing, .. – Basemm Nov 19 '13 at 4:35
  • @Basm.Egy i'll try what you said, also if you have a real example would be good , many thanx to you ^_^ – Mohammed Al Ashaal Nov 19 '13 at 6:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.