I am programming in PHP and I have an include file that I have to, well, include, as part of my script. The include file is 400 MB, and it contains an array of objects which are nothing more than configuration settings for a larger project. Here is an example of the contents:

$obj = new obj();
    $obj->name = "myname";

$objs[$obj->name] = $obj->name;
return $objs;

This process repeats itself 40,000 times and ultimately is 650,000 lines long (the file is generated dynamically of course.)

If I am simply trying to load a file that is 400MB, why then would memory usage increase to 6GB? Even if the file is loaded into memory, wouldn't only take up 400 MB of RAM?

  • 4
    Why, oh why would you use a 400MB script? Can't you generate a simple datafile you can read in a streaming fashion? Or database? – CodesInChaos Feb 28 '13 at 8:36
  • CodesInChaos and @cHao: I am new to this, so I am sure there is a better way! I am actually trying to get each of these into the database. I wanted to include them, loop through them, and then each object would be a separate configuration setting in the database. Although a tad off topic, in the comments if you know of a better way, please advise or provide a link that can help me understand a bit better. I thought about one file for each object, but then I have to loop through 40k files, and that isn't working out that well either! Thanks! – user658182 Feb 28 '13 at 8:52
  • 2
    This usually happens when you allocate memory (with new()) without deleting it accordingly. Then even a few lines of code soon can consume all the memory available. – SChepurin Feb 28 '13 at 9:07
  • 1
    @user658182: Looping through 40k files should actually be less demanding memorywise; you can insert the stuff from each file into the DB, and then forget a bunch of it. But ideally, you wouldn't be doing this with PHP scripts at all. If you had a data file you read in instead, you could read it line by line, and PHP wouldn't have to compile it. – cHao Feb 28 '13 at 16:33
  • Instead of a 400MB PHP file that consumes 6GB, you need a 4kb php file that uses a binary search to search through a sorted 6GB file for what you want (if you do this, then DO NOT load the entire 6GB file into memory, rather jump around the hard disk until you get down to like a 2kb segment that you can read into memory and continue the search in memory). – Jack Giffin Jul 7 '17 at 21:37

There's not just the file. There's the parse tree that gets generated from parsing the file, the bytecode it gets compiled to, the memory taken up by the variables defined in it, etc. 6GB still sounds like a lot, but for a 650,000 line file it doesn't surprise me much.

At some point along the way, someone should probably have thought of using a database. :) Unless you're using every item in that file every time, loading half a million lines of stuff to pull out a few things is incredibly wasteful.

  • Ah, that's very informative, thank you. I did not know so much extra overhead was involved. I thought it was simply a 1 to 1 ratio, or at least, until I started manipulating it. – user658182 Feb 28 '13 at 8:54

File size is the size of the code that is executed.

Memory usage is the size of the contents of all the variables created by the execution of the code.

An algorithm computing pi may take less than 10 lines of code and need billions of bytes just to store the result.

  • -1: does not answer the question – amara Feb 28 '13 at 17:43
  • 1
    @sparkleshy - I would be happy to learn from your answer then. – mouviciel Feb 28 '13 at 18:22
  • 2
    +1: answers the question correctly, also for handling criticism politely – Michael Shaw Mar 1 '13 at 0:31

A script is typically a fixed size. It is read in and effectively executed line by line. This is your file size.

While the script is running, it can say to the computer "store this information which I have generated for me", and the computer dutifully gives it a bit of memory. This is your memory size.

Your script can ask repeatedly for more and more places to store the information it has generated, often storing the same or similar things in many different places. This can take up many megabytes, and even gigabytes. The amount of storage (in memory) is almost totally independent of script size, although the same script will normally use the same amount each time it is run.

  • -1: does not answer the question – amara Feb 28 '13 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.