3

Assume that we have a 4GB RAM processor. Let's say we want to play a 4GB movie in VLC player. Now, processor first starts VLC Process and then starts to load 4Gb data. My question is how does this data transfer occur, Is it like getting almost all of 4Gb and storing some in swap area and bringing whenever it is required or first getting the starting data and continuosly stream data between RAM and hard disk drive?

3

At the level a user-mode process, there are two basic ways in which sequential file IO like playing a movie from start to finish can take place:

  • The process can allocate a buffer, which is typically sized somewhere between a few kilobytes up to a megabyte or so, but rarely any larger than that, and then asks the operating system to fill it up. When it is finished with the data from that buffer, it then asks the operating system for more, and so on, until it reaches the end of the file. As a slight variant on this, in order to reduce latency it may allocate two buffers, and ask the operating system to fill the second one while it is processing the data from the first, and then swap between the buffers when it finishes processing the data from the first.

  • Or, it might instead ask the operating system to "map" the file (or a section of it) into memory. In this case, the operating system tells the processor that the memory is currently not available, and then when the reading process tries to use the memory, the processor notices this and switches back to the operating system, causing the operating system to load in the next page of data from the disk. This may be very slightly more efficient because the operating system is able to manage the memory itself rather than having to allocate its own internal buffers and then copy the data after it has been read into the process's buffers, but the effects are likely very difficult to notice in most cases.

My suspicion is that most media players use the first approach, probably with the variant using two buffers. The third can be more efficient, but it has a problem: you need to allocate addresses for the entire content of the file[1], which can be a problem with large files under 32-bit operating systems, so any media player designed to work under such a system is unlikely to do this.

At the operating system level, the kernel sees a series of requests for sections of files to be loaded, and the filesystem driver translates those to requests for disk blocks to be loaded. The IO manager allocates memory blocks from its buffer pool (aka IO cache) and asks the disk device driver to transfer the disk blocks' contents into the memory blocks. Once this is done, either the data is copied into the user process buffer (case 1 above) or the system buffer page is mapped into the process's memory space using a page table and the process resumes (case 2).

A modern operating system is also likely to notice that the process is reading blocks sequentially and start prefetching them before requests come in to improve performance.

[1] - you can just map part of the file and then move the window around, but my understanding is that very few programs actually do this.

  • Does that mean that physical memory need not bother about the size of data file as irrespective of the size, the process only make requests for buffers in the size of Mega bytes? – AV94 Jul 21 '16 at 17:24
  • 1
    Yes. The only effect that having more memory will have is that (as long as it isn't already being used) the operating system will use it to increase the size of its buffer pool/IO cache, which might make things a bit smoother if you start jumping around in the file rather than just watching straight through from beginning to end. – Jules Jul 21 '16 at 17:32
  • Thanks, till now I though video files takes much of RAM. Could you if possible can specify me what kind of processes actually take significant amount of physical memory? – AV94 Jul 21 '16 at 17:41
  • 1
    Mostly things that handle large amounts of data that can be changed -- image and video editors, games, and so on -- or programs that jump around a lot in files and therefore need large caches -- databases, compilers, and things like that. – Jules Jul 21 '16 at 17:47
  • one more question, This filling of memory bffer is processor dependent right? i.e. when data transfer starts, does the data come from disk to I/o buffer to processor and processor stores it in RAM or it happens directly from disk to memory without processor's intervention? – AV94 Dec 5 '16 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.