2

The problem was already discussed here. But there was not consensus on this topic.

I have some thoughts on how insert operation can be implemented for some popular file systems. If FS has extent-based structure (eg ext4, ntfs, probably btrfs) we can utilize this feature to make modification of middle file's parts independent of other its parts. I suppose it will require handling of length for each such part independently of other's. But in some situations advantage may be drastically big. From my experience: I sometimes faced with the problem of slow processing of one big file. So the functionality may be in demand. And I don't even mention databases here which always required such funcionality.

A good usage case would be distribution of read/write operations across multiple disk layers. It is rather relevant for modern multidisk (often ssd-based) multicore and mulithreaded SMP (or even NUMA) systems.

I already took a look at MPI-IO v.2 system. It has something simialar (especially regarding parallel processing). But it does not provide dynamic file resizing capability which I suppose to introduce.

I need your opinions on this topic. What drawbacks / shortcomings can arise in trying to implement such feature. One of such drawback can be odd irregular length of contigous file blocks resulting in breaking memory mapping mechanisms for example. I just want to notice that someday such functionality will be implemented because:

  1. Data volumes grow rapidly and so files also do
  2. Parallel processing is already a modern reality. And there are still no other good ways of future technology improvement in terms of performance.
  • 1
    Of course, there are reasons why file systems were composed of blocks in the first place, and you would have to give up those advantages to create your new file system. Speed is one of them; require addressing each byte individually and you will take a tremendous performance hit. It will also cost you dearly in terms of storage efficiency. Is it worth it, considering that the workarounds for ordinary block-style file systems are trivial and well-understood? – Robert Harvey Jan 14 at 15:58
  • 1
    So yes, it is possible. Anything is possible given enough time, money and donuts. Is it practical? Probably not. – Robert Harvey Jan 14 at 16:02
  • 1
    @RobertHarvey, yes of couse I took into account all opinions presnt in linked question. But there were almost no alternatives to the mainstream fs design present. And I am sure that main obstacles are not the granularity of I/O acesses (one can and should continue to operate with independent 4k block I'm sure) – Anton Astafiev Jan 14 at 16:06
  • 1
    To find out what drawbacks will arise when you try to implement a feature, you could try to implement a feature. – user253751 Jan 14 at 16:41
  • 1
    Databases just roll their own insert mechanism. The block nature of the devices doesn't stop them from managing inserts, which they can each do with their own custom application-specific approach, probably way better for them than having insert in the file system. – Erik Eidt Jan 14 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.