We have a Ftp server implemented. The manager wants to add reliability to it. He wants me to write incoming streams into some fast and reliable system (like hbase or redis) before writing them to server's hard disk. His point is if server has crashed (e.g power failure) before storing file, we have one copy of it in reliable system.

But I believe this method is redundant and useless because if we want to persist client file, we can simply write it into disk in first place. It also makes ftp server slow (because it needs to write data into two different place).

So am I wrong or he is? And if I'm wrong what is the best solution to use as reliability back-end?

  • 3
    When reliability is your goal, then this proposal is counter-productive, because it just introduces more points of failure. It might be a solution for performance though, if your bottleneck is writing to disk (which is unlikely, it will usually be the network). What exactly is the business problem you are trying to solve? – Philipp May 3 '17 at 9:40
  • I didn't understand how it increase performance exactly. Manager wants to store file in another fast and reliable system, before saving them into server disk. His point is if server has crashed (e.g power failure) before storing file, we have one copy of it in reliable system. But I think it's ridiculous – vakarami May 3 '17 at 9:53
  • And what happens if your reliable system crashes before storing the file because of a power failure? – immibis May 3 '17 at 12:05
  • @immibis yeah, that's another point that I've already mentioned to him – vakarami May 3 '17 at 12:32

It is redundant and useless.

If the server crashes during a file transfer, you will at best have half a file, which will in most situations not be much better than no file at all. The only situation where this configuration could help, is one where a crash happens between receiving the last byte and writing it to disk. Assuming that you are streaming files to disk while receiving them (as opposed to storing them in a buffer and writing them after the transfer is completed, but that would be stupid, because it would easily result in memory exhaustion when multiple large files are uploaded at the same time), we are talking about milliseconds here (microseconds with SSDs).

You can further reduce that time-window by having the FTP server store files on a SAN or NAS disk via network. Most of these systems will cache filestreams in memory before writing them to disk, which means that they will behave pretty much like the proposed in-memory databases.

This plan will in fact reduce reliability, because with hbase/redis you have another component in the system which can potentially fail or cause bugs.

But what you could propose as a compromise is:

  • The ability to store received files in multiple file locations simultaneously for redundancy. Admins can configure as many locations as they want.
  • Adapters for treating redis/hbase as file locations

That way your manager could set up a configuration which writes files both to redis/hbase and the filesystem (as pointless as that might be from a reliability perspective), while saner people would get two useful other features:

  • Have your FTP server replicate to offsite location(s) automatically, for example an offsite backup location or a global content distribution network
  • Use your FTP server as a middleware for dumping files directly into their favorite NoSQL database without the detour via a "normal" filesystem. This could be useful for integrating a system which only speaks FTP with a NoSQL database.
  • Thank you for the time that you have spent. So in nutshell manager is wrong and writing streams into hbase/redis isn't worthy and you suggest that ftp server writes streams into NAS because NAS is more reliable and faster. Am I right? – vakarami May 3 '17 at 10:47
  • 1
    @vakarami Yes, that's what I wrote. I wouldn't say that directly to your manager's face, though. – Philipp May 3 '17 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.