2

I'm wondering what the best practice would be for the following task:

I start a program on a remote node, which will generate subsequently new files in a publicly accessable folder. Whenever a new file is created on the remote node, I need to transfer this file to another node, say local node / client immediately.

Please note, the nodes are two regular Windows 10 Desktop computers connected by a LAN in our lab room.

I could monitor the remote directory periodically, but this would not transfer the data as soon as it is available. I'm just wondering if there is already a principle or design pattern I could apply or if I do have to set-up some own client/server communication. I think this questions is programming language agnostic however I'm using C#.

  • 2
    This problem doesn't seem like it would need a special design pattern to solve it. Just code up a solution. – Robert Harvey Aug 17 '18 at 14:38
  • Thank you Robert. I was thinking the same. Thanks for confirmation. – stephanmg Aug 17 '18 at 14:40
  • For what it's worth, this isn't how design patterns should be used. Searching for a design pattern to solve a coding problem is like trying to find a word in the dictionary by looking through the definitions. Instead, become familiar with all of the major software patterns so that you can recognize when they are applicable. – Robert Harvey Aug 17 '18 at 14:42
  • 2
    If you're doing this in C#, there's a FileSystemWatcher class that sounds like a good start: docs.microsoft.com/en-us/dotnet/api/… – Phil N DeBlanc Aug 17 '18 at 14:46
  • 1
    Unfortunately there isn't enough information in your question for us to offer advice on techniques or technologies. Your nodes could be microservices or gateways or .. ? – Robert Harvey Aug 17 '18 at 14:47
2

You need to know not only that a file exists but that it is complete and ready to be copied. Otherwise, wouldn't there be a danger of copying over an incomplete file?

You could include completeness indication in each file, like an end-of-file marker. (But what happens if the file creator dies with an incomplete file?)

You could use the existence of another file as an end marker. As files are created, the marker file is touched, updating it's change time. Your local system would then be free to copy all files older than the end-marker file.

You could use signaling of some other form from the remote system to the local system to prompt the local system to come and get a completed file.

  • Moving the file from an "intermediate" to a "final" folder once the write is finished is another option. – D Drmmr Apr 17 at 8:41
0

There are essentially two approaches that you could use here; the first is simply to haev the remote node be the one to initiate the transfer as that node will know exactly when the file is ready to be sent off.

The second option is a follow-up of the first essentially where the local node would be connected to the remtoe node and await a message that the node is done. In a sense it is the same as the first option, but the local node would be the one to perform the file moving.

0

I had to do a similar thing, triggered when a file arrived in an FTP directory.

It was a while ago but I believe we used incron

https://www.howtoforge.com/tutorial/trigger-commands-on-file-or-directory-changes-with-incron/

I think .net has System.IO.FileSystemWatcher which can be used to a similar effect on windows PCs.

https://superuser.com/questions/226828/how-to-monitor-a-folder-and-trigger-a-command-line-action-when-a-file-is-created

  • Actually, FileSystemWatcher is part of .NET Standard 2.0, so it will not just work in .NET Framework on Windows but also in .NET Core 2.0 on Linux, macOS, and Windows, Mono 5.4 on BSD, Linux, macOS, Windows, and all other supported platforms, Xamarin.iOS 10.14, Xamarin.Android 8.0, Xamarin.Mac 3.8, and UWP 10.0.16299 in addition to .NET Framework 4.6.1. – Jörg W Mittag Aug 18 '18 at 11:52
0

If you are able to modify the program that will be running on the nodes, you could change it to POST the contents to an HTTP server at the destination.

If you aren't able to modify the program, but it is able to redirect its output to stdout, you could write a wrapper program that launches it with a redirected stdout stream, and forwards it to the destination (again, HTTP POST).

0

Personally I think the main software engineering principle that would come into play for what you have described is not re-inventing the wheel. I don't know the specifics that might come into play, but I would look at it in terms of what is the minimum amount of logic that you need to put in to this to make it work.

For example, in my opinion if you end up writing code to transfer files between these computers, you are doing it wrong, because you can easily setup network folder sharing between two Windows 10 PCs. So at most it seems like you need to write code to monitor for changes and do the local copying of the shared files to the local drive. Depending on what exactly you are doing, you might just have to write a program that refreshes the shared folder every few seconds and uses normal Windows copy commands to copy the file content locally. If you are getting to that point, there is probably off the shelf software that does this automatically, although I don't know of any obvious ones that are real time. If you search for "real time folder sync" there are definitely possible options out there.

0

There are ways for you to watch a folder, depending on the technologies you are using, so that is not really an issue. You also need to determine your need for the files: do you need them right away? Do you need them at a specific time? This will help with your folder read strategy.

But the main issue for you here is the integrity of your data. If you decide that you need the files right away, how do you know they are complete? Even if you need them sometime later, how do you know they have been successfully transferred, and are not corrupted? A common strategy here is to create an additional file with the info you need: probably the transferred file name and a hash value for it. You'll get the hash value prior to sending, then re-calculate after getting the file. You compare both values, and if they match, then you know you have a successful transfer and can continue your process.

In this scenario, you won't need to look for the actual files you need, only the files with the hash data. They should be transferred last, after your original files. These can be .txt, .js or .xml, whatever you want, but they will be small, ensuring rapid transfer and reliable process.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.