4

I'm trying to think of a scalable solution for my current system. The current system is enter image description here

3 microscopes 1 processing machine

 1. 60-100GB Files come from 2-3 microscopes every 30 minutes
 2. That data is transferred to a (local) network mount of the processing machine
 3. The processing machine runs and contains the ETL(airflow)

Scaling issue

Right now it currently works well. I am concerned in the future that as the demand and load (size of file, processing times, etc..) increases we may face bottleneck(s). I was thinking of using a cluster of machines (via cloud computing or buy a couple more machines), but our network is not the fastest, maybe transferring around 100-200mbps. I worry with distributed computing the transfer speed would nullify the benefit of multiple machines.

Current thinking

I'm considering an idea where a group of machines are in a queue, if the top of the queue is not busy then the microscope can transfer the initial file to that machine and the rest of the process(Steps 2-3) can run as normal. I'm just wondering if this is a sane approach or if there is anything I can improve on.

Improve this question
2
  • @DocBrown Added the current system as a diagram (haven't made one in years so apologies for inconsistencies). The cloud computing is not currently part of the system but would be replacing the current singular machine of an ETL/orchestrated by the ETL machine.
    – user3145912
    Jan 6, 2021 at 14:42
  • An obvious first step would be to have one processing machine per microscope - unless they need to be connected. There is no bottleneck because each processing machine can have its own network connected to its own microscope. Alternatively, set up a 1Gbps (cheap) or 10Gbps (more expensive) network in your lab and bypass the IT department's network. (but do tell them what you're doing and maybe they can help set it up)
    – user253751
    Jan 7, 2021 at 2:23

2 Answers 2

Reset to default
1

I would have the microscopes transfer their files immediately, so they are not blocked for future work. The files would get transferred to a "holding" system (would need to have a large amount of storage available), that just holds the files until there is capacity to process them on the main ETL system.

Improve this answer
1

Option 1, which is more easily scalable than option 2 below, would be to create a queue. Every time a new file is dropped it's added to a queue that the processing machines pull from. To be clear though, the systems that create the files would not be paying attention to the queue at all, you want them decoupled so a backup in processing wouldn't affect the microscopes. This approach would make it very easy to add processing machines, the code and configuration would be the same on all machines.

Option 2, which I think would be simpler to initially set up, would be to have processing subdirectories on the NFS mount for each processing machine. So when processing machine 1 is ready for a new file, it would move that file from the common location into a processing subdirectory unique to that machine. Processing machine 2 would have its own directory, so they'd never process the same file. You would need to take steps to ensure that machines never looked at exactly the same time for files to prevent a race condition, but that shouldn't be too hard if you only have a few machines. A downside to this is that a new subdirectory would be needed for each additional machine, and the code would need to be modified on that machine to use it.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.