Skip to main content
Software Engineering

Return to Answer

added 379 characters in body
Source Link
nanotek
nanotek
  • 301
  • 2
  • 9

Based on @Mike_Robinson's comment I was able to better research the problem.

Essentially I've described a High Performance Compute (HPC) cluster with different instance types. This is referred to as a heterogeneous compute cluster For historical on prim clusters it seems a common tool for workload management is slurm. AWS has AWS ParallelCluster which seems to have a slurm base and has support for heterogeneous clusters.

This is also a similar problem solved by AWS Batch

Searching for task scheduler system design you can find a number of resources like drop box's task framework. There are a-lot of references to this as an interview question as well as a number of discussions on youtube with these keywords

Based on @Mike_Robinson's comment I was able to better research the problem.

Essentially I've described a High Performance Compute (HPC) cluster with different instance types. This is referred to as a heterogeneous compute cluster For historical on prim clusters it seems a common tool for workload management is slurm. AWS has AWS ParallelCluster which seems to have a slurm base and has support for heterogeneous clusters.

Based on @Mike_Robinson's comment I was able to better research the problem.

Essentially I've described a High Performance Compute (HPC) cluster with different instance types. This is referred to as a heterogeneous compute cluster For historical on prim clusters it seems a common tool for workload management is slurm. AWS has AWS ParallelCluster which seems to have a slurm base and has support for heterogeneous clusters.

This is also a similar problem solved by AWS Batch

Searching for task scheduler system design you can find a number of resources like drop box's task framework. There are a-lot of references to this as an interview question as well as a number of discussions on youtube with these keywords

Source Link
nanotek
nanotek
  • 301
  • 2
  • 9

Based on @Mike_Robinson's comment I was able to better research the problem.

Essentially I've described a High Performance Compute (HPC) cluster with different instance types. This is referred to as a heterogeneous compute cluster For historical on prim clusters it seems a common tool for workload management is slurm. AWS has AWS ParallelCluster which seems to have a slurm base and has support for heterogeneous clusters.