I currently have a file processing task that runs on an on-prem VM, it's a .NET executable that also calls other services such as ffmpeg (for videos). As is, the task runs fine, but it's really only possible to process one file at a time due to computational resources.
I'm looking to change the architecture in such a way as to allow multiple files to be processed at once, theoretically well beyond the number that could be processed even by giving the VM additional resources (which isn't an option anyway).
Based on my research I think that what I'm looking for is probably some kind of container based solution, but I'm new to the concepts of containers and so I'm hoping that someone can help me ensure I'm on the right lines. Because I'm new to containers I think that Azure Container Instances is the way to go - it gives me the least to manage.
I think the rough idea I'm trying to map out is as follows:
- Create a new container instance from an image
- Start the instance
- Download the file to the container from Azure Blob Storage
- Process the file using the existing executables
- Stop and delete the instance
In order to do that, I'm going to need to do the following: create a container image that includes all of my executables; find some way to trigger the creation of the instance when a new file is added to Blob Storage; and figure out how to make the task to download and process the files start automatically once the container has started.
Now I believe that all of this is possible, and I'm hoping it seems sensible, but if anyone can spot a flaw in this plan then please could you highlight it for me.
Likewise if anyone has any tips on how to achieve my goals then please share, or share a source where I can find out more about what I need to learn.