0

I currently have a file processing task that runs on an on-prem VM, it's a .NET executable that also calls other services such as ffmpeg (for videos). As is, the task runs fine, but it's really only possible to process one file at a time due to computational resources.

I'm looking to change the architecture in such a way as to allow multiple files to be processed at once, theoretically well beyond the number that could be processed even by giving the VM additional resources (which isn't an option anyway).

Based on my research I think that what I'm looking for is probably some kind of container based solution, but I'm new to the concepts of containers and so I'm hoping that someone can help me ensure I'm on the right lines. Because I'm new to containers I think that Azure Container Instances is the way to go - it gives me the least to manage.

I think the rough idea I'm trying to map out is as follows:

  1. Create a new container instance from an image
  2. Start the instance
  3. Download the file to the container from Azure Blob Storage
  4. Process the file using the existing executables
  5. Stop and delete the instance

In order to do that, I'm going to need to do the following: create a container image that includes all of my executables; find some way to trigger the creation of the instance when a new file is added to Blob Storage; and figure out how to make the task to download and process the files start automatically once the container has started.

Now I believe that all of this is possible, and I'm hoping it seems sensible, but if anyone can spot a flaw in this plan then please could you highlight it for me.

Likewise if anyone has any tips on how to achieve my goals then please share, or share a source where I can find out more about what I need to learn.

  • I think that would work, the unusual step is deleting the instance immediately after processing a single file. Why not keep the instance alive to process another file? – Ewan Jan 28 '19 at 13:14
  • @Ewan thanks for your feedback. The reason I'd delete the instance would be because I wouldn't want to have the instance running (and incurring costs) while waiting for a new file. I'm imagining each new file would spawn its own instance when it is created in Blob Storage. – Matt Shepherd Jan 28 '19 at 23:48
  • The reason behind the above thought process is that I expect I might get a number of files in quick succession that I want to process all at the same time (in their own container instances), but I might then not get any for quite a while after that (say a number of hours), and I don't want to be being charged for the instances I spawn for the first set of files while waiting for the next set. – Matt Shepherd Jan 28 '19 at 23:50
  • I guess the unusual part is that that only works for a very specific load profile. large spikes where you can afford exactly the instances you need and long stretches of nothing. You lose a bit of chargable time during spin up, so say you have 10 tasks, its cheaper to have 5 containers run two each before shutting down than 10 run 1 each. – Ewan Jan 29 '19 at 0:11
  • also, looking into it, its actually the container group which controls the cpu resources. you need to spin up container groups. You may find it cheaper just to assign more cpu to a single container. In my experience the cheapest cpu is an on premises one – Ewan Jan 29 '19 at 0:21
1

I like your proposed solution, but I would add that maybe a serverless architecture would suit you well. Did you take a look at Azure Functions? They can be triggered by some events like the upload of file to Blob Storage. See here.

The thing is that with a serverless architecture for your service, you will have a auto scalable solution, as well as less operation tasks (managing servers).

| improve this answer | |
  • Thanks @Cassiano, I originally though serverless was what I was looking for too, but then I got a bit confused by how Azure functions actually worked. Is there a way to ensure that e.g. ffmpeg among others are present on the machine where the function is being executed? This was the bit that made me think I needed to use containers. – Matt Shepherd Jan 30 '19 at 7:26
  • I see. As I'm not really a .NET developer, I don't know what kind of dependencies we are talking about here. These are system dependencies, I assume? Maybe there is a way to install them at runtime? It should be faster than spinning a new instance, depending on their size. – Cassiano Jan 30 '19 at 14:53
  • Also, if your dependency is available as a NuGet package, you should be able to add it as a dependency for the function. See here – Cassiano Jan 30 '19 at 15:01
  • It's a call out to a separate program (not .NET), and I need that program to be present on the machine. I think from your first comment what I could do is download a copy of the executable (it doesn't even need an install) as part of the program, that might work. Thanks! – Matt Shepherd Feb 1 '19 at 2:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.