1

I wanted to get some advice on moving a system to "the cloud" ... specifically, I'm looking to move into some of Windows Azure's managed services, as right now I'm managing a VM. Basically, the system operates on some data stored in a github git repository. I'll describe the current architecture:

Current system (all hosted on a single server):

  • GitHub - configured with a webhook pointing at ...
  • ASP.NET MVC application - to accept the webhook from git. It pushes a message onto ...
  • Azure service bus Queue - which is drained by ...
  • Windows Service - pulls the message from the queue and ...
  • Fetches the latest data from the git repository (using GitLib2Sharp) onto the local disk and finally ...
  • Operates on the data in git to produce a static HTML website hosted/served by IIS.

The system works really well, actually ... but I would like to get out of the business of managing the VM, and move to using some combination of Azure web and worker roles. But because the system relies so heavily on the git repository on the local filesystem, I'm finding it difficult to figure out how to architect in the cloud. I know you can get file system access, so in theory I could just fetch the repository if there's nothing on disk ... but the performance/responsiveness of the system sort of depends on the repository being available and only having to fetch diffs, which is relatively quick. As opposed to periodically having to fetch the entire (somewhat large) git repository if the web or worker role was recycled, or something.

So I would love some advice on how you would architect such a system :) Ultimately, the only real requirement is to be able to serve HTML content that's been produced from the contents of a git repository (in a relatively responsive manner, from a publishing perspective) ... please feel free to ask any clarifying questions if there's something I omitted. Thanks!

  • Can't you just get the data you want with the api? developer.github.com/v3/git/trees – Luc Franken Aug 21 '14 at 14:31
  • @LucFranken no, it's too much data. The repository is rather large (several gigs), so the ability to only fetch diffs and maintain it locally is somewhat of a requirement. – Joel Martinez Aug 21 '14 at 14:51
  • ok clear, if you really need the repository that sounds like the only option. To be complete: You can actually just get the diffs: developer.github.com/v3/repos/commits as you can see here: developer.github.com/v3/media/… How many changes do you handle per say: day? – Luc Franken Aug 21 '14 at 16:22
  • @LucFranken: Not that many, definitely a tolerable amount by any accounts ... as far as update frequency. The problem I suppose is that I'd have to write this to disk, so the code would have to be written such that if (!repoOnDisk) Fetch() ... and this would be unreliable from a publishing speed perspective depending on how often the worker role gets recycled (which wipes out the local disk, right?) – Joel Martinez Aug 21 '14 at 18:48
  • 1
    Still the numbers are unclear making it hard to answer but an idea. Your feeling seems right, it can take some time to clone the repository. Except, you can also fetch the latest version: git clone github.com/torvalds/linux.git --depth 1 which will make it much better doable. Off course don't recycle the workers constantly and keep at least 2 workers active if possible. Then it doesn't matter if one goes, the other one will serve while a new one is cloning and starting. – Luc Franken Aug 27 '14 at 17:48
1
+50

Made an example because it did not fit the comments: It's PHP but I believe quite readable for you to understand the concept.

<?PHP

/**
 * Simple example of loading the git repository
 * Then do a conversion on some files
 * Then publish them in the /public repository
 * 
 * Handles loading the repository from github and understands whether it 
 * needs to clone because it's the first run or update.
 * 
 * Timing on slow internet connection:
 * 
 * update: 0.96 - 1.53 seconds
 * new clone: ~100 seconds
 * 
 * Clearly most time happens in the downloading of the repository.
 * 
 * Uses rename() as atomic function to allow updating and processing to
 * happen without the users having issues. When processing is totally complete
 * the public directory is instantly replaced.
 * 
 */

$time_start = microtime(true);

//git repo
$repoDir=__DIR__.'/linux/';

//public directory
$publicationDir=__DIR__.'/public/';

//temporary directory for publication to be built
$newPublicationDir=__DIR__.'/tmp-'.time().'/';

//archive location for old version (for fallback in case of errors)
$archiveDir=__DIR__.'/archive-'.time().'/';

//ensure we have all directories, yes ugly without error catching
@mkdir($repoDir);
@mkdir($newPublicationDir);
@mkdir($archiveDir);

//detect a new install or an update of the repository
if(!file_exists($repoDir.'.git')) {
    //new install
    exec('git clone git://github.com/torvalds/linux.git --depth 1');
}else{
    //update
    exec('cd '.$repoDir.'; git fetch --depth 1');
}

//processing your contents (no idea what you do on them)
//in example we convert newlines of a textdocument to <br> tags
$documentationFile=$repoDir.'Documentation/aoe/aoe.txt';

file_put_contents(
    $newPublicationDir.'index.html',
    '<h1>Converted document from git</h1>'.nl2br(file_get_contents($documentationFile))
);

//back-up the current publication
if(file_exists($publicationDir)) {
    rename($publicationDir, $archiveDir);
}else{
    @mkdir($publicationDir);
}

//add the new publication
rename($newPublicationDir, $publicationDir);


$time_end = microtime(true);
$time = $time_end - $time_start;

echo "ready in $time seconds\n";

This example works with the linux repository which is quite big in my opinion with a lot of irrelevant history. It uses the ideas discussed in the comments. Generally this thing will update in 2 seconds. So, while waiting for your exact numbers about repository size and processing times, I think this should be sufficient to solve your problem.

queue

I understand you now use a queue etcetera. You can keep all of that if you want and process it like the example.

simpler alternative

Other more simple alternative (in my opinion) is: Just web servers, no workers.

Start one or more web workers. They process the script at boot time. When they are done (validate if public directory is full) they open the port and they are ready for service. Maybe they should notify some load balancer or whatever to let system know they are ready for requests.

Then just run a cron (scheduled task) to update them every x minutes/seconds whatever. That will ensure you don't run the task multiple times at once and you don't have to configure your hooks.

If that is not allowed you can also choose to use the hooks etcetera with your queue but I think it makes the process just too complex if your question is correct and with all relevant information.

processing in workers

Now you could choose to put a separate layer in between with the workers, to parse the data from the repository, if you really need that. You could use the same principles but just publish on a shared disk. The web servers read that disk and you are done.

Alternative to the shared disk you could put the worker results also into a git repository and use the same script pointing to a azure hosted git repository which just loads the results.

On Azure you could process this script I think with: http://www.hanselman.com/blog/IntroducingWindowsAzureWebJobs.aspx

For the shared storage you can take a look at File Storage in this document: http://azure.microsoft.com/en-us/documentation/articles/storage-introduction/ There is also a quite clear listing of the alternatives. In general you can share the storage on azure between multiple types and multiple instances of servers which fits your needs.

beware I put in archive directories, remove that for sure in production cause it keeps filling your filesystem.

  • Thanks for the tips ... this gives me a lot of information that I can go on. The shallow clone reduces the size of the repository by several gigs, but it's still about 3gigs in size, so it takes a good bit to clone still. – Joel Martinez Sep 2 '14 at 13:04
  • tnx! You're not saying that you're storing binaries in git right? ;) – Luc Franken Sep 2 '14 at 13:11
  • Not binaries as in dlls and exes, however there are a number of image and zip binaries. – Joel Martinez Sep 2 '14 at 19:55
  • That's your next most important issue. I would say that you should try to remove them. Git has an option to handle those binaries in a more efficient way though I did just Google this and did not work with it yet: git-annex.branchable.com In general keep them out of your repository. If you need it to be in git try a sub-repository. That way you can still work fast with the normale repository. In general it's bad practice so try to find some different solution. devcenter.heroku.com/articles/git#repo-size If you need more info post specifics on this issue please. – Luc Franken Sep 3 '14 at 5:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.