Made an example because it did not fit the comments: It's PHP but I believe quite readable for you to understand the concept.
<?PHP
/**
* Simple example of loading the git repository
* Then do a conversion on some files
* Then publish them in the /public repository
*
* Handles loading the repository from github and understands whether it
* needs to clone because it's the first run or update.
*
* Timing on slow internet connection:
*
* update: 0.96 - 1.53 seconds
* new clone: ~100 seconds
*
* Clearly most time happens in the downloading of the repository.
*
* Uses rename() as atomic function to allow updating and processing to
* happen without the users having issues. When processing is totally complete
* the public directory is instantly replaced.
*
*/
$time_start = microtime(true);
//git repo
$repoDir=__DIR__.'/linux/';
//public directory
$publicationDir=__DIR__.'/public/';
//temporary directory for publication to be built
$newPublicationDir=__DIR__.'/tmp-'.time().'/';
//archive location for old version (for fallback in case of errors)
$archiveDir=__DIR__.'/archive-'.time().'/';
//ensure we have all directories, yes ugly without error catching
@mkdir($repoDir);
@mkdir($newPublicationDir);
@mkdir($archiveDir);
//detect a new install or an update of the repository
if(!file_exists($repoDir.'.git')) {
//new install
exec('git clone git://github.com/torvalds/linux.git --depth 1');
}else{
//update
exec('cd '.$repoDir.'; git fetch --depth 1');
}
//processing your contents (no idea what you do on them)
//in example we convert newlines of a textdocument to <br> tags
$documentationFile=$repoDir.'Documentation/aoe/aoe.txt';
file_put_contents(
$newPublicationDir.'index.html',
'<h1>Converted document from git</h1>'.nl2br(file_get_contents($documentationFile))
);
//back-up the current publication
if(file_exists($publicationDir)) {
rename($publicationDir, $archiveDir);
}else{
@mkdir($publicationDir);
}
//add the new publication
rename($newPublicationDir, $publicationDir);
$time_end = microtime(true);
$time = $time_end - $time_start;
echo "ready in $time seconds\n";
This example works with the linux repository which is quite big in my opinion with a lot of irrelevant history. It uses the ideas discussed in the comments. Generally this thing will update in 2 seconds. So, while waiting for your exact numbers about repository size and processing times, I think this should be sufficient to solve your problem.
queue
I understand you now use a queue etcetera. You can keep all of that if you want and process it like the example.
simpler alternative
Other more simple alternative (in my opinion) is: Just web servers, no workers.
Start one or more web workers. They process the script at boot time. When they are done (validate if public directory is full) they open the port and they are ready for service. Maybe they should notify some load balancer or whatever to let system know they are ready for requests.
Then just run a cron (scheduled task) to update them every x minutes/seconds whatever. That will ensure you don't run the task multiple times at once and you don't have to configure your hooks.
If that is not allowed you can also choose to use the hooks etcetera with your queue but I think it makes the process just too complex if your question is correct and with all relevant information.
processing in workers
Now you could choose to put a separate layer in between with the workers, to parse the data from the repository, if you really need that. You could use the same principles but just publish on a shared disk. The web servers read that disk and you are done.
Alternative to the shared disk you could put the worker results also into a git repository and use the same script pointing to a azure hosted git repository which just loads the results.
On Azure you could process this script I think with: http://www.hanselman.com/blog/IntroducingWindowsAzureWebJobs.aspx
For the shared storage you can take a look at File Storage in this document:
http://azure.microsoft.com/en-us/documentation/articles/storage-introduction/
There is also a quite clear listing of the alternatives. In general you can share the storage on azure between multiple types and multiple instances of servers which fits your needs.
beware I put in archive directories, remove that for sure in production cause it keeps filling your filesystem.
if (!repoOnDisk) Fetch()... and this would be unreliable from a publishing speed perspective depending on how often the worker role gets recycled (which wipes out the local disk, right?) – Joel Martinez Aug 21 '14 at 18:48