The architecture and technologies to use for a secure, fast, reliable and easily scalable web application [closed]

Question

^ For actual questions, skip to the lists down below

I understand, that his is a vague topic, but please, before you turn the other way and disregard me, hear me out. I am currently doing research for a web application(I don't know if application is the correct word for it, but I will proceed w/ that for now), that one day might need to be everything mentioned in the title. I am bound by nothing. That means that every language, OS and framework is acceptable, but only if it proves it's usefulness. And if you are going to say, that scalability and speed depend on the code I write for this application, then I agree, but I am just trying to find something, that wouldn't stand in my way later on.

I have done quite a bit reading on this subject, but I still don't have a clear picture, to what suits my needs, so I come to you, StackOverflow, to give me directions. I know you all must be wondering what I'm building, but I assure you, that it doesn't matter. I have heard of 12 factor app though, if you have any similar guidelines or what is, to suggest the please, go ahead.

For the sake of keeping your answers as open as possible, I'm not gonna provide you my experience regarding anything written in this question.

^ Skippers, start here

First off - the weights of the requirements are probably something like that (on a scale of 10):

• Security - 10
• Speed - 5
• Reliability (concurrency) - 7.5
• Scalability - 10

Speed and concurrency are not a top priority, in the sense, that the program can be CPU intensive, and therefore slow, and only accept a not-that-high number of concurrent users, but both of these factors must be improvable by scaling the system

Anyway, here are my questions:

• How many layers should the application have, so it would be future-proof and could best fulfill the aforementioned requirements? For now, what I have in mind is the most common version:
  1. Completely separated front end, that might be a web page or an MMI application or even both.
  2. Some middle-ware handling communication between the front and the back end. This is probably a server that communicates w/ the front end via HTTP. How the communication w/ the back end should be handled is probably dependent on the back end.
  3. The back end. Something that handles data through resources like DB and etc. and does various computations w/ the data. This, as the highest priority part of the software, must be easily spread to multiple computers later on and have no known security holes. I think ideally the middle-ware should send a request to a queue from where one of the back end processes takes this request, chops it up to smaller parts and buts these parts of the request back onto the same queue as the initial request, after what these parts will be then handled by other back end processes. Something map-reducey, so to say.
• What frameworks, languages and etc. should these layers use?
  1. The technologies used here are not that important at this moment, you can ignore this part for now
  2. I've been pointed to node.js for this part. Do you guys know any better alternatives, or have any reasons why I should (not) use node.js for this particular job.
  3. I actually have no good idea, what to use for this job, there are too many options out there, so please direct me. This part (and the 2. one also, I think) depend a lot on the OS, so suggest any OSs alongside w/ the technologies/frameworks.

Initially, all computers (or 1 for starters) hosting the back end are going to be virtual machines.

Please do give suggestions to any part of the question, that you feel you have comprehensive knowledge and/or experience of. And also, point out if you feel that any part of the current set-up means an instant (or even distant) failure or if I missed a very important aspect to consider.

I'm not looking for a definitive answer for how to achieve my goals, because there certainly isn't one, for I haven't provided you w/ all the required information. I'm just looking for recommendations and directions on what to look into. Also, bare in mind, that this isn't something that I have to get done quickly, to sell and let it be re-written by the new owner (which, I've been told for multiple times, is what I should aim for). I have all the time in the world and I really just want to learn doing something really high-end. Also, excuse me if my language isn't the best, I'm not a native.

Anyway. Thanks in advance to anyone, who takes the time to help me out here.

PS. When I do seem to come up w/ a good architecture/design for this project, I will certainly make it an open project and keep you guys up to date w/ it's development. As in what you could have told me earlier and etc.

For obvious reasons the very same question got closed on SO, but could you guys still help me?.

Answer 1

I think it would be useful for you to look at the method by which some big players like Twitter, Google, Facebook, Amazon, etc. grew their applications from humble beginnings to global leaders. I recently went through articles on the High Scalability Blog - here's the points that came back again and again:

• Keep it simple, complexity will come naturally over time.
• Automate everything, including failure recovery.
• Iterate your solutions, be prepared to throw away a working component when you want to scale it up to the next level.
• Use the right tool for the job, but don't be afraid to roll your own solution.
• Use caching, where appropriate.
• Know when to favor data consistency over data availability, and vice versa.

Above all else, don't get hung up on the choice of web application framework and architecture when you are starting out, because you can always rewrite parts of it if required. Facebook famously used PHP in the beginning and then later wrote a PHP to C++ tool to allow them to compile in C++ to speed things up. When they got popular, Twitter moved its back end messaging platform from Ruby to Scala (a JVM language) for reasons of performance.

Answer 2

Sir I recommend the following language because it is a full-stack language, it is growing enmasse, and as the most compiled language (among other characteristics) it has immense staying power for your long term plans.

/*************************3_REASONS***********************************/

1. Full-Stack. I started writing it, like many perhaps, in an internship around 2000. Under the impression, for many years, with many colleagues, that JavaScript was a "Toy-Language". A "Toy", like PlaySchool would make - plastic and shiny. Eating humble-pie a decade later, I slowly grew to realize that my long-held impression was wrong.

Prior to AJAX, we underestimated simplicity as disability because we were not seeing extensible flexibility. No one could have guessed how flexibile JS could be. Since the asynchronous adaptation of JS over HTTP, we now see it implemented in just about every environment, if not every environment, from Operating System, Database, Server, Front-End, to Mobile Operating System.

Wikipedia labels JS a multi-paradigm language. Frankly, as an old C++ guy, I find it impressive that JS has more than a single static paradigm, let alone, conducive to so many it encounters. But that is only half the story.

2. Flexibility - as it turned out, being a toy meant that the toy matchbox inspired a radio-controlled car, then a go-cart, on to a model-T, ferrari, a monster truck, street-bike, and what ever else anyone needed. Here is the centrifical point, CouchDB, MongoDB, RiverTrail, three.js, node.js, jQuery siblings, micro.js mobile libraries, PhoneGap, Compiled JavaScript, Testing frameworks, a new building framework(grunt.js), Boilerplates, Bootstraps, MVC derivatives, WebRT, and now Windows 8 - JS is everywhere. More than ever, JS is everywhere. One might ask why?

3. Staying Power - JS is not successful in so many diverse ecosystems because it is flexible; it is successful because it is capable.

Transmission - with JSON as an effective data-transmission mechanism, the same object-literal can be transferred Front-to-Back, OS-to-OS, without data-translation, without language-linking, and without re-interpretation. That is remarkable.

Education - if you had the opportunity to learn a single language and apply it cross-profession... wouldn't you? More than ever, JS is a learn once and apply anywhere language. Coming out of college, many of us experienced a painful gauntlet of learning-curves. If you could, would have traded for a single language? I would.

Growth - JS will continue to grow, and this is not to mention its current place of prominence as the behavior-layer of the Internet. Microsoft swallowed hard to adopt it into the operating system paradigm, not because they could; but because it is good. This, is precisely what we could not have anticipated, in 2004.

/********************HOW-DOES_THIS_PERTAIN-TO-US-NOW***************************/

As I mentioned in the comments, I am currently researching...for similar stack requirements unrelated to my personal insights above. I also need a '10' - score security clearance architecture.

Is JS capable or incapable of being secure? To me this is the challenge in this architecture.

Given the track record. I would not underestimate it again. Validation and Filtering are key. But then again the MySpace worm XSRF was unexpected and history repeats itself.

/*********************DIRECT-QUESTIONS_DIRECT-ANSWERS**************************/

that the program can be CPU intensive

• Something amazing may be happening at IBM with RiverTrail. The notion is multi-threaded JS, but I am merely the messenger.

How many layers should the application have

• A question of abstraction is a tradeoff, but I would point out that: If it has many layers, it has many 'interchangable-parts'. This worked out well for Eli Whitney, I propose the same holds true in the evolving-web. Especially JS.

so it would be future-proof

If you learn JavaScript, well, you can be virtually assured you will be using it until the web that we have, is no longer. JS is as "future-proof" as I think a case can be made (unless you read binary).

Completely separated front end

Yes, this is a beautiful thing. Options are AJAX via DWR. A popular oneis REST and data-transfer with JSON. Fully-duplex Web-Sockets sound good too.

Some middle-ware handling communication w/ the front end via HTTP.

Node.js is highly effective. It yields enormous potential.

The back end...various computations w/ the data.

Here is a description of what can be accomplished with CouchDB: It is basically a JSON database, that uses functions for stored-procedures.

must be easily spread to multiple computers later on

JSON.

and have no known security holes

I was informed on the IT Security exchange this seems to be the current industry response:

Anything can by made insecure through usage. You still need to validate all your inputs.

• And validating inputs is like beating encryption. It is an arms race. So this is the challenge as I see it.

What frameworks, languages and etc. should these layers use?

My choice, given infinite resource, would be: AJAX via DWR. MVC via backbone.js, Template by _underscore.js, AMD with require.js (very important), RWD with CSS3 Modernizer and RESS, HTML5 with HTML5boilerplate, data-transfer with JSON. Server - Node, Database - Couch.

why I should (not) use node.js for this particular job

Exhibit A. It is command-line and less adopted on windows. If command line is entirely foreign to you. That is a good reason to not use it. The learning-curve would be a barrier of entry. It had some adoption by Microsoft last year, even now in Azure, evidently, but much remains to be seen. It may be safe to say that robust implementation is non-trivial.

Exhibit B. Node.js is still very early in it's overall potential lifespan. It is immature in some respects and I suppose Ryan Dahl, would be the first to point out which respects those are.

Exhibit C. We, as a community need to ask these security questions, in order to advance these topics. There is a saying that open-source is supposedly more secure because there are more eyeballs. I've seen this to be true in practice.

suggest any OSs alongside w/ the technologies/frameworks.

I would recommend a flavor of Linux/Unix. Perhaps a VM is sufficient for others.

point out if you feel that any part of the current set-up means an instant (or even distant) failure

Good question. To me, this is where interchangable-parts is key. Think of this question in regards to your car. Instant or distant failure. Is this traumatic to you? Well, no not really because we can generally get the new part at the corner mechanic. JS is like that. In Node.js it is called the npm. On the front it is micro.js. On the back it is the data versioning mechanism, SHA tags(I think).

One reason JS can continue to adapt, is that it can change out it's parts. When something better like AMD comes along, it can meld into the namespace at runtime and not fry everything else.

I'm just looking for recommendations and directions on what to look into. I have all the time in the world and I really just want to learn doing something really high-end.

It is important to mention the coding-enjoyment-factor. It is easy to discredit, but... if you are going to spend a significant portion of you life on this endeavor, you would do well to pick an architecture that is actually enjoyable to extend. This is not true of all languages. Some can be nightmarish, and to each his own. I very much enjoy the empowerment, and flexible freedom of Full-Stack JavaScript. Capability has only expanded to date.

I will certainly make it an open project and keep you guys up to date w/ it's development

If you haven't discovered it already, use GitHub. It is worth the command-line and bash scripting learning curve if you haven't ventured there already. I scaled-back tortoiseGit personally.

/*****APOLOGIES-FOR-LENGTH**********/ /*****And any typos*****/

I hope it helps. All the best! Nash