How do I stop designing and start architecting this project as suggested by my lead? [closed]

Question

I'm a junior developer (~3 years' exp.) and at my job we're in the process of architecting a new system. My lead developer will be the principal architect, however he's challenged me to try architecting the system myself (in parallel).

Over the course of a few iterations of brainstorming ideas and proposing what I saw as architecture suggestions, my lead has given me the feedback that most of what I've been doing was "designing" and not "architecting".

He described the difference as architecture being implementation-agnostic whereas a design is the description of an implementation. He said I need to take off my designer hat and put on my architect hat. He gave me a little bit of advice on how to do so, but I would like to ask you as well:

How do I get out of software designer mode and start thinking more like an architect?

---

Here are some examples of "designs" I came up with that weren't seen as relevant to the architecture by my lead:

1. I came up with an algorithm for loading and unloading resources from our system and my lead said that algorithms are categorically not architecture.
2. I came up with a set of events the system should be raising and in what order it should raise them, but this too didn't seem to cut it as architecture.

I seem to be getting caught up in the details and not stepping back far enough. I find that even when I come up with something that is at an architecture level, I often got there by trying out various implementations and mucking around in the details then generalizing and abstracting. When I described this to my lead, he said that I was taking the wrong approach: I needed to be thinking "top down" and not "bottom up".

---

Here are some more specific details about the project:

• The project we're architecting is a web application.
• I'm estimating around 10-100 thousand lines of code.
• We're a start up. Our engineering team is about 3-5 people.
• The closest thing I could compare our application to is a lightweight CMS. It has similar complexity and deals largely with component loading and unloading, layout management, and plug-in style modules.
• The application is ajax-y. The user downloads the client once then requests data as it needs it from the server.
• We will be using the MVC pattern.
• The application will have authentication.
• We aren't very concerned about old browser support (whew!), so we're looking to leverage the latest and greatest that is out there and will be coming out. (HTML5, CSS3, WebGL?, Media Source Extensions, and more!)

---

Here are some goals of the project:

• The application needs to scale. In the near term our users will be on the order of hundreds to thousands, but we're planning for tens of thousands to millions and beyond.
• We hope the application will be around forever. This isn't a temporary solution. (Actually we already have a temporary solution, and what we're architecting is the long-term replacement for what we have).
• The application should be secure as it may have contact with sensitive personal information.
• The application needs to be stable. (Ideally, it'd be stable around the level of gmail but it doesn't need to be at the extreme of a Mars rover.)

Answer 1

First of all I'd say that the difference between architecture and design is mainly semantics. Some teams have check points between the two. Your technical lead defines the architecture as before the design and architecture as implementation agnostic. From that I assume we are talking about design as in the waterfall model and not Industrial Design which would help you design the product with a view to users before you get to the software architecture. I think architecture often slips into design and that is not necessarily a bad thing, it is often very helpful for the architect to have a deep knowledge of what is possible within the system at hand.

Having said all that, you need some advice for the situation you are in. There is a whole world of software architecture out there, papers, books, conferences but you are generally looking for patterns and abstractions. Without more details on the project I can only give a broad example. For instance, if you were working in integration there is the Service Oriented Architecture (SOA) pattern where you split up parts of the system into 'services' so you can work with each part in a defined way, in web program this is often then implemented as Web Services (although shouldn't be though of as limited to that) and more recently the rise of RESTful APIs with JSON, again I would say this is a design coming from the architecture of SOA. I would say Model, View, Controller (MVC) is another example of an architecture pattern in common use, splitting up responsibility of the components of a system to allow for parts to be swapped out, to contain errors and testing.

From a 10,000ft level, if you can draw it on a whiteboard and explain it to a competent programmer who doesn't work in your field and doesn't know your programming language and current implementation details then it is probably architecture. If you can write a book about it that anyone outside of your company would care about then it is probably architecture. If you find your self explaining detail and can't generalise it to other code-bases / companies / industries then it is probably design.

I would agree that the two examples you give are code design and not architecture. The first because I think when you say you came up with an 'algorithm' for loading resources I think you mean you designed a set of instructions to accomplish that task, and not that you designed a new algorithm that they will be teaching in 1st year COMSC next year. In the second example, again I agree it is design. If you showed me either of these ideas I wouldn't be able to use them in my random software project. You have to go to a 'higher level', Object Oriented (OO) in Java rather than I want the Customer Class to be a sub-class of the Person Class. Even talking about Exceptions in general could be considered too low level (too close to the implementation).

To try to address the specifics that you list, I think what you should be thinking about is how to architect a web based CMS. Wordpress has a site architecture codex where they talk a lot about design implementation details but it is clear from the post that their main architecture centers around making Wordpress extensible with themes. Architecting a clear interface for a theme such that it could be written by someone out side of the company was clearly an architecture decision they took. These are the kinds of things it is good to get down on paper when architecting your 'long-term' (not temporary) solution so that all the design and implementation decisions that are made during development (by all the developers not just the architect) are in-line with this idea.

Other examples of architecture for your situation:

1. Putting the whole thing on virtual machines, hosted on a cloud provider or in house and having stateless machine instances, so that any machine failing can be replaced with a new instance of a virtual machine without having to copy across any state or losing any information.
2. Building in live production failure testing from the beginning with chaos simians.

Maybe try drawing the whole system on a whiteboard. Try it at different levels of detail, first board could be GUI->dispatcher->backend->DB or something, and then drill down until you start using pronouns.

Answer 2

The distinction between these two ideas is really important where I work.

What you call "architecture," we call "programming in English." This is partly important because if you can't describe it to a non-programmer, then something is wrong. It might be that you don't understand the problem well enough, OR it might be that you're solving a "phantom" problem (not discussed here).

The terms used for these two different aspects of design are often different, but the principles are readily recognized everywhere. One aspect (in your case the architect, in my case the designer) programs in English, whereas the other (in your case "designer," in my case "developer") programs in a particular language. They're also quite commonly distinguished as "design" and "implementation." "Design" being what it's supposed to accomplish, and "implementation" being the code that makes it happen.

Some examples from what I've worked on:

The architecture of one program is: we need a centralized Manager or hub that we can easily add modules to. This Manager would distribute events to all of the registered modules. A module can register itself with the Event Manager, and thereby publish events to the rest of the system, and receive events that it cares about. Each module has a "mailbox" that it can check and empty as it likes. This would let us accommodate new modules that we don't know we need yet.

No code there. Could be written in any language. Implementation isn't dictated by this description.

The architecture of another project is: we need a way to reliably start and stop other programs without waiting for them. We could have a manager that is responsible for a particular program. We can just tell it to start or stop its program and the manager takes care of it. If this other program is asked to stop and doesn't in a given amount of time, the manager knows how to force it to stop, and clean up the mess. The program isn't started or stopped by anything else, and the manager can be asked whether its program is running, stopped, or waiting to stop. This lets us carry on with the other things we need to do, while still getting these other programs started and stopped properly.

Again, nothing here dictates implementation, though some implementations are clearly more useful than others.

The difference between design (what we'd call patterns or implementation) and architecture (what we'd call design) is that one of them solves a coding/implementation problem and the other solves a real-world problem.

Answer 3

Maybe this will help. I've always seen the seniority of a engineer as a question of how big a problem they can solve on their own.

Roughly, to convey the idea:

• You can give someone new to programming small, contained tasks with lots of explicit instructions about how the task needs to integrate with other pieces

• A mid-level dev is someone who can take a description of some portion of an application, and make it work within the context of that application.

• A senior dev can build a medium-sized application from scratch, within the technical constraints of a shop.

• A more senior dev can do that, and make technology choices along the way about what technologies to use to make it work well.

...but those aren't hard and fast rules. And some people come out the gate as "senior" IMHO, even if they have to spend some time with a different title.

What the architect is asking is to view the problem even more generally than that. If you had to put together a number of applications to make the system work:

• What applications and services will you need?
• What pieces interact with customers, and which interact with each other?
• How will they communicate?
• Where will they store data?
• Where are the risks of failures?
• How will you provide reliability?
• How will you provide security?

So, in a sense technical architecture is like a building architecture. It's the layout, or the plan. It shows whats needed for the various parts, how they hold together, and just as importantly why.

(BTW, I've had a similar growth curve explained to me for architects, ranging from architecting a family of related applications or a set of very complex features, to setting technical direction for a group, to making strategic technical decisions for an entire organization.)

That said, I think most engineers at all levels have to do some "architecting" as well. It's not a bright line. It sounds like if you focus on the Big Picture first, and not get hung up on the implementation details, you'll be more in line with what he's looking for. BTW the ability to see the Big Picture as well as the Little Details is a huge asset in this industry, so this sounds like a great opportunity.

...Here's an analogy. Let's say you're asked to create a Magic Black Box. As an engineer, you're expected to obsess about the inner workings of the Magic Black Box. But as an architect, your focus is different. You might peek into the box out of curiosity, but you're expected to obsess about everything around all the Magic Black Boxes.

Similar to that analogy, you might think about the architecture role as viewing the whole system as the magic box. So if take a Gigantic Glass Box and you put the customers, the client apps, the firewall, the service tier, the database, and even the devops guys inside, then as architect you're to obsess about how to make that huge system box work well.

Answer 4

The exact difference between "design" and "architecture" is a bit subjective and there is some overlap. However, this is the difference as I understand it:

Architecture looks at high level concepts. Who are the actors in the system? What are the major objects and which ones are responsible for what? When I think architecture, I think Visio, not code.

For example, an event system might have an event manager that accepts incoming events and dispatches them to event handlers. The idea of threads, asynchronous v. synchronous, and other lower level concepts do not come into play here. MVC is another example: specific details are not specified at the high level of how the three pieces interact, only that there are three separate concerns handled by separate code packages.

Design involves prototyping, sketching out code interfaces, code skeletons, etc. It sits between the abstract architecture and the low level "code monkey" work.

In an event framework, the design might say "an event uses this interface," and "there is a thread pool which the event manager uses to dispatch events to workers." A design for MVC might be "use Hibernate for the model, Spring for the controller, and GWT for the view."

When I do design work I have sketch out interfaces and code skeletons, then hand off the results to programmers. Sometimes I am that programmer. But it is two separate phases, and both exist more toward the concrete than the architecture.

Putting on the architecture hat means clear your mind of the code and think objects on a whiteboard. Think objects, packages, frameworks, and the flow of messages between them. If you are thinking of even one line of code you are doing it wrong. Do not get bogged down in something like "oh, that message could be a string, or use SOAP, or whatever." At this level, the fact that communication is occurring is enough. Details are irrelevant.

Answer 5

If I can add anything here, it's that: don't think code. At all.

Don't think how you will write the code to accomplish something, but think about what the best method would be for accomplishing it. Your description of what you need to do should be language-agnostic, so you will be talking design patterns - which are a common "language" between users of different programming languages - to discuss how to proceed forward.

For your specific use-case, more architectural questions in my opinion would be along the lines of:

• Why are you using MVC? Is it all you know of? Are there better patterns to use? Why?
• What framework will you be using, and why?
• How will you scale? Not code-wise, because that doesn't matter yet. What will the conditions be to scale horizontally; what service (AWS) will you use to do this?
• How is the authentication going to be performed? Not code-wise: are you going to be generating a random, unique token on each request and having it checked against the expected token? Don't think how you will do this in code. Think about why you are doing this - because this can be done in practically any web language.

Basically, don't talk at all about code. Even if you don't know how to do something, when there is a will, there is a way. Think more about how the pieces of the puzzle will fit together best, before you worry about actually putting them together.

Answer 6

Think of all the operations (ie. use-cases) your system can perform. For each operation, document what happens in terms of your business domain for each operation. You should only talk in terms of your problem domain and not describe any implementation details. Draw a big block and call it System. The combination of this "big block" and the operation descriptions is the highest level system architecture.

While this high level architecture does provide a decent starting point, it really isn't of much value when building an actual system. You must take it down one level of detail to turn this into a useful architecture.

So you follow the same general idea as the "big block" approach only you begin adding "sub-blocks" that are necessary to accomplish each operation. These "sub-blocks" are frequently called domain classes or modules. These are easily identified by using your operation descriptions (from the big block approach) and drawing sequence diagrams. They are called domain classes because they aren't intended to be "real" classes, but they are intended to describe your system in terms of the problem domain of your system.

The end result of creating all the sequence diagram and identifying all the "sub-blocks" is that you now have a list of domain classes and their lists of operations. This usually ends up resulting in a fairly usable software architecture, where each of the "sub-blocks"/modules could be parceled out to different developers to design and implement.

Of course, if you leave it as is, then your designers will have a lot of interaction with each other in defining the interfaces between the modules. Thus, the architect may also decide on the specific interface mechanisms and data types to be used between modules.

Also, some "sub-blocks" will still be very complicated under-the-hood. So another iteration of the "sub-block" approach may be necessary but only this time on one of the newly identified modules.

Finally, there may be some specific patterns/limitations between interfaces that the architect wants the system to adhere to (e.g. event callbacks versus direct method calls), so those decisions would need to be talked about in the architectural design. Plus there may be "common" modules that the architect wants everyone to use.

Answer 7

As a developer, you are probably accustomed to providing solutions. That is a very good way of thinking, but may hinder you when it comes to thinking about architecture.

I find that it helps to describe the the problem that you are trying to solve first. What are the requirements? What are the constraints? Can you talk to stakeholders to find out these requirements?

It might help if you can describe your own design goals as well. Does your solution need to scale, or is a design for single user sufficient? How long does your solution need to remain valid? Is it a one-time fix, or a long term infrastructure solution? PErhaps as important as well: What are the accepted limits of your architecture?

And since this is a learning experience, do not be afraid to ask questions, even if they are silly.