I like William Pietri's answer a lot (+1), but I believe it needs to be added to. Even presuming that what you mean by systems comprises solely of software.
But before I go into the meat of it, I don't know of a book to help. All that which follows, I learnt from experience (meaning the three points that William made).
What you are talking about spans at a minimum four broad roles. Sometimes one person might fill in all those roles, for small to medium sized projects, but when you start on large projects you need to at least somewhat separate those roles out. It is difficult for anyone to be expert at them all in any meaningful way.
Business analyst
That's the person who talks to the customer and translates their requirements into something an architect can make sense of. Basically a list of properly formulated requirements. This includes the obvious functional requirements (what must this system deliver?), but also non-functional requirements (what are the general characteristics that the system must fulfil? This may include security, reliability, availability, resilience, capacity, performance, robustness and other such requirements from a user point of view).
This is the first pass at what the system must do, the very start of serious thinking.
System architect
This person produces the high level technical framework within which to work. They give the outline match plan. The general tools, techniques, constructs. They break down the whole system into smaller components, how they fit with one another, how they fit with the outside world...
This helps in many ways refine what needs to be thought about. Very often problems will be discovered at that stage about the requirements as written by the business analyst. Back to them for some iterations to improve their understanding of what they want and their expression of it.
System designer
This role is about how to make it all work. This might be more team work than a one-man show. But there is likely a lead designer to oversee the whole system design. This person must dig into the detail and make sure the architect's view is something that can actually be built.
Expect further refining of the architecture of the system, and therefore potentially of the business analysis.
Tests manager
This role is very often forgotten. But at the end of the day if you can't test it, how can you prove that you can build it? There must be a review of the results of all stages : business analysis, architecture and design by someone competent in testing who will be able to highlight deficiencies and therefore enable early corrections, way before any code gets written.
That's a short summary.
Those guys/gals are only the general run of the mill people in the chain to think about what must be thought about.
For complex projects such as large banking or space applications just to take two examples (think many hundreds to many thousands of man-days), there are many subject matter experts as we call them to review and support projects at every stage. These roles include security analysis, system sizing, capacity, performance, databases, clustering, and many other such narrow areas of expertise, including precise business areas. The variety of roles depends on the size and complexity of systems.
All that to say that you should not try and know it all, you won't. You can however get a grasp of the overall picture and on small projects you can delve into much more than on large projects, simply because the level of complexity allows you to be more rounded.
If you want to know how to design systems, then you need to start asking questions by thinking outside the box. Put yourself enough in the customer's shoes and try and think what could go wrong, what needs testing. Then get together with a real customer and push them to explain the extents and limits of the system they envision they need. Plus whenever I say 'customer', you must understand that this encompasses several very different people. There's the person who uses the system day in day out for what it was designed to do. There's the operator, technical support, the manager who needs some report or other, the auditor, the infrastructure team, the stakeholder who paid for it, the quality manager who needs means to test your system... Ask all of them (and if they are one person, ask them to put all these hats on one at a time), so ask them all what they need and you will have a good start at knowing what your system requirements are. From there you can derive the architecture, and from there the design.
For complex systems (whether software only or to integrate with hardware in the most generic sense) not only is one person not enough for each of the four roles I listed above, but you need to project-manage even the definition of what the system must do, let alone the other phases.
HPH,
asm.
