I am novice in "serious" programming i.e. applications that deal with real-life applications and software projects that go beyond school assignments.

My interests include optimization, operations research, algorithms and lately i discovered how much I do like software design/development/engineering.

I have already developed some simple desktop applications for some "famous" problems like TSP using heuristc approaches, a VRP solver (in progress) and so on.

While developing this kind of software I actually used basic concepts taught at school such as object-orientation analysis and design. But, I found these courses rather elementary and quite boring (for my expectations).

So I decided to go a little further and start developing "real" software (and this is where I realized how important and interesting software engineering/design is.)

Now, here's my issue: I can not find a "study guide" for developing software of this kind.

Currently, there are numerous resources out there (books, websites, tutorials) in designing and developing complex IS, web applications, smartphone apps but I can't find a book for example entitled "optimization software development". Definetly, someone could claim that "design patterns apply to software in general" but that's not my point.

My point is that I could simply use my imagination for "simple" implementations, but what happens, when my imagination can not go further?

In other words I'm looking for a guide/path to bridge the gap between: Mathematics-Algorithm Design-Software Engineering-Optimization-Software development

  • 1
    There are two rules of optimization: 1. Don't. 2. Don't do it yet (for experts only). Commented Dec 21, 2011 at 21:24
  • 1
    @Spencer Rathbun: Optimization isn't necessarily optimization of code. It can refer to optimizing anything, frequently using linear programming. Commented Dec 21, 2011 at 21:26
  • 1
    I am novice in "serious" programming Yeah, me too... After 15 years at it...
    – yannis
    Commented Dec 21, 2011 at 21:29
  • @SpencerRathbun You're referring to code optimization but that is not what I'm talking about. I am talking about software that is evaluated by its performance and solves real-life application using optimization techniques. I do know these rules (theoritically) and I have learned them (according to my little expereience).
    – Flo
    Commented Dec 21, 2011 at 21:30
  • @Florenc Is there a particular approach to optimization that you are more interested in than others, i.e., linear programming, statistical, etc.? Is there a particular problem area in which you are interested, i.e., scheduling, routing, etc.? Commented Dec 21, 2011 at 21:33

4 Answers 4


You seem to want to dive in to all aspects of software engineering, which is great, but the disadvantage is that there's no one place to get all these types of answers: ours is a young and rapidly evolving field. The correct answer for "how do I learn to design algorithms?" leads to one whole area of discussion, "how do I learn to write complex programs?" is another, "how do I learn to write fast programs?" a third, and "how do I find the best known ways to solve this type of problem?" is yet another.

What jumps out to me is that you have a passion for what are considered "hard" algorithmic problems. This is not a common career passion (for some reason) and you won't find as broad a range of resources as you will for more common focuses. On the other hand, for those with the combination of mathematical and practical interests and talents, it's a viable niche (although it's hard to break into -- you don't find jobs on Craigslist...). Most algorithmic resources in this area are going to be academic and technique-specific, while low-level performance programming is much more the world of obscure Internet fora and technical documents from the manufacturer.

On the specifics area of function optimization, it sounds like you have mostly worked with fully deterministic precise algorithms so far. These have all sorts of advantages, but are sometimes impractical in problems with very large solution spaces. You might enjoy exploring probabilistic approaches, such as Markov chains, simulated annealing, genetic algorithms and evolutionary programming, ant algorithms, etc.

Additionally, you should at least be aware of the field of approximation algorithms, which forego precision for higher performance (for instance, precise bin packing is NP, but can be solved in polynomial time with an arbitrarily low acceptable error).

I know I haven't given you any direct links, but hopefully I've thrown out enough subjects to fill some time with Google!

  • Your approach is very accurate and completely correct. Except from one thing: I have already studied metaheuristics, at least the basic concepts/frameworks (tabu search, simulated annealing, ACO... ). Actually I enjoy working with heuristic approach rather than exact methods.
    – Flo
    Commented Dec 22, 2011 at 18:09
  • You're example about the bin packing problem is very helpful because it's exactly the kind of problem I am talking about. Routing problems for instance, are quite easy because you can simply illustrate a route using a Linked List or Even an Array. But how could you "code" the packing of some objects (of different sizes) to many bins. Could I simply create a class Bin and some other classes like this? Or something else?... These are the kind of design questions I'd like to answer.
    – Flo
    Commented Dec 22, 2011 at 18:09
  • Yes, probably what I'd expect in a bin-pack solver would be an interface for Bin that has a capacity and various summary functions (total profit, free space, etc.) and an interface for Packable that had cost (space consumed) and profit. Then you solve for MAX(bin.totalProfit). Commented Dec 22, 2011 at 18:44

I believe part of the problem you are experiencing is that such software developed for businesses, rather than software developed for mass market (aka "shrink-wrapped" or consumer) are developed and sold as solutions rather than as applications, for the most part.

Some of these businesses have a base product, and sell the package with customization consulting, which is essentially developing an formal encoding of the "business rules" or "business logic" for that client's business. This is necessary as the client actually wants a solution to their problem, not a complex system that can manipulate symbols and numbers. The companies train and foster their own in-house (e.g. IBM) and external consultants (SAIC) who specialize in supporting their product and its customization for clients.

I suspect the majority of this knowledge falls into either the academic side of fence (SIAM may be your best source of journals and books) or in-house corporate knowledge that is not widely shared, as the particular markets may be perceived as small (few competitors).

One place where you may find a community of people willing to share their knowledge is academic supercomputer centres, which may even provide a workplace to transition from your seemingly academic background to a more development oriented experience and role.

I don't know for sure, it is totally outside my sphere of knowledge/ experience, but one potential area to consider might be the field of data mining, where the data sets are so large that more advanced approaches to extracting useful information may tap Optimization and OR approach.

Military and domestic intelligence organizations (e.g. NSA, CGHQ, CSE) may be places of application and/or employment if you are interested.

I don't know if a department like U Waterloo's C&O department may have additional suggestions. Otherwise what is generally called as "scientific computing" may be the closest to what you are looking for.

Such focus is outside my own background, but I hope that helps give you some places to consider or look into.

  • Great points. Thank you! As for "data mining", I agree. Today there is a trend to combine OR techniques with "data mining" (to be precise: business intelligence-analytics )and I find this approach very interesting and promising too.
    – Flo
    Commented Dec 22, 2011 at 23:52
  • What might be a radical change in direction, but may be of interest is Quantitative Analysis, an intense financial field of advanced applied mathematics.
    – mctylr
    Commented Dec 23, 2011 at 18:05

The free Machine Learning online class from Stanford University may be exactly what you are looking for. It includes programming assignments as homework and covers:

This course provides a broad introduction to machine learning, datamining, and statistical pattern recognition. Topics include: (i) Supervised learning (parametric/non-parametric algorithms, support vector machines, kernels, neural networks). (ii) Unsupervised learning (clustering, dimensionality reduction, recommender systems, deep learning). (iii) Best practices in machine learning (bias/variance theory; innovation process in machine learning and AI). The course will also draw from numerous case studies and applications, so that you'll also learn how to apply learning algorithms to building smart robots (perception, control), text understanding (web search, anti-spam), computer vision, medical informatics, audio, database mining, and other areas.

  • Thank you! Very helpful, but I have to wait a few days. There is also this ai-class.com
    – Flo
    Commented Dec 21, 2011 at 21:46

If you can get an algorithim in code to O(1) then it is fully optimized and you have reached the holy grail of algorithmic programming, however, usually the time and effort it takes to get to that level of optimization is not worth it as today's computers are fast enough to handle just about anything. But there are times that one would need to optimize the algorithum to make it more efficient. So understanding the complexity of the code and how it executes is a good start.

Here's a good article to get you started:


  • Thank you but I do have a basic understanding in Algorithms Analysis, Data Structures
    – Flo
    Commented Dec 21, 2011 at 21:52
  • 1
    O(1) isn't necessarily the "holy grail" of programming. An O(1) solution to something could take 3 weeks to process, while some O(n^5) process could take just a moment for the range of n expected in your problem. The usefulness O(1) vs O(n^whatever) depends on the application. Commented Dec 22, 2011 at 3:02
  • @Florenc: If you have a basic understanding of algorithms, I'm supposing you are fresh from a class where big-O was glorified, and if constant factors were mentioned at all, it was something like "Oh, just go learn gprof." That is baloney. Here's a slide show where you can see how to get rid of constant factors, and they can be huge. Commented Jan 20, 2012 at 20:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.