How can dev teams prevent slow performance in consumer apps?

Question

When I previously asked what's responsible for slow software, a few answers I've received suggested it was a social and management problem:

This isn't a technical problem, it's a marketing and management problem.... Utimately, the product mangers are responsible to write the specs for what the user is supposed to get. Lots of things can go wrong: The product manager fails to put button response in the spec ... The QA folks do a mediocre job of testing against the spec ... if the product management and QA staff are all asleep at the wheel, we programmers can't make up for that. —Bob Murphy

People work on good-size apps. As they work, performance problems creep in, just like bugs. The difference is - bugs are "bad" - they cry out "find me, and fix me". Performance problems just sit there and get worse. Programmers often think "Well, my code wouldn't have a performance problem. Rather, management needs to buy me a newer/bigger/faster machine." The fact is, if developers periodically just hunt for performance problems (which is actually very easy) they could simply clean them out. —Mike Dunlavey

So, if this is a social problem, what social mechanisms can an organization put into place to avoid shipping slow software to its customers?

Answer 1

With correctly written and complete requirements, there is no such a thing as a distinction between bugs and poor performance. Because you specify the performance as a non-functional requirement, poor performance becomes a bug just like any other bug, and will be caught by QA and solved by developers before release.

Is there a social problem? I don't think so. The major issue is that requirements are incomplete. Working for years as freelancer, I never ever saw a non-functional requirement telling that a specific task must perform in maximum N seconds on average. If the manager/customer/stakeholder or whatsoever doesn't bother about performance asset, why I, as a developer, would bother about it, since people who must care about it don't care at all?

There is another factor which influences poor performance: the fact that developers work on expensive PCs which perform well. When you're working for years on a quad-core PC with 8 GB of RAM, a high end SSD, the latest OS, etc., it's very difficult to imagine how your application will run on Windows XP on a dual-core PC with 512 Mo of RAM and an old hard disk filled at 90% and not defragmented for years. Unfortunately, in some countries, the last case is the one we see for most consumers of an app. The larger the gap between developer PCs and consumer PCs, the more complicated it is for a developer to take care of performance of his app.

Answer 2

The problem(?):

• The customer (or end-user) does not complain about it (enough)
• Thus the project(/product) manager does not consider it a requirement
• Thus the developer does not get the time to fix it.

You have to start at the beginning, educate the customers. But if they buy the iPhone instead of a faster, less shiny phone, the developers are right to spend their time on looks instead of performance. The organization is not the problem.

Of course, some things can help anyway. Waiting for automated tests is annoying, so if you have automated tests the developers have constant feedback about performance issues, and they will be more likely to solve it (as a technical issue, not as a feature).

But you can't do everything. It's optimize or add features, and those who spend the money decide.

But some good news: I've noticed that SaaS/Cloud/Buzzword-applications help a lot here. When people choose between a few similar web-applications and get to test live instead of first creating artificial lists of 'required' features, they are more quickly influenced by responsiveness, and thus performance will get more attention.

Answer 3

Sadly, I find the biggest issue is you can't do everything. You have a ship date, and you know it's slow, but you NEED to get features X,Y,Z out to market.

In your mind, slow you can fix later, but the app at least works.

So, you worry about functionality and aesthetics (because users focus on aesthetics all to often). Next release you'll fix performance.

But the PM just gives you a list of Features, and no time to fix the performance.

And the vicious circle continues.

Answer 4

I agree with others, that we should find ways to make developers care more about the problem, like making them test on slower hardware, and having performance goals. That's all fine, but really, when it comes to performance tuning --

People Gotta Know How - And They Don't

They may think they do, but just look through all the performance-related questions and answers on StackOverFlow and on this forum. It's painful how many show very little common sense about performance.

It's not something to just talk about, people need to learn by doing it. The only time they are in that mode is when they are taking a class, or learning new things from a book or blog.

So the only way I can think of to solve this problem is to get hold of the people who teach programming, and teach them how to do it.

Heaven knows, I've tried on these forums, as in -

• The basic super-simple method
• A 43x speedup - do that with your profiler
• The reasons why, in nauseating detail

Anyone can do it. They just need to actually do it.

Answer 5

Make performance a requirement.

Answer 6

Writing performant code is hard. It requires a solid grasp of concepts like threading, asynchronous event handling, caching, and asymptotic complexity. Judging by the groups of programmers I've worked with, around 20-40% of any given group doesn't understand those concepts well enough to incorporate performance considerations as a matter of course into their daily work.

However, those programmers are obviously still useful to the company, but they get assigned to tasks not considered performance critical, so you end up with a blu ray player that can play Netflix streams flawlessly without dropping any frames, but it takes 30-60 seconds to open the menu item that displays your queue.

Unless you're some hotshot software company that can afford to fire 20% of your staff and replace them with more experienced (and more expensive) developers, the only real way to fix it is developer training and filing bug reports. I don't know how it is at other companies, but here if we developers see a performance issue that we don't have time or business priority to fix, we're fully entitled to file our own bug report on it. It might take a couple of releases to work its way to the top of the backlog, but they usually do get addressed eventually.

Answer 7

If performance is a requirement, test for it.

Otherwise Wally can write an infinite loop and leave early "It takes a while." He can claim.

Your software acceptance test should have a detailed acceptance test for various operation performance characteristics.

If you don't do this you're not engineering any performance into the product.

The performance ( like the resource consumption ) should get budgeted out to sub-systems. Then the sub-system acceptance tests can check them.

Then you can test early and often for performance. Even unit tests can then check it.

So now developers have it as an acceptance criterion, and can organise their approach to suit it.

Where I'm working now, the performance stress test is 2x bigger than any customer data set we know of. It regularly breaks a new version of the product. Good testing.

Answer 8

I remember once in the mid-90s and I was spending some time trying to optimize something and a coworker told me, "This is running on pentiums, who cares?" .... that was an eye opener. Sadly, it was just the tip of the iceberg, I've heard that attitude throughout my career - albeit the "pentium" part has changed over time.

The only way to get the average developer to care is to get lack of performance to be viewed as a bug on the part of the customer. Depending on the application and audience this can either be an easy or a hard task (I've seen both). If the audience doesn't care about the poor performance, the developers never will (quickly, good, fast - choose two).

Answer 9

But it shouldn't take a letter from QA for a programmer to realize that a 3 second lag between keypress and response is unacceptable

Agree it shouldn't. It should take more than that: a proof that obtained lag is relevant for end users.

Given that you provided no context, it looks entirely possible that lag in dev/QA environment is caused by their local issues of slow disk / memory / network access. If that's the case, your QA and dev will be simply wasting their efforts fixing things that just don't matter to end users.

Relying on developers in performance testing is about as productive as rolling a dice to pick a piece of functionality to speed up. Oh and it's about as reliable as that - "developers generally have horrible intuition about where the performance problems in an application will actually be" (Brian Goetz).

• I've been in a project where lame management once decided their bright marketing guys and smart programmers are good enough to handle performance concerns of the customers. What a great lesson it was. Rejected release, half year of efforts gone to trash bin, and company almost lost a strategic partner. In the end, they invited professionals (experts in benchmarking, statistics, UX, low level optimization, stuff like that) and professionals fixed that mess.

---

should all programmers do their own QA so they see such issues immediately?

Fact that it's doable doesn't mean it's the way to go. Rather opposite - in my experience this was one of the most reliable ways to lose on programmers productivity. Almost as good as endless meetings and even better than interviewing candidates.

• As an ex-tester I once thought it shouldn't be a problem to combine development and QA activities. It looked like the difference between routine dev testing and systematic QA won't matter much. I thought that dev/QA separation is merely a tradition in software industry. Learned rather hard way that this ain't so. Separation turned out to be a matter of focus and productivity, and quite a serious one.

If there's a performance issue, just give me a benchmark and set target performance and I'll do my best to hit it. I'm not that good in performance testing but know a bit or two about optimization.

Answer 10

I think you would find that 99% of the time the problem is the scope creep. With the dvr for example. You would think this is easy but then TIVO or a competitor introduces a new feature that is well received. Next thing you know a new feature is on the plate. It may or may not becompatible with the existing product and we are not redoing the existing product that will take to long. So the feature gets jammed in and it sucks performance. Sure the data is there to get the infromation but if there was not thought to getting that information then there is a good chance it will not be easy to get. So now it has a complex process to build that information everytime you go near the program list.

Answer 11

Ignoring the developers that don't seem to care...

I think that often times the developers that work on the code don't have the tools to measure performance on a continuous basis.

e.g. If it is possible to measure the response time for your app (e.g. its a web based application, or queries a database, etc.) - Are you currently getting notifications (email, SMS, whatever) that indicate the "top 10" worst performing (or over a determined threshold) responses?

In many, many cases - developers are not getting this info from the "real-world" deployments and as a result its very easy to ignore the information that you don't see.

If however, every day/few hours you get an email that indicates that screen "x" takes 13 seconds to load and it is running the following SQL query SELECT TOP 1.... JOIN... OUTER JOIN... OUTER JOIN... CROSS JOIN... you'd better believe that a developer could (and hopefully would) be all over fixing it.

Thus although I'd like to believe that all programmers do take performance seriously I think that lack of visibility to the issue(s) is often the culprit.

Note: I think this is something that both the developers should be requesting access to (or even developing such a feature) and management should be providing/funding such tools.

Answer 12

Can you come up with better examples where we can actually pin the blame on programmers? Apart from Eclipse, and one commenter has already pointed out that it's plugins that do it(my first install of each new Eclipse version runs like lightening, but when I add the other tools it starts to slow), your examples may not be programmer and code related but environment related.

The days of running a program on a computer in isolation and determining if it is 'fast' or 'slow' are gone. The other examples you give depend on their environment - the current network congestion, whether the back end servers are overloaded, badly configured network cards, a faulty cable, the number of other people using it in your vicinity, or hundreds of other variables. eg. our hosting provider charged extra for server gigabit connections but eventually we determined it all went through an ancient firewall device with 10Mb ports. These problems shift around and are hard to find.

Agreed, there are lots of things programmers can do (minimizing bandwidth, UI tricks that improve responsiveness and show progress to give the impression it is fast). But when you roll out to the real world there are all sorts of circumstances that you only learn by experience (and you watch your assumptions fall apart in front of you).

Answer 13

How much are you willing to pay for better software? How much will the market wait for better software? How little cruft will want adding to the next release?

It is a cut-throat market out there where many compromises are made. If it is truly crap then the market will (or should) fail the product. Maybe there are enough customers who can live with the status-quo?

Answer 14

I think the most general answer to this problem is also the most difficult to manage, which is that every programmer should be mindful of performance with anything they do. I realise also that's a bit of a cop out.

Depending on the size of the project, and corresponding team, I believe there can be a lot of value in having dedicated performance programmers.

As an example, I have worked on a team where the project team (including about 30 devs) had at least 2 people dedicated to performance optimisation. This particular app was also quite prone to performance issues as there were multitudes of interoperability components, not only over web services but also in terms of legacy layers with various data mapping and adapter components.

Answer 15

First hand experience is important. Give the developers a fast computer to compile and build on, but a really slow overloaded computer (as some percentage of users may actually have) to run their app(s) on. (This can be done in cloud/server based development environments by partitioning VMs or servers by function.) Give them a cell phone with a half dead battery, and require them to use only it for the days initial mobile app testing.

If the developers empathize with the customer with respect to performance and battery life, then they won't regard performance as some semi-bogus management spec to put at the bottom of the priority list. (As in: "hey, I thought it was premature optimization" until too late.)

Answer 16

Stop buying the stuff, and, comment on the poor performance on any online reviews you come across.

If the devices are still selling then the software is "good enough" not to invest more time and money in. If bad reviews about performance reduce sales significantly then the software is "not good enough" and needs fixing.

The only metrics that interests a producer of consumer goods are sales and profit per unit.

Answer 17

I agree that programmers should be taught better on maximizing performance, etc.

But I think the solution is not giving programmers an almost-dying hardware for daily use. How stressful it will be if your visual studio crashes twice a day, it took x seconds to build the stuff, y seconds to open the solution, z seconds to change windows. I don't think it was very cost-efficient for the company either since hardware is cheap and programmers time is expensive.

Since the next hand that will be handling the code is QA (testing) team, wouldnt it be better to teach them about the importance of performance, have a standard of whats acceptable performance standard, etc as an enterprise standard (well, in perfect world, the performance standard should be in spec but that doesnt happen very often)? (i.e regular enterprise ee-changing page/tab should happen instantly, "save update should happen in x second", if it's a critical app then...). The computer that the QA teams run in should be the typical user computer. (we dont wanna piss them off by giving them a 386, but dont give them a quad core with 8GB of ram for example). Wouldnt they vent to the programmers if they get pissed off enough with the performance?

I think client/project manager should force programmer/qa team/developer/rep of company team to do their presentation on the lowest typical hardware that they have. (the weakest computer in the company for example). If it's rewrite, they will need to gather data on how fast it is to do x process in the old software and compare it with how fast it is on the new software (it could be slower, since new software might involve extra business process but there should be an acceptable window).

Answer 18

I've said it before, but I'll say it again: I think one of the best ways to handle this is to simply have your developers work on (relatively) trailing edge hardware.

For your typical programmer, most official performance considerations are secondary to simply: "is it annoying when I try to run it?" If they find it annoying, they'll (at least try to) fix it. If they're happy with the way it runs, the very best you'll get is a half-hearted attempt at fixing. This also helps in finding problems early, before they become much more expensive to fix.

If it gets to the point that QA has to enforce rules the developers really don't believe in because they think performance is adequate, chances are pretty good that most of the "fixes" you get will get creative ways of getting around the rules, not real fixes that improve life for the end user.

At the same time, I should add that I've rarely seen this as a problem. If anything, I've seen developers spending far too much time trying to improve performance where I really didn't matter enough to bother (a sin of which I'm often guilty as well).