What causes bad performance in consumer apps? [closed]

Question

My Comcast DVR takes at least three seconds to respond to every remote control keypress, making the simple task of watching television into a frustrating button-mashing experience. My iPhone takes at least fifteen seconds to display text messages and crashes ¼ of the times I try to bring up the iPod app; simply receiving and reading an email often takes well over a minute. Even the navcom in my car has mushy and unresponsive controls, often swallowing successive inputs if I make them less than a few seconds apart.

These are all fixed-hardware end-consumer appliances for which usability should be paramount, and yet they all fail at basic responsiveness and latency. Their software is just too slow.

What's behind this? Is it a technical problem, or a social one? Who or what is responsible?

Is it because these were all written in managed, garbage-collected languages rather than native code? Is it the individual programmers who wrote the software for these devices? In all of these cases the app developers knew exactly what hardware platform they were targeting and what its capabilities were; did they not take it into account? Is it the guy who goes around repeating "optimization is the root of all evil," did he lead them astray? Was it a mentality of "oh it's just an additional 100ms" each time until all those milliseconds add up to minutes? Is it my fault, for having bought these products in the first place?

This is a subjective question, with no single answer, but I'm often frustrated to see so many answers here saying "oh, don't worry about code speed, performance doesn't matter" when clearly at some point it does matter for the end-user who gets stuck with a slow, unresponsive, awful experience.

So, at what point did things go wrong for these products? What can we as programmers do to avoid inflicting this pain on our own customers?

Answer 1

Good question. What I see daily is this.

People work on good-size apps. As they work, performance problems creep in, just like bugs. The difference is - bugs are "bad" - they cry out "find me, and fix me". Performance problems just sit there and get worse. Programmers often think "Well, my code wouldn't have a performance problem. Rather, management needs to buy me a newer/bigger/faster machine."

The fact is, if developers periodically just hunt for performance problems (which is actually very easy) they could simply clean them out.

Instead, the "state of the art" is:

1. rely on aphorisms like "eschew premature optimization" and 90/10 hoo-haw.
2. talk bravely about profiling, and sometimes actually try it, often with disappointing results, as you see in all the SO questions about it.
3. fall back on good old guesswork, disguised as experience and knowledge.

But really, that's negative. To be positive, this method works nearly all the time, and it couldn't be simpler. Here's a detailed example.

Answer 2

This isn't a technical problem, it's a marketing and management problem.

You may be rolling your eyes at this point, but please bear with me.

What a company sells is their "product", and the folks who define what that is are "product managers". In tech firms, a lot of other people weigh in on that - user experience experts, yadda yadda. But ultimately, the product mangers are responsible to write the specs for what the user is supposed to get.

So, let's take your Comcast DVR. Ideally, things would work like this:

1. The product manager writes in a spec, "When a user presses a button on the remote control, and is within 25 feet of the DVR, the DVR must respond to the press within 250 milliseconds".
2. The technical folks build the hardware, write the embedded software, etc.
3. The QA testers either approve that the system meets the spec, or bounce it back to the technical team for a fix.

Of course, lots of things can go wrong:

• The product manager fails to put button response in the spec
• The QA folks do a mediocre job of testing against the spec
• Somebody selected technologies that don't permit the spec to be met, so the requirement gets punted
• The technical staff is short-handed, or somebody accelerated the schedule, and some manager says, "Forget about responsiveness - get this other feature finished."
• The product manager doesn't publish the responsiveness requirement until so late in the game, it can't be met by the ship date
• Management decides not to submit anything for QA testing until so late, accelerating slow code would destabilize the product

Did you see all the feckless programmers in there? There weren't any.

I'm not saying we bear no responsibility at all for bad performance - often, it's just as easy and fast to write good, robust, efficient code as it is to write junk.

But really, if the product management and QA staff are all asleep at the wheel, we programmers can't make up for that.

---

FWIW, I completely agree about the abysmal interfaces of most consumer products. I've been writing UI code now for about 25 years, and I strive for elegance and simplicity. It's actually a problem because I think about it so much, I'm now lousy at figuring out badly-designed user interfaces, so my poor wife winds up running most of the devices in our media center.

Answer 3

Because programmers are not perfect.

I am a programmer of embedded things. Some of my code is not perfect. Most of my embedded code is fast.

To fix performance problems at the end of a project is very hard.

Sometimes, to keep things simple ( and therefore testable, develop-able in realistic time, not fatally buggy) we layer things, like the remote input to a "service" that isn't part of the main application. End result, latency. The alternative is to put everything in a monolithic application is a buggy disaster in C or C++ (the two most common embedded languages.)

Sometimes your embedded device has a process scheduler that doesn't do what you as a user wants. Damn hard to fix as an embedded developer.

Complexity causes the lagging, because of latency on layering. You asked for the features. Try a really old Nokia, like the old 3210. Zippy fast UI. Not many features.

I'm arguing that developers don't get any smarter, so faster hardware gets absorbed on abstractions to prevent features killing each other. (Or not, in the case of your iPhone)

UI performance needs to be a requirement that you test to as the design progresses.

If it isn't specified, the developer will get used to it. We all do this. "My baby is not ugly"

And it's not the GC languages; embedded Realtime Java is so quick it's scary. (Embedded Python on the other hand is a total dog)

I write a program the reads switches on digital inputs as the UI. Still have to de-bounce the switch, so really fast flicking the switch doesn't work, because the de-bounce is a couple of layers up. Ideally I'd have de-bounce logic at the bottom of the firmware stack, but that's not how the hardware works.

Some DVD players just run a "eject" script to do eject. You DVR may have taken this approach to keep development costs sane. Then you skimp on CPU or RAM and it sucks.

Answer 4

Is it because these were all written in managed, garbage-collected languages rather than native code?

No. Slow code will perform poorly regardless. Sure, a particular language may introduce certain classes of problems while solving others. But good programmers are quite capable of finding workarounds given enough time.

Is it the individual programmers who wrote the software for these devices?

Partly. In many cases it is quite likely at least a contributing factor. This is an unfortunate side-effect of an industry where good programmers are in high demand and short supply. Also the gulfs between various levels of technical ability can be quite large. So it stands to reason that sometimes the programmers tasked to implement certain software could be congratulated just for getting it to work (sort of).

In all of these cases the app developers knew exactly what hardware platform they were targeting and what its capabilities were; did they not take it into account?

Partly. For a start, the exact hardware platform is probably not known, as that is often negotiated with various manufacturers in parallel during software development. In fact, there can even be small (but not necessarily insignificant) changes to underlying hardware after initial release. However, I would agree that the general capabilities will be known.

Part of the problem is that software probably isn't developed on the hardware, it's done in emulators. This makes it difficult to account for true device performance even if the emulators are 100% accurate - which they aren't.

Of course this doesn't really justify insufficient testing on the appropriate prototype hardware before release. That blame probably lies outside of dev/qa control.

Is it the guy who goes around repeating "optimization is the root of all evil," did he lead them astray?

No. I'm pretty certain they don't listen to him anyway; otherwise he wouldn't be misquoted so often (that's supposed to be "premature optimisation ..."). :-D

It's more likely that too many programmers take one of 2 extremes with regards optimisation.

• Either they either ignore it completely.
• Or they obsess themselves with minutiae that has nothing to do with the actual performance requirements. The net effect being that budget runs out and the code is too obfuscated to optimise the real performance problems safely.

Was it a mentality of "oh it's just an additional 100ms" each time until all those milliseconds add up to minutes?

Possibly. Obviously if Sleep(100) has been used as the equivalent of tissue paper used to slow the bleeding of a severed limb - then problems are to be expected. However, I suspect the problem is more subtle than that.

The thing is modern computing hardware (including embedded devices) is much faster than people give them credit for. Most people, even "experienced" programmers fail to appreciate just how fast computers are. 100ms is a long time - a very long time. And as it so happens, this "very long time" cuts 2 ways:

• The first is that programmers worry unnecessarily about the things that a computer does extremely quickly. (It so happens that it was just such a concern "incrementing a value 300 times per second" that led me here in the first place.)
• The second is that they sometimes fail to show due concern when things do take a very long time (on the computing timescale). So:
  • if they ignore the effects of latency when communicating over a network or with a storage device;
  • if they ignore the impact of a thread blocked and waiting for another thread;
  • if they forget that because computers work so quickly it is very capable of repeating a task far more often than it should, without the developer being aware of a problem
  • ... if any combination of such oversights occur, a routine will unexpectedly run very slowly (on the computing timescale). A few repeats and it will even be noticeable by humans - but may be tricky to pin down because hundreds of interconnected things are all running quickly by themselves.

Is it my fault, for having bought these products in the first place?

Yes definitely. Well, not you personally but consumers in general. Products are sold (and bought) by feature checklists. Too few consumers are demanding better performance.

To illustrate my point: The last time I wanted to buy a cell-phone, the store couldn't even offer a demo model to play with in-store. All they had were plastic shells with stickers to show what the screen would look like. You can't even get a feel for the weight like that - let alone performance or usability. My point is that if enough people objected to that business model, and voted with their wallets to voice their objection, we would be one small step in the right direction.

But they don't, so we aren't; and every year new cell-phones run slower on faster hardware.

(The questions not asked.)

• Are marketing people to blame? Partly. They need release dates. And when said date looms, the choice between "get it working" and "make it faster" is a no-brainer.
• Are sales people to blame? Partly. They want more features in the checklist. They hype up feature lists and ignore performance. They (sometimes) make unrealistic promises.
• Are managers to blame? Partly. Inexperienced managers might make many mistakes, but even very experienced managers may (quite rightly) sacrifice time to resolve performance issues in favour of others concerns.
• Are specifications to blame? Partly. If something is left out of specification, it's that much easier to "forget" about it later. And if it's not specifically stated, what's the target? (Although I do personally believe that if a team takes pride in its work, they would worry about performance regardless.)
• Is education to blame? Maybe. Education will probably always be on the back-foot. I certainly disapprove of "education" that rapidly churns out beginners with a superficial understanding software development. However, education that is backed up with theory, and instills a culture of learning can't be bad.
• Are upgrades to blame? Partly. New software, old hardware really is tempting fate. Even before version X is released, X + 1 is in planning. The new software is compatible, but is the old hardware fast enough? Was it tested? A particular performance fix may be rolled into the new software - encouraging an ill-advised software upgrade.

Basically, I believe there are many contributing factors. So, unfortunately there's no silver bullet to fix it. But that doesn't mean it's doom and gloom. There are ways to contribute to improving things.

So, at what point did things go wrong for these products?

IMHO we can't really identify any single point. There are many contributing factors that evolved over time.

• Bean counters: cost cutting, market timing. But then again would we have made the advances we have achieved without the pressure?
• High demand and low supply of skilled people in the industry. Not just programmers, but also managers, testers, and even sales-people. Lack of skills & experience leads to mistakes. But then again it also leads to learning.
• Bleeding-edge technology. Until a technology matures, it will regularly bite in unexpected ways. But then again it often provided a number of advantages in the first place.
• Compounded complication. Over time, the industry has evolved: adding more tools, technologies, layers, techniques, abstractions, hardware, languages, variation, options. This makes it somewhat impossible to have a "full" understanding of modern systems. However, we are also capable of doing a lot more in a far shorter time as a result.

What can we as programmers do to avoid inflicting this pain on our own customers?

I have a few suggestions (both technical and non-technical) which may help:

• In sofar as it's possible - use your own product. There's nothing like first hand experience to reveal things that are awkward, slow or inconvenient. However you will need to consciously avoid bypassing deficiencies due to "insider knowledge". E.g. If you have no problems synching contacts because you do it with a backdoor Python script - you're not using "the product". Which brings up the next point...
• Listen to your users (preferably first hand, but at least second hand via support). I know programmers (generally) prefer to stay hidden away and avoid human interaction; but that doesn't help you discover the problems other people experience when using your product. E.g. You might not notice that the menu options are slow, because you know all the shortcuts and use those exclusively. Even if the manual fully documents all shortcuts, some people will still prefer the menus - despite being insufferably slow.
• Strive to improve your technique skills and knowledge on a continuous basis. Develop the skill to critically analyse everything you learn. Reassess your knowledge regularly. In some cases, be prepared to forget what you thought you knew. Which brings up...
• Some technologies / techniques can be very tricky leading to subtle misunderstandings and incorrect implementations. Others through the evolution of common knowledge or available tools may fall in or out of favour (e.g. Singletons). Some topics are so tricky that they breed a bunch of "hocus-pocus pundits" that propagate a huge body of misinformation. A particular bugbear of mine is the misinformation surrounding multi-threading. A good multi-threaded implementation can significantly improve user experience. Unfortunately a lot of misinformed approaches to multi-threading will significantly reduce performance, increase erratic bugs, increase dead-lock risks, complicate debugging etc. So remember: just because an "expert" said it, doesn't make it true.
• Take ownership. (No seriously, I'm not playing boardroom bingo.) Negotiate with managers, product owners, sales people for performance features taking precedence over some checklist items. Demand better specifications. Not childishly, but by asking questions that get people thinking about performance.
• Be a discerning consumer. Pick the phone that has less features but is faster. (Not faster CPU, faster UI.) Then brag about it! The more consumers start demanding performance, the more bean counters will start budgeting for it.

Answer 5

Your first mistake, and probably why you have got a down vote it deserves the blatantly obvious exaggeration. Do you really expect to believe the iPhone and iPad are that bad.

Ultimately the customer is responsible. It comes down to cost and what the customer is prepared to pay and what they get in return. If they choose features over speed, that's what they get. If they choose price over speed, that's what gets built and sold. If brand image is more important..... Ultimately the customer decides with their wallet, whats important and whats not. You have a choice to be a brand whore and buy products because everyone else does, or be an independent thinker, look behind the gloss and marketing hype, and buy something that meets your needs.

You are blaming the programmers. They wrote the code, sure, but they did not define, and should not define, the customers requirements, the hardware, the BOM cost, the R&D cost, the marketing budget..... all the things that go to make a product, that is not software.

The technologies used, languages used etc, are nothing to do with this. Bad vs good developers, nothing to do with it. Any half decent programmer can make a piece of code run faster, be more responsive, be the ultimate it could. My experience is decent programmers don't bankrupt the business when left to make the decisions, while half decent ones complain how much "better" it "should" be.

Answer 6

Premature optimization is sometimes bad, but not when required for good user experience or good battery life in a sufficiently constrained system. The failure is the fault of giving a higher priority to clean maintainable software engineering over cooking in whatever it takes to provide good user experience and decent battery life as a higher priority at the beginning of a project, even if it's much harder to maintain and short circuits some cleanly architected software stack and methodology.

If you have an iPhone 3G, Apple released a couple OS updates that were only optimized for newer devices. Later OS updates for the 3G may provide slightly better performance on the 3G.

Answer 7

Your DVR takes that long to change channels because it has to first dump the buffered data, and then queue up another buffer full of data for the new channel you are watching. This buffer is most likely stored on the hard drive so these operations take time (plus it can only fill the buffer in real time). With a DVR you are never watching "live" programming, it is always delayed (not coincidentally, it is delayed by the same time as your perceived delay when switching channels). This can easily be verified by watching a sports program at the same time that you listen to it on a radio.

Answer 8

I think the reason is that most consumer directed apps are controlled and marketed by people who don't know anything about software, and hire developers based on their resumes or the recommendations of some no-nothing manager, as opposed to their actual skills and knowledge.