How do you manage the testing of your Android software on physical devices?

Question

I'm in charge of managing mobile application development at my company, and I am currently building a mobile device "library" for testing. Essentially, we want to have a representative device in-house for each of the OSes we are developing for, currently iOS (iPhone-only), Blackberry, and Android. Simulators only go so far, but I'm placing into the process a step to test software on the devices themselves.

The problem we're finding is with Android. I don't think any of us here ever really understood just how fragmented the whole platform is until we started looking at devices to acquire. We are going to wait until v2.3 of Android is released, but which products to choose? Do we go by the most popular by market share? Do we get a small range of products by specs from least to most powerful overall?

We're trying to avoid having to manage a dozen different devices to test each app, if not because of cost if only for the repeated time sink.

How do you manage the testing of your Android software on physical devices?

UPDATE: To bring some context to what I am asking, the applications being created are for educational purposes only, and there are no plans to use any of the hardware features of the devices themselves. We are simply displaying content in a variety of ways: eBooks, test questions, flash cards, etc. that may or may not be downloadable content. So, no cameras, GPS, user-driven content management, or anything like that; just as device-agnostic as we can possibly make it.

Answer 1

Well, the good thing about Android's code is that you can implement it in so many ways - from a Tablet to a wrist-watch, thus creating a testing group for it a bit complicated.

The way I see it, excluding the version differences for now, there are a few test groups: Tablets, High-End Handsets, Low-End handsets and finally, like the guy above me said - old handset.

I specifically made the distinction between low-end and old since while both might be considered nowadays to be around the same level, old devices would usually run an earlier version of Android which might cause the code to behave quite differently.

There are specialized online services, like Perfecto Mobile or DA that gives you RDA access to real devices (as opposed to emulators, which like you said - only goes so far) without having to physically own them.

Perfecto offers a free hour-long demo on their website (http://www.perfectomobile.com), and I think this might exactly suit your needs.

Answer 2

Presumably, you won't have to code exceptions for various devices (i.e. all of the Android devices should behave the same). Accordingly, you should only need one Android device; it should be a relatively popular one. By the time v2.3 of Android is released, maybe the market will be less fragmented.

Answer 3

We go by market popularity and platform version, making sure that things at least work on the most popular devices available. Depending on what your app does and what hardware it interacts with, there may or may not be significant differences in how it runs between handsets. For instance, Camera drivers are incredibly different between devices.

Solving the screen layout and density differences between low, medium and high density devices is a problem that should only need to be solved once if done properly.

In my experience, the majority of our issues were found by testing on devices that run the full range of available/popular platforms. You have to draw a line somewhere, for a small team, 5 or 6 devices is probably the limit of what is reasonable to test (this of course depends on your release schedule).

--Addition--
After working in the Android world for some time, I would revise my advice a little. We now see the most oddities when going to different manufacturers. I would recommend having equal representation across manufacturers, i.e. HTC, Motorola, Samsung, LG.

You just won't know what these manufacturers have done to the stack until you test. We've found odd things like text fields with different character limits along with tons of camera driver and audio driver issues that are unique to each manufacturer but not specific to 1 device.

Answer 4

Pick a device with the oldest OS version your code supports.

Pick a relatively slow device, so that you notice any performance problems.

Answer 5

To be honest, I would buy 4 phones.

• 3 Phones for each resolution (currently android only support 3 resolution)
• These 3 phones would be the least powerful if possible
• And the fourth phone would be a dev phone

Edit

The dev phone has some advantages. When developping you need a platform that works. Other phones might have modified version of android that could cause problems. The debugging might be disabled. Some hardware that will behave differently (consider physical keyboard)... The dev phone is aimed for developping so you won't end up with some weird issue that slow you down.

The dev phone might be slightly faster than the least powerful phone of the same screen resolution. For example the Nexus S and Nexus One are kind of beasts. So when you'll want to test your app for performance, it might be good to test on a least powerful phone. And even if your code is efficient and fast, you can't expect the user interface to react as fast on a less powerful phone.

The other phones would be only needed when you actually are going to release anything. It's pointless to test your app on all phones everytime.

So to summary you need phones to test user experience and one phone to test your code.