UI Testing with Visual Studio 2010 Feature Pack 2

Question

One of the most intriguing items in the recently released Visual Studio 2010 Feature Pack 2 is the ability to create and edit UI tests in Silverlight.

Here is an example of a coded UI test.

I haven't had much time to use it yet, but for people who have, I am curious as to what your thoughts are. Is this something you have found to be particularly useful?

I would like to be able to automate a significant amount of regression testing that we are currently performing manually.

In your experience, has it made a major impact on the resources that you would normally have to dedicate to testing?

Thanks

Answer 1

This boondoggle of automated UI testing is great in the short-term if you're running a QA department. Your staff will look super busy while they constantly write and re-write automated UI scripts. You can add another line on a report to brag about how much test coverage you do.

Over time, your department becomes a very obvious bottleneck that undermines productivity. Developers can't pass your automated tests because QA hasn't updated them to work against the new UI. This makes developers hesitant to introduce any UI changes.

Meanwhile, your business (or your client) isn't making the production release deadlines. The schedule slips while QA is scrambling to update the automated UI scripts against the new UI.

So, at this point, some reflection is in order.

The sell of automated UI testing is that it somehow liberates your QA staff from the testing effort required to go through the entire UI of an application.

What you've actually done, however, is replaced the tedious steps of user testing with the tedious process of scripting out those steps to a machine.

That initial cost would only be worth it if the steps are invariant, or if your application is very simple. It doesn't take science to determine that most UI development is anything but invariant. If your application is simple, then the testing effort is minimal.

I've seen two different companies try and implement this, and it's always ended in rolling heads and lawsuits.

Answer 2

My team is currently using WebAii. Automating tests takes a significant investment, but regression tests combined with a CI build & test environment is incredibly valuable.

Our current SilverLight CI environment is a bit more complicated than implied above because we use multiple test frameworks for different areas of the code and to attempt to get a variety of metrics out our product. In particular, code coverage has been fairly problematic. We have a partial solution that is far from ideal.