How to programmatically control a GUI application?

Question

I have a couple of users who have a User Interface for an application that they must use to do their job. The UI can be cumbersome to navigate to perform some time critical tasks.

This UI does not expose any API and has no integration points.

I would like to be able to do something to help the users to perform various tasks.

I've been looking into the feasibility of using VS2013's coded UI tests to allow me to control the UI from some of my own code which would run on the user PCs. Obviously this is not really what coded UI tests are intended for.

Has anyone else tried this technique to integrate code with a closed UI? How successful were you? Are you aware of any other frameworks which would achieve the same result?

In some more detail (not required reading for question): The UI in question is a market making tool within finance. For simplicity sake, let's say the use of it is enforced by regulation. Currently the users will run a model and then manually type several values into the UI as their bids and offers. This is time consuming and has potential for human error (even with copy and paste). I would like to create an app that took the correct values from the model and posted them into the correct boxes on the screen.

Things I can control:

• The application they take the values from
• The integration software

Things I cannot control:

• Anything to do with the market making UI

Any suggestions or advice much appreciated.

Answer 1

You can probably make it work. But, I find when using coded UI, or other test frameworks you are quite dependent on changing the code to make the various buttons and other elements easy to pick out and interact with.

I think you will find it easier to just buy an off the shelf automation product

Answer 2

Have you looked into Spock and Selenium? Spock can load pages and selenium can manipulate UI elements. This can be set up for purposes other than testing and should run just fine. Beware, things can get tricky if you have UI elements that need to be activated, then you have to wait for something to load, and then you can proceed. Some creative coding may become necessary when it comes to telling it to wait the required time.

Selenium is not generally loved by most people who have to use it a lot, but it works. It should be possible to set up a script that would run and a sequence of actions in the UI occur.

Answer 3

How to programmatically control a GUI application?

I guess that you have written, or at least can modify that GUI application, so have access to (and understanding of) its source code (and the capacity and resources to improve it).

Then you could add some RPC-like service to that application. You could consider adding some JSONRPC, some REST or other Web service, some SOAP, etc...

Adding such service to an existing GUI application is generally easy (since GUI frameworks have some event loop).

If you have no control (and no access to its source code) on the GUI application, in general you cannot control that GUI application programmatically. But depending on the display server (X11, Wayland, etc...), the operating system and the widget toolkit used by it, in some specific -and rare- cases you might control it programmatically (for example, some recent GTK applications using Gobject introspection).

Perhaps the best solution is to go back to the vendor of that GUI application and ask him (and pay him) to develop some programmatical interface (and discuss with him the technical details, e.g. the inter-process communication protocol to use).

If you know the source of the data ("The application they take the values from") and the computation that this GUI is making, you might perhaps consider redeveloping your code doing that computation (Only you could know if that makes sense).

Read also Comparison of GUI testing tools and GUI testing wikipages.