10

I will be putting together an application to automate an external web site/application. In some instances I will need to navigate the site as a user would (some links I need to follow cannot be predicted and must be parsed from a response)

I am already using Html Agility Pack, and am aware of Tidy if that is needed.

Are there any other technologies I should be aware of?

Are there any recommended patterns for being able to quickly adjust in the event that the external web app changes? I’m envisioning encapsulating the validation of responses as some type of strategy or similar pattern that can be easily separated/plugged in as necessary, but any specific suggestions would be great.

Improve this question
1
  • About it, any step-by-step guide, or Getting Started - Quick Start that can be completed in 15-20 minutes? I get you a good step-by-step guide sample like berniecook.wordpress.com/2013/01/13/… I wanted to quickly cover off several assumptions before we get started: install (requisites, tools required), configure, an running quickly. Target will be an "go and ready" sample. Maybe better an real application sample.
    – Kiquenet
    Dec 29, 2014 at 13:12

4 Answers 4

Reset to default
4

If you're looking to automate navigation of an external website as a user would, Watin is perfect for that. It will drive a web browser through an object model, and has a wide range of parsing capabilities built upon the DOM (as well as non-DOM abilities you would find in a browser, including scripting).

Here's a link:

Watin.org

Improve this answer
2
  • Watin is like your favorite JavaScript library for C#.
    – The Muffin Man
    Jul 28, 2011 at 20:52
  • Any final solution with full source code sample application ? IMHO, better samples for minimize learning curve are real applications with full source code and good patterns&practices
    – Kiquenet
    Dec 29, 2014 at 13:13
0

If you're looking for the ability to quickly make changes to track those in the external web app, then I'd look at scripting. If C# is your language of choice, then look at compiling code on the fly. If you're using a database, the script code can be stored there, otherwise, plain text files would work too.

As to GoF patterns, Strategy is the textbook choice.

The other "pattern" to consider, especially since you mentioned validating responses using them to select the next action, is a finite-state machine. For example, the states for a web-mail client could be "not logged in", "at inbox", "reading message", "editing reply", etc. You'd execute some action depending on the state, then based on the response, select the next state.

Improve this answer
0

The pattern really should stem from the problem but here are some choices:

If you are mainly concerned with ongoing development cost/annoyance: Strategy Pattern (already mentioned) - use this to implement the component level interfaces, and use a dynamic binding methodology to resolve specific strategies (implementations). An IOC container (I like Autofac these days) would work well.

If you are need to support the above plus have a need to scale: Map Reduce (anyone know of a good m+r fx for .net?).

Improve this answer
-1

Depending on how much the webpage changes this could be tough. You could keep a list of standard regular expressions in a DB and then when one no longer works you can loop through the base set with others until you hit one. Then you'd want to mark that for future use.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.