1

I'm researching ways to build a web service to periodically traverse a predetermined list of web pages (of another external website) to detect if a page's content has changed from

  1. editing of the page, and
  2. deletion of the page.

The end goal is to have this web service post push-notification events to mobile devices.

FYI, I've searched and read "Questions with similar titles" here.

Thank you for sharing your answers.

Improve this question
3
  • Can you share your current research? I feel that your question is too open ended for programmers as it stands.
    – Oded
    Commented Mar 24, 2012 at 20:52
  • I don't think you really want a web service, since you're not actually providing a service. This type of program would be more of a periodic task or cron job.
    – TMN
    Commented Mar 24, 2012 at 21:40
  • Thanks for your feedback. I was looking at the problem as a web service because this service (to monitor an external website) will then initiate push notifications to an app on a mobile phone.
    – Global nomad
    Commented Mar 25, 2012 at 5:44

2 Answers 2

Reset to default
2

What about just reading the page with cURL? Maybe store a checksum of what you read and compare against that next time.

Improve this answer
1
  • Thanks for the suggestion. I was thinking of a checksum solution but thought I'll seek alternative approaches before deciding.
    – Global nomad
    Commented Mar 25, 2012 at 5:45
2

Editing of the page can be checked by:

  1. Check if the HTTP header last-modified is present. If it is, compare it to the previously stored value and you are done.
  2. If the last-modified HTTP header is not present, store the page checksum or checksum of the important part of the page. Since some of the web pages contain randomly generated parts, you want to ignore these parts. You can define the 'important part' of the page using e.g. regular expression. Next time you will compare the stored checksum with new one.

Delete of the page can be checked simply by trying to connect to it.

Improve this answer
1
  • Many web servers, web applications, and even proxies by default set the last-modified to NOW or don't implement it (properly) at all - so the page is ALWAYS changed.
    – Ben DeMott
    Commented Mar 31, 2012 at 10:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.