1

So, as part of my final year project, I'm writing a web crawler in Java to gather website data that I will then process. One of the attributes I need to gather is "number of popups". I know a pop-up blocker can display the number of popups blocked on a given website, and this information would be sufficient.

Alternatively, I can define popups as avascript alert windows, and other websites opened in a new tab or window, and gather a count manually.

Which method would be easier, and how could I attain it?

2

There are two techniques to open content in a new tab or a popup: target="_blank" and JavaScript's window.open().

While the first one is relatively easy to detect for a crawler (although it's not the simplest task either, as Brian explained in his comment), the second one will require you to parse JavaScript, and by parse, I mean execute it in a sandbox. Given the dynamic nature of JavaScript, you can't just grep for window.open\(*\), because the method itself can be invoked as w.open() where w was previously assigned to window, or a.a() if the code was minified, or... well, you get the pattern.

More importantly, you need to know when the popup is opened. A page can spam user with five popups automatically as soon as it's opened, or a popup can be opened on a click event from the user, or maybe it's buried deep inside complex logic which means that it will be opened only in a very specific case which happens once per year for two-three users. This will make it difficult to impossible to determine the number of popups programmatically, because:

  • It comes to determining what should you include, and what you shouldn't. Should your crawler consider only popups which are shown on page start? Or maybe on page start and on click events? Or on a timer?

  • Finding all possible execution paths through the code may take a very long time your servers may not necessarily have.

  • The first isn't necessarily so easy for a crawler, either. Sometimes, target is set to _blank dynamically. Further, if the target is set to foo, it will only open one foo popup window; any future popups with target set to foo will share the same window. Further, anchor tag could also be programmatically clicked. Finally, it's worth noting that most web browsers will tend to carry a user click across all synchronous calls, allowing only one popup per click, after which the click is used up and any further popups will be stopped by the popup blocker. – Brian Oct 25 '16 at 20:36
  • Okay, so that option seems like it'd be more work than its worth. Is there any way of interfacing with a popup blocker, maybe? – Sophie Brown Oct 25 '16 at 21:50
  • @SophieBrown: a popup blocker is a browser plugin; it runs in client's browser. You're building a crawler; it runs on a server. You can run the web app in a sandbox on your server, but this is similar to what I already suggested in my answer, and you don't need any plugins, since you're in control of the sandbox (and know whenever the web app performs specific calls.) – Arseni Mourzenko Oct 25 '16 at 22:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.