Questions tagged [web-crawler]
The web-crawler tag has no usage guidance.
23
questions
0
votes
0answers
12 views
Storing string parsing function
I'm writing a system where data will be extracted from HTML formatted strings.
Different types of strings will contain the same type of data, but the methods to extract that data will vary between the ...
0
votes
1answer
405 views
How to check if user is logged in after logging using http post?
I'm developing a Scrap app to extract some information from a sit. To get that information I have to be logged in to that site.
So I use Http post and pass the data needed for login using FormData ...
0
votes
1answer
116 views
How large-scale production web crawlers improve performance when it comes to running into already visited links
I have built a very basic webcrawler running off my laptop so it has limited memory and limited hard drive space. The way I have it now is I'm using MongoDB to store the links I find on pages. I make ...
2
votes
1answer
732 views
How to reverse engineer URL routes from a bulk of HTTP requests/responses
I am building a web application crawler that crawls for HTTP requests (GET, PUT, POST, ...). It is designed for one specific purpose; bug bounty hunting. It enables pentesters to insert exploit ...
1
vote
0answers
1k views
What are the best practices for picking selectors for web scrappers?
The following is an example using https://github.com/GoogleChrome/puppeteer
'use strict';
const puppeteer = require('puppeteer');
(async() => {
// const browser = await puppeteer.launch();
// ...
1
vote
1answer
772 views
detecting website opening in new tab/window?
So, as part of my final year project, I'm writing a web crawler in Java to gather website data that I will then process. One of the attributes I need to gather is "number of popups". I know a pop-up ...
3
votes
1answer
200 views
Is it considered bad practice to crawl through the mobile version of a site?
I am building a web spider to crawl through several different sites, but one of them uses javascript buttons instead of links for several functions. And while I could learn to follow them, it adds an ...
4
votes
1answer
277 views
How much processing to do in the crawler? - good crawling practices
I am currently working on a pet project in Python with scrapy that scrapes several ebay-like sites for real-estate offers in my area. The thing is that some of the sites do seem to provide more ...
5
votes
1answer
646 views
Can I whitelist user agents that will execute JavaScript?
I'm building a SPA (single page application) so that when a browser request a page from my server, it only receives a small HTML and a big JavaScript app that then asks the appropriate data from the ...
-2
votes
1answer
2k views
Crawler - programming language choice [closed]
I'm currently developing a web crawler. The first version was developed in Node.js and runs pretty well.
The issues that I encountered with Node.js are in no particular order:
slow URL and query-...
4
votes
0answers
109 views
IRLBot Paper DRUM Implementation - Why keep key, value and auxiliary buckets separate?
Repost from here as I think it may be more suited to this exchange.
I'm trying to implement DRUM (Disk Repository with Update Management) as per the IRLBot paper (relevant pages start at 4) but as ...
2
votes
2answers
1k views
Patterns for creating adaptive web crawler throttling
Im running a service that crawls many websites daily. The crawlers are run as jobs processed by a bunch of independent background worker processes, that picks up the jobs as they get enqueued.
Now ...
1
vote
2answers
332 views
How do I ensure my site will be crawled when articles are generated by the database?
I wasn't sure how to ask the question. But basically, it's a textbook scenario. I'm working on a site that's article based, but the article information is stored in a database. Then the page is ...
-1
votes
1answer
3k views
web spider for facebook
I just subscribed to a facebook page which post links to different open source projects or code archives. I'll like to save those links and descriptions to a local db.
How can I do that? I heard ...
5
votes
1answer
13k views
What is the way to go to extract data from websites? [closed]
I've been thinking about a side project that envolves web data scraping.
Ok, I read the Getting data from a webpage in a stable and efficient way question and the discussion gave me some insights.
...
4
votes
2answers
5k views
How to find a 'good' seed page for a web crawler? [closed]
I started building a web crawler and read somewhere that it's a very hard problem to find a good seed page for the crawler. Can anyone explain me if there is any pre-defined procedure/ guidlines of ...
28
votes
3answers
17k views
What will happen if I don't follow robots.txt while crawling? [duplicate]
I am new to web crawling and I am testing my crawlers. I have been doings tests on various sites for testing. I forgot about robots.txt file during my tests.
I just want to know what will happen if ...
3
votes
1answer
190 views
Ways of Gathering Event Information From the Internet [closed]
What are the best ways of gathering information on events (any type) from the internet ?
Keeping in mind that different websites will present information in different ways.
I was thinking 'smart' ...
2
votes
5answers
803 views
Website Country Detection
I have a web crawler, and I'm looking for hints that will help me automatically detect a website country of origin.
And by country of origin I generally mean the country the website is targeting. For ...
6
votes
4answers
4k views
Development of a bot/web crawler detection system
I am trying to build a system for my company which wants to check for unusual/abusive pattern of users (mainly web scrapers).
Currently the logic I have implemented parses the http access logs and ...
8
votes
3answers
3k views
Looking for good books about the theory behind search engines [closed]
I am working on a project that requires that I understand different techniques used by search engines for the web.
I have a strong scientific and development background, so I am not afraid of highly ...
0
votes
2answers
3k views
Roll your own web crawler to crawl one specific website that has multiple entries
What sort of languages would be able to handle writing your own web crawler?
Could PHP handle this? I'm quite good with PHP (following best practices etc).
But I'd like a good reason to learn a ...
85
votes
7answers
19k views
How to be a good citizen when crawling web sites?
I'm going to be developing some functionality that will crawl various public web sites and process/aggregate the data on them. Nothing sinister like looking for e-mail addresses - in fact it's ...
