Questions tagged [web-scraping]
Web scraping is automated information extraction from web sites.
44
questions
2
votes
2
answers
190
views
Async scraper in Python
I've written a web scraper and would like it to run as quickly as possible. The scraping isn't trivial; I scrape a few web-pages, gather links from them, scrape those, then gather links from those, ...
- 159
-3
votes
3
answers
166
views
How to identify whether or not 2 pieces of text are identical? [closed]
Let's say I was to create a scraper. At some point I'll need to come up with algorithm of identifing whether or not a piece of a newly scraped text matches the one that's already in the DB. How would ...
0
votes
1
answer
83
views
Is Web Scraping Viable When Information Needs to Be Formatted and Displayed
I'm working on a project where I want to display a list of wedding venues within X miles of a users location. My first thought is that I will use some type of web scraper to pull in a list of venues. ...
- 131
-1
votes
3
answers
71
views
Is there a secure way to ensure a data in an API endpoint of mine came from an Instagram endpoint?
Is there a way through encryption/keys/jwt or anything else to ensure that the data being sent through a POST request is only data coming from another request I made on the client to a 3rd party ...
- 229
-2
votes
1
answer
72
views
How to manage source data for movies and series?
I am trying to build a system that tells the user on which platforms (like Netflix, prime, etc.) a movie or series is available. What is the best way to go about it?
I have considered the following:
...
- 11
2
votes
1
answer
68
views
Scraper in separate repo from visualization component?
Let me explain my thoughts about architecture of the project I'm working on.
The project code repository consist of:
Scrapy component - of course it serves to scrape data, process it and calculate ...
- 23
4
votes
1
answer
6k
views
How to approach a large number of multiple, parallel HttpClient requests?
I have a website which offers pages in the format of https://www.example.com/X where X is a sequential, unique number increasing by one every time a page is created by the users and never reused even ...
- 51
1
vote
2
answers
97
views
Normalising pagination regardless of the target source
I have a service that fetches data from a target source (not through an API but via scraping) which can change. I want to do pagination so that I return 35 items per page but the target source is 25 ...
- 359
2
votes
1
answer
381
views
Advice on designing a scraper DSL
I am creating a DSL for a scraping library I am writing. I would like advice on how to design a DSL, and if the designs I have below are good ones. Apologies if this is an open-ended question, but it ...
- 121
1
vote
1
answer
593
views
How to check if user is logged in after logging using http post?
I'm developing a Scrap app to extract some information from a sit. To get that information I have to be logged in to that site.
So I use Http post and pass the data needed for login using FormData ...
- 303
1
vote
1
answer
146
views
How large-scale production web crawlers improve performance when it comes to running into already visited links
I have built a very basic webcrawler running off my laptop so it has limited memory and limited hard drive space. The way I have it now is I'm using MongoDB to store the links I find on pages. I make ...
- 141
1
vote
1
answer
93
views
Lightweight data mining + organization & visualization
I'm looking to do some simple data mining that consists of going once per day to a single page and collect the following information:
List of movie theaters
Movies today on each theater
Session times ...
- 21
3
votes
1
answer
219
views
Is it possible to layer an API (REST, GraphQL, etc.) in front of data that is currently only accessible via an enterprise desktop GUI?
Currently, my thoughts are that GET requests would be feasible by using the concept of screen scraping combined with a cron job that runs at a set interval to scrape data from the GUI and sync to my ...
- 147
1
vote
1
answer
154
views
Should I opt to not use someone's API?
At a job that I recently started, I inherited some of the projects from the guy who previously held this position. One of the projects was a program that used a Website Platform's public API to get ...
- 287
2
votes
1
answer
33
views
Web Crawling Form Help Needed [closed]
If I have a form built on my website. How can I get that form 1 to plugin results into another websites form 2 and extract information to display on my website?
For example, if I wanted to create a ...
- 29
2
votes
0
answers
372
views
How to maintain table relationship across services in with a micro-service architecture
I have a scraper service that looks for media online. For every source type,
there are two tables. A common source table and a source_type_media table that are related by source.id to ...
- 121
1
vote
1
answer
104
views
Modeling web scraper
I'm working on a small application that is supposed to scrape/parse a few website and I'm wondering which would be the best way to achieve this (keeping DRY/SOLID in mind).
Here's some pseudocode:
...
- 11
7
votes
3
answers
2k
views
Preferred approach to mock a site to test a scraper
Subj. Atm I'm using Selenium and Python, but the same applies to any other scraping solution.
I'm wondering:
which of the options outlined below are optimal/recommended/best practices
if there are ...
- 593
2
votes
0
answers
152
views
Collecting sports data from many sources in many formats
I'm trying to solve a problem that exists for a sport. The sport is fragmented into many federations, and each federation is completely isolated from other federations. The sport is the same for each ...
- 131
5
votes
2
answers
3k
views
How to make a webdriver run reliably in Selenium?
I have been having quite a time getting this to work reliably for 100s of thousands of terms and potentially millions of pages per source and ETL the resulting data into a database in an automated ...
- 207
-1
votes
2
answers
909
views
Does Multithreading help in web scrapping?
To my knowledge multi-threading is just an illusion since CPU schedules time for each of the process. So when scrapping at a particular time, only one content from the website will be scrapped and it ...
5
votes
1
answer
299
views
How much processing to do in the crawler? - good crawling practices
I am currently working on a pet project in Python with scrapy that scrapes several ebay-like sites for real-estate offers in my area. The thing is that some of the sites do seem to provide more ...
- 153
0
votes
1
answer
570
views
What should I consider when scraping content from 6k+ Websites?
I am working on a project that will scrape data from roughly about 6,000 websites. I am considering using a server/client model in which the server is responsible for scraping the data and importing ...
- 109
1
vote
2
answers
4k
views
Is there a limit to how often I should access a website programically to avoid DDOS? [closed]
I have my own website that displays some data I get from webscraping. I run the webscraping script on my website host server. I just realized that maybe I'm accessing other websites too quickly ...
- 27
4
votes
1
answer
1k
views
Android App with Ruby Backend Server
I'm working on a personal project to help me branch out and learn some new/different technologies. I'm a .NET programmer but I want to learn Ruby and how to develop Android apps. I have developed ...
- 181
4
votes
1
answer
292
views
Scraping data from website and passing into Office - a lot of restrictions [closed]
Recently, I was asked to help with some side optimization project at our company, I've made some good research. I'm still not 100% sure if this is most efficient way to do this.
Problem:
Scraping ...
- 143
-1
votes
1
answer
2k
views
Crawler - programming language choice [closed]
I'm currently developing a web crawler. The first version was developed in Node.js and runs pretty well.
The issues that I encountered with Node.js are in no particular order:
slow URL and query-...
- 135
1
vote
1
answer
1k
views
Can I scrape a website for font stylings?
I am trying to scrape websites for valuable text for example the title of an article, the author's name, and other distinguished text. I cannot always guarantee that this sort of text will have ...
- 409
1
vote
1
answer
1k
views
Create a loop that visits every link on a site [closed]
I want to create an HTML scraper in Ruby on Rails and I want to implement a progress bar and an ability for it to pick up where it left off if it's interrupted while scraping.
I think the best way to ...
- 123
1
vote
1
answer
938
views
Scrape Intranet site without a web server
I'm trying to rebuild a simple c# time tool that displays certain statistics about time worked for the web, however, I don't have access to a server so I can't use server code like PHP which I would ...
- 121
2
votes
3
answers
7k
views
When there is no API
When it is necessary to integrate with a web application, and an API is unavailable, is it a viable solution to simulate a web browser interacting with the web application as a real user would ...
- 31
0
votes
1
answer
177
views
Getting Started with Data Collection and Analysis [closed]
I have a very very limited understanding of SQL and PHP. I have a few projects I want to try and work on, which will require doing some web scraping for data collection and then storing, sorting ...
- 11
4
votes
1
answer
13k
views
What is the way to go to extract data from websites? [closed]
I've been thinking about a side project that envolves web data scraping.
Ok, I read the Getting data from a webpage in a stable and efficient way question and the discussion gave me some insights.
...
- 49
3
votes
1
answer
965
views
Preventing crawler from interfering with user tracking
I'm scraping text from various webshops (no images/videos or other data). I'm no expert on user tracking, so I'd like to know if there's a way for me to write my crawler so it won't interfere with the ...
- 180
0
votes
5
answers
2k
views
Blocking IP address for web-scraping service
Background
Consider the following scenario:
Link. User provides a link to some poorly formatted website (e.g., creative commons content).
Scrape. Server downloads the content (web scrape), always ...
- 753
30
votes
3
answers
19k
views
What will happen if I don't follow robots.txt while crawling? [duplicate]
I am new to web crawling and I am testing my crawlers. I have been doings tests on various sites for testing. I forgot about robots.txt file during my tests.
I just want to know what will happen if ...
- 419
1
vote
3
answers
338
views
From where do financial firms obtain the stock data to analyze [closed]
I've been thinking about the subject and I've always wondered, from where do they obtain the data that is analized in applications, I've looked at Nasdaq's website and they do not seem to have any ...
- 177
3
votes
1
answer
6k
views
Improving performance for web scraping code
I have a website in which the code scrapes other websites for getting the accurate data. While the code works good but there a decent lag in performance because the code firsts downloads the html ...
- 5,052
-1
votes
2
answers
690
views
How do I capture information from a website that doesn't provide an API? [closed]
Do you know any good tutorials, frameworks, anything that can help me to write code that captures information from a website that don't have a public API, or hasn't been written in a RESTful way?
I'...
- 219
1
vote
2
answers
9k
views
A good tool for browser automation/client-side Web scripting [closed]
I'm interested in adopting a tool/scripting language to automate some daily tasks connected with fighting forum spammers. A brief overview of these tasks: analyze new registrations and posts on a ...
- 836
2
votes
5
answers
2k
views
Data Scraping - One application or multiple?
I have 30+ sources of data I scrape daily in various formats (xml, html, csv). Over the last three years Ive built 20 or so c# console applications that go out, download the data and re-format it into ...
- 21
10
votes
4
answers
13k
views
Patterns and practices for Web Scraping in .Net (C#) [closed]
I will be putting together an application to automate an external web site/application. In some instances I will need to navigate the site as a user would (some links I need to follow cannot be ...
- 1,267
85
votes
7
answers
19k
views
How to be a good citizen when crawling web sites?
I'm going to be developing some functionality that will crawl various public web sites and process/aggregate the data on them. Nothing sinister like looking for e-mail addresses - in fact it's ...
- 43.8k
3
votes
4
answers
13k
views
Which language is the most flexible for scraping websites? [closed]
I'm new to programming. I know a little python and a little objective c, and I've been going through tutorials for each. Then it occurred to me, I need to know which language is more flexible (python, ...
- 183