0

At a job that I recently started, I inherited some of the projects from the guy who previously held this position. One of the projects was a program that used a Website Platform's public API to get certain data from their site. The program itself is pretty simple, and is written correctly from what I can tell. It's not that complicated, so any big errors probably would have stuck out. When testing this program however, I get inconsistent results compared to their website. Ex. Searching using a keyword in the program yields 30 results and Searching using the exact same keyword on their official website yields 60 results.

After researching, I found that this was because the two searches use different internal mechanisms to return the results. Fair enough. I was thinking about approaching my boss about re-writing this application and using an approach where I query to website directly, get the HTML, find the certain class where the text I want resides and get it from there. It shouldn't be that hard (I know, famous last words...) and it would help our results be more accurate and more extensive.

Before I could approach him about this, he said that he wants to do the same kind of thing, but with another website with a similar platform. They have an API as well, but I'm wondering if we're just going to be running into the same problems here as well down the road.

My boss is not a programmer and doesn't know about the specifics of APIs and web queries, so that's why I'm asking here. Is it a bad idea to not use someone's public API, and instead just process the raw HTML for the data I want?

|improve this question
  • 30 results vs. 60? I think you answered your question. – Phil N DeBlanc Sep 12 '18 at 20:45
  • @PhilNDeBlanc Is that enough reason to opt for this approach on the new site as well? Or is this a bad idea? – SH7890 Sep 12 '18 at 20:48
  • I can't answer that. Sorry. Go with your gut on this one. – Phil N DeBlanc Sep 12 '18 at 21:01
  • Research the terms of service, you may put your company in a bad spot if you're not compliant with them. – Paul Sep 12 '18 at 21:10
  • 3
    @SH7890 I'm saying that some folks who publish an API and a website don't want you to scrape their site for data for various reasons, which is why they provide an API. You should make sure that they are OK with you scraping vice using the API before doing it, or else they might block your company's crawler and/or issue a cease and desist order. – Paul Sep 13 '18 at 3:21
7

Yes. It is bad practice to scrape a webpage rather than use the published api.

  1. Just because its on the web doesnt mean its free. The site owner pays for the bandwith and servers to run their site and may not want you to hit it with a bot. They may consider it an attack and sue or take techincal measures against you.

  2. Websites change. Everytime they change the style of the page you will have to update your scraper. You won't get notice of this change, so you will get errors when it occurs.

If the api doesnt fit your needs I would call up the site owner and ask if they can add another method, or tell you how to achieve the result you want.

|improve this answer
  • If they offer an API, as long as you abide by the same usage limits, they're not likely to get agitated as you describe. – whatsisname Sep 13 '18 at 5:35
  • 1
    @whatsisname they offer an api, they are more likely to be annoyed if you ignore it and hit the webpage – Ewan Sep 13 '18 at 6:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.