-1

I'm working on a project where I want to display a list of wedding venues within X miles of a users location. My first thought is that I will use some type of web scraper to pull in a list of venues. My concern is that the data will be in different formats and I won't be able to list item with a picture, description, price range and url to the venues web page.

Am I correct that web scraping is best to gather data that won't be directly mapped into a user interface?

I am trying to think of a better way to do this and the only other thing I can think of is to find venues with a public facing API and pull data directly from them. This seems like it would be tedious however.

  • 1
    Tedium is writing a custom scraper for each website. There are things you can do to streamline the process, but writing scrapers is a craft, and you will be spending a fair amount of time on that craft figuring out how each website works. If they exist, public-facing API's are a far, far better way to go. – Robert Harvey Aug 17 at 14:24
1

My concern is that the data will be in different formats and I won't be able to list item with a picture, description, price range and url to the venues web page.

Sure. So your scraper has to be smart enough to understand the different pages - that's not going to be a trivial problem if you're trying to do anything more than pull out any nicely marked up metadata that happens to be on the pages you scrape, but it's not impossible to do with a reasonable degree of accuracy.

Am I correct that web scraping is best to gather data that won't be directly mapped into a user interface?

That doesn't really seem relevant here - once you've scraped the data for the venues, you almost certainly going to be putting them into some kind of database anyway, and at that point you can have a separate presentation layer to display the data.

Realistically here, potentially other than price (which is rarely listed on the page anyway in my experience), you seem to be trying to recreate something like the Google Places API. Why not just use that instead?

| improve this answer | |
  • I think Google has a few APIs that I could use here so I will look into that more. I eventually want to build my site out to include other wedding related things (caterers, bands, decorations) and for these things I don't think Google has an API. – tdammon Aug 17 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.