0

I'm looking to do some simple data mining that consists of going once per day to a single page and collect the following information:

  • List of movie theaters
  • Movies today on each theater
  • Session times for each movie
  • Availability of tickets for each session (boolean yes/no)

I want to catalog this information and after 1 year see an overlay graph showing for each movie (if the same movie exists on 2 theaters I want 2 distinct instances):

  • x axis: time (discrete dates)
  • y axis: number of sold out sessions

Aditionally, it would be useful to get another heatmap graph letting me know for each movie what times were the most sold out.

What would be a lightweight solution to go about this? I'm not familiar with data scraping tools or simple lightweight databases for this. I'm inexperienced in this line of work and looking to do the least coding possible. Excel would be fine for me. I've had past experience with Java and C# but would like to avoid killing a fly with a bazooka.

I'm currently thinking about doing this with ParseHub but not sure how to merge the information from 1 csv file per day to obtain the information I'm looking for.

PS: I found the JSON source that populates the web page so I could fetch that once per day, I just need to grab the important information from that file and structure it (in an Excel). Just don't know what tools I could use to accomplish such a task.

Thank you

Improve this question
3
  • There is no free lunch. You want to have a full data pipeline from a non-structured data source; have it parsed merged indexed and see interactive representations tailored to your instance. I advise you to reframe your question/approach to this matter.
    – nadir
    Commented Jan 23, 2019 at 16:24
  • Well it seems that you understood exactly my need, so I beg the question why reframe or in what sense? By the way, I added new information at the end of the post, just found out the JSON data source for the page I want to scrape.
    – dR_
    Commented Jan 24, 2019 at 4:21
  • Yes, I understand. What I tried to convey in my comment is that the system you mention can't be "hacked" by throwing some closed applications that weren't intended for this specific purpose. You'll have to put effort into this and code it (you don't need to invent the wheel on each stage and your requirements don't demand you to write the craftiest software, but you still have specific demands that need to be resolved here).
    – nadir
    Commented Jan 24, 2019 at 10:57

1 Answer 1

Reset to default
1

My recommendation is to pick up Python. There are lots of tutorials to get you going, and there are TONS of examples on how to scrape websites using tools like BeautifulSoup, and if you found a JSON api then you can grab it using tools like "requests".

Then you can schedule your program to run daily using tools like cron if you are under Linux. On Windows you can use the Task Scheduler to run your Python script.

The Python program can append to a comma-separated-variable (CSV) file each day, which you could then open in Excel to play around with plotting. There are also Python plotting utilities -- use whatever tools work for you.

Once you have these tools in your toolbelt, you'll be able to do all kinds of other projects while you wait a year to build up your database of movies!

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.