I'm currently working on a project with a partner where we analyze large datasets of past sporting events. There are approximately 30,000 events per year and we have historical data for five years. Thus, we're looking at analyzing 150,000 events.
I wrote our first version in Python with a heavy reliance on Pandas (great tool, by the way). However, each time I add a "factor" or calculation to the process it slows down by quite a bit. Simply put, it can read all the various data files and data queries we need at a rate of 25-events-per-second.
The problem is once I start working with the data, it drops fast. If I add a simple calculation, it drops to 24... and each calculation after that it drops again.
With just 10-12 "factors" calculated from the data (we're not talking complex loops or anything crazy), it's now down to 6-events-per-second. ... and we're barely scratching the surface. At this rate, it'll take this thing days to work through all 150k events!
I've optimized our data queries, flat file reads etc. Those really aren't the issue. I can live with doing 15-20 events per second. However, I can't live with such drastic slowdowns each time a new factor calculation is added.
I've read a lot where Python as an interpreted language is slow etc., but my question to the experts out there - what language should this project be done in?
One of the key "DataFrames" I am using through Pandas is fairly large. 350 rows x 70 columns, as an example. Even when trying to simply multiply one column's value by another...
data['value'] = data['col1'] * data['col2']
...I see what I could consider significant drop in performance. It's very puzzling, and very frustrating. I understand this is a large set of data, but I can't believe this is anything so crazy that Python would be slowing to a crawl.
If I just read the data files and do no calculations on them at all, it reads 67 events in 2.807 seconds. If I add one simple calculation where I execute code like the example above, it slows to 2.877 seconds. Based on our research, we're needed to add upwards of 100 calculations on the data.... so 7 seconds worth of slowdown? Seems too hard to believe.