My question is about "big data". Basically, big data involves the analysis of a large amount of data to make meaningful insights from it.
I would like to know:
Whether or not large amounts of data can be pre-processed? (like say for example you are running some matching service for people, so you take all the information you have on the people and you process it at a certain point for use later on)
If pre-processing is possible, how would you normally go about doing this?
To help narrow the scope of my question, please look at this hypothetical scenario.
Say I have a customer database and my company is a global retailer that is using some type of points system to reward the shoppers (for arguments sake, the points are tallied up on a type of electronic card or mobile app).
So based on my rewards system, I am now able to fully aware of exactly what a shopper is purchasing and when they normally make purchases of recurring items.
My database is growing all the time with this information and I would now like to make recommendations (or send notifications) to shoppers about special offers of products they buy or related products that may interest them, when they enter 1 of the stores.
Instead of processing all the accumulated data when a shopper enters the store, I would like to continually process the data-stream as the data comes in (meaning from previous shopping experiences), so that when it comes time to make a recommendation (for the next time a shopper walks into the store), it is simply a matter of retrieving the recommendations and providing a list of it to the shopper.
With this method in mind, I can easily space out my CPU-intensive tasks, instead of say: processing all customer data on a busy day when foot-traffic is at peak volumes.
By asking how I would do this, I would be referring to common methods available for achieving this. This can include any special databases or programming techniques or even specialized software that can carry out these timed calculations that can "pre-process" the data at specific times, in order to balance out CPU-intensive tasks.
You can consider the customer-recommendation scenario as the "situation". It is the best example scenario I could think of that would explain why "pre-processing" (or calculating the recommendations at specific times) would make sense.