My question is about "big data". Basically, big data involves the analysis of a large amount of data to make meaningful insights from it.

I would like to know:

Whether or not large amounts of data can be pre-processed? (like say for example you are running some matching service for people, so you take all the information you have on the people and you process it at a certain point for use later on)

If pre-processing is possible, how would you normally go about doing this?

To help narrow the scope of my question, please look at this hypothetical scenario.

Say I have a customer database and my company is a global retailer that is using some type of points system to reward the shoppers (for arguments sake, the points are tallied up on a type of electronic card or mobile app).

So based on my rewards system, I am now able to fully aware of exactly what a shopper is purchasing and when they normally make purchases of recurring items.

My database is growing all the time with this information and I would now like to make recommendations (or send notifications) to shoppers about special offers of products they buy or related products that may interest them, when they enter 1 of the stores.

Instead of processing all the accumulated data when a shopper enters the store, I would like to continually process the data-stream as the data comes in (meaning from previous shopping experiences), so that when it comes time to make a recommendation (for the next time a shopper walks into the store), it is simply a matter of retrieving the recommendations and providing a list of it to the shopper.

With this method in mind, I can easily space out my CPU-intensive tasks, instead of say: processing all customer data on a busy day when foot-traffic is at peak volumes.

By asking how I would do this, I would be referring to common methods available for achieving this. This can include any special databases or programming techniques or even specialized software that can carry out these timed calculations that can "pre-process" the data at specific times, in order to balance out CPU-intensive tasks.

You can consider the customer-recommendation scenario as the "situation". It is the best example scenario I could think of that would explain why "pre-processing" (or calculating the recommendations at specific times) would make sense.

  • 5
    preprocessing is usually a necessity to make large amounts of data usable. The map-reduce strategy is basically built around a cascade of filters and combinators. However, “big data” is a really hazy term to which different people ascribe very different meaning. Could you edit your question to make it clearer what your problem is? As it stands, it's just far to broad, and will likely be put on hold as such.
    – amon
    Commented Feb 24, 2014 at 16:10
  • Also, by 'how', do you mean implementation specifics like what kind of database, database design, and how to start processing jobs? Or are you asking more of a design question, such as 'what factors will influence how we construct our preprocessing?'
    – Snagulus
    Commented Feb 24, 2014 at 16:13
  • 2
    pre-processing is a generic term which is almost as unspecific as "processing" - which means there is no "normal" way of doing it. Its like asking for "how do I solve a problem" without telling which problem. This is why you can expect your question to be closed soon as "too broad".
    – Doc Brown
    Commented Feb 24, 2014 at 16:55
  • Edited question and added more details.
    – Joe
    Commented Feb 25, 2014 at 13:31

3 Answers 3


Typically I've heard of this being handled via the OLTP vs. OLAP model. Essentially the T in OLTP means "transactional", so this is the typical databased used for day-to-day operations. Then you write some kind of translational logic that transforms the OLTP database into an OLAP database (the A stands for analytical).

Basically you're talking about the same data represented 2 different ways. The OLTP database focuses on normalization but the OLAP database is structured in more of a "star" pattern with a lot more data repetition. It's read-only and optimized for querying.

Then the engineering is in figuring out how to do the translation from OLTP to OLAP, how often to do it, and if you can do it incrementally so the OLAP database isn't too far behind "real-time".


In a past job, I was a DBA for a global solutions company where databases with millions and billions of rows were the norm.

As datasets got larger, it became more and more problematic to turn around complex queries in a timely manner.

Among many strategies we adopted, 4 spring to mind:

  • Result sets for common queries were stored in what we called "strips". These were basically index-organised tables that stored keys to stop repeated joins in subsequent queries

  • Denormalising tables yielded huge benefits to reduce the number of joins

  • Tables were partitioned in line with common queries e.g. postcode/zip code etc

  • Whilst all data was available in the repository, only fully formed data and cleansed data was allowed thru to the mart for querying

On top of this you can overlay pre-calculated segments. For example - rather than try to pull say, all blue collar workers in the country, you can use segmentation to drill down only in those areas which are predominantly blue collar.

EDIT (following Joe's update)

In that case you might want a reporting mart in addition to the mart and repository I described above which is lean and mean and optimised for fast queries and MI reports.


Sure that's what an incremental map reduce is for. Essentially you perform an operation on the collection which processes the existing documents and puts them in a new collection, and as you add new documents you merge those rows into the derived collection.

  • MapReduce whilst undoubtedly a useful tool is not a silver bullet. The techniques described in the canonical MapReduce paper pre-date Google by several decades. The PL/I implementation I had to code-walk as a junior programmer was dated 1981!
    – Robbie Dee
    Commented Feb 24, 2014 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.