Can python be efficiently implemented in big data field? To be precise I am building an web app that analyses really big data in medical health care field consisting of medical history and huge personal information. I need some advice on how to handle very big data in python efficiently and with high performance. Also are their some open source packages in python available which have high performance and efficiency in big data handling?

About users and data: Each user has about 3gb of data. Users are grouped based on their family and friend circles and the data is then analysed to predict important information and co-relations out of that. Currently I am talking about 10,000 users and will be rapidly increasing number of users.

  • 1
    About how much data you have to process at one are you talking? What kind of operations?
    – johannes
    Commented Jan 6, 2013 at 16:48
  • Each user has about 3gb of data. Users are grouped based on their family and friend circles and the data is then analysed to predict important information and co-relations out of that. Currently I am talking about 10,000 users and will be rapidly increasing number of users.
    – Akshay
    Commented Jan 6, 2013 at 19:19
  • shop.oreilly.com/product/0636920023784.do - O'Reilly recently released this book. Might help you out.
    – user28988
    Commented Jan 6, 2013 at 19:24
  • I have seen python be used in a custom written run-time that would parallelize the code on its own and run it on clusters for huge batch processing. This happens :).
    – MrFox
    Commented Jan 7, 2013 at 18:15

3 Answers 3


That is a very vague question, there is no canon definition for what constitutes big data. From a development point of view the only thing that truly changes how you need to handle data is if you have so much that you can't fit it all in memory at once.

How much of a problem that is depend greatly on what you need to do with the data, for most jobs you can do a single pass scheme where you load a block of data, do whatever needs to done with it, unload it, and go on to the next.

Sometimes issues can be solved by doing an organization pass, first going through data organizing it into chunks of data that need to be handled together, then going through each chunk.

If that strategy doesn't fit your task you can still get a long way with OS handled disk swapping, handle the data in blocks as far as possible, but if you need a little arbitrary access here and there it is still going to work.

And of course an always excellent strategy when dealing with a lot of data is to dwarf it by hardware. You can get 64 GB of memory in 16 GB blocks for $500, if you are working with that much data it is an easily justified investment. Some good SSDs is a nobrainer.

Specific case:

A big part of this job is definitely to reduce those 3GB data per person. It is often a bit of an art in its own right to find out what can be thrown away, but given the amount I must presume that you have got a fair amount of bulk measurements, in general you should first find patterns and aggregations for those data, and then use those results for comparing persons to one another. The majority of your raw data is either noise, repetition or irrelevant, you have got to cut that away.

This reduction process is easily suitable for a cluster as you can just give each process its own pile of persons.

The processing thereafter is a bit more tricky, what is optimal depend on a lot of factors, and you will probably have to do some trial and error. If you can make it fit the job try to load select pieces of data from all persons on the same computer and compare those, do the same with other pieces of data on other computers. Use those results as new data sets etc.

  • I have added some details about user data in the question. Please see them for further details. Also thanks for your valuable remark but I believe quite intensive computing methods like parrallel computing and map-reduce would be necessary to use because of steep growth in data and computation with time.
    – Akshay
    Commented Jan 6, 2013 at 19:23
  • @AkshayBhardwaj There are people who think they have a lot of data when there is 1000 rows in their database. How would I before your edit know that I have run into one of those who actually mean it? ;-) Commented Jan 6, 2013 at 20:39

It depends on what you want from your handling of big data. This concept is relatively vague. For example, if you're talking about MapReduce jobs across disparate data sources, then you may be interested in using Hadoop Streaming with the Dumbo library. If you're talking about statistical analysis, then NumPy and SciPy (as mentioned by Akira71) are interesting, as well as pandas (a data analysis toolkit). If you want graphing, look into matplotlib.

However, if you're talking about the storage and querying of big data, Python is not your best bet. You will want something like the Hadoop ecosystem to make this perform well, perhaps with layers on top for querying and building intermediate data sets. One project that really interests me is Spark; you may want to look at it as well. Unfortunately, this type of application framework does not play to Python's strengths.

  • Well Hadoop seems pretty solid option, but now I am confused whether using python with hadoop (inherently developed in Java) will severely effect performance of computation! I would be thankful if you can lay some light on it. Also is their any map reduce application inherently built for python?
    – Akshay
    Commented Jan 6, 2013 at 17:50
  • 2
    Hadoop can interact via the streaming interface with any language that can consume stdin and write to stdout. It is not as fast as raw Java, but it works well for many use cases. MapReduce as a pattern can be implemented in Python; any parallelization framework can really perform the concept. However, there are many technical concerns that take it out of the realm of programming language libraries: distribution of input chunks, local data optimization, and so on. MapReduce in practice is bigger than a simple library can provide.
    – asthasr
    Commented Jan 6, 2013 at 19:23

Python is used extensively in the big data field. There are a couple of packages that tend to get used quite a bit and they are probably the main reason Python has made inroads so deeply into big data:

  • NumPy - The Fundamental package for scientific computing in Python
  • SciPy - Mathematics, Scientific and Engineering package

Given that they are both open source and the popularity and ease of learning Python has pretty much catapulted it's use in Academia. This in turn has caused it to be used more and more outside academia and in larger companies or when students move into work roles they bring these packages with them.

These are very good packages and I had dabbled with them in a few projects. I have not used Python enough in Big Data projects to answer your ancillary question on how to handle Big Data efficiently with Python.

  • It seems the OP doesn't exactly want to do advanced analytics, which are what the packages you mention are used for.
    – S4M
    Commented Jan 6, 2013 at 15:32
  • @S4M He didn't specifically state that when I answered. I am sorry. However, Big Data typically is analysis & statistical analytics that those packages are for. It is what I used them for in our BI package.
    – Akira71
    Commented Jan 6, 2013 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.