5

I am working with quite large files (pytables) and I am having problems with the Memory Error when I try to load the data for processing.

I would like some tips about how to avoid this in my python 32bits, since I am new working with pandas and pytables, and I do not know how to work splitting the data in small pieces.

My concern also comes when, if I get to split the data, how to calculate statistics like mean, std, etc without having the entire list or array, etc.

This is a sample of the code that I am using now, this works fine with small tables:

def getPageStats(pathToH5, pages, versions, sheets):

    with openFile(pathToH5, 'r') as f:
        tab = f.getNode("/pageTable")

        dversions = dict((i, None) for i in versions)
        dsheets = dict((i, None) for i in sheets)
        dpages = dict((i, None) for i in pages)


        df = pd.DataFrame([[row['page'],row['index0'], row['value0'] ] for row in tab.where('(firstVersion == 0) & (ok == 1)') if  row['version'] in dversions and row['sheetNum'] in dsheets and row['pages'] in dpages ], columns=['page','index0', 'value0'])        
        df2 = pd.DataFrame([[row['page'],row['index1'], row['value1'] ] for row in tab.where('(firstVersion == 1) & (ok == 1)') if  row['version'] in dversions and row['sheetNum'] in dsheets and row['pages'] in dpages], columns=['page','index1', 'value1'])        

        for i in dpages:


            m10 = df.loc[df['page']==i]['index0'].mean()
            s10 = df.loc[df['page']==i]['index0'].std()

            m20 = df.loc[df['page']==i]['value0'].mean()
            s20 = df.loc[df['page']==i]['value0'].std()

            m11 = df2.loc[df2['page']==i]['index1'].mean()
            s11 = df2.loc[df2['page']==i]['index1'].std()

            m21 = df2.loc[df2['page']==i]['value1'].mean()
            s21 = df2.loc[df2['page']==i]['value1'].std()

            yield (i,m10, s10), (i,m11, s11), (i,m20,s20), (i,m21,s21)) 

As you can see, I am loading all the necessary data into a Pandas DataFrame to procoess it, just the mean and the std by now.

This is quite fast, but for a pytable with 22Millions of rows I get Memory Error

3
  • How much memory does a given row take? How much memory is allocated to the process?
    – user40980
    Jun 18, 2014 at 19:18
  • Thanks for your answer Michael, I do not know exactly how much memo takes each row, but its just a float32 value in the filed of the row that I am interested in. And, about how much memo allocated, I do not know how to check this, sorry I am quite inexpert.
    – codeKiller
    Jun 19, 2014 at 6:02
  • Might be worth adding the 'pandas' tag to this. Nov 20, 2015 at 16:00

2 Answers 2

6

As far as I know Pandas is not the best tool if you can not store everything in the memory.

Additionally you are creating some extra data that you might try to avoid. I'm talking about the list comprehensions.

For once they are a bit too big/complex to be a list comprehension as for me.

Secondly due to the its nature for short period of time your are holding too much data: the complete list + its copy representation within the DataFrame. With the second assignment (df2) you are holding a new DF, the list and df. Not to mention all other object you have already created.

  1. Try to use a generator instead of a list comprehension. Either in line by replacing [...] with (...) or a proper generator that will be more readable too.

  2. (When solution 1 is not enough) Drop Pandas and do calculations manually. That might require you to iterate over the data twice, but will get you the result.

0
0

Have you tried using groupby and merge? I think that will significantly reduce your code and memory overhead. Here is an attempt to rewrite your code. I haven't tested, but you should get the idea.

def getPageStats(pathToH5, pages, versions, sheets): with openFile(pathToH5, 'r') as f: tab = f.getNode("/pageTable") # Avoid looping over rows, the following is way more efficient d = pd.DataFrame({ 'version': dversions, 'sheetNum': dsheets, 'pages': dpages }) d_fv0_ok1 = tab.where('(firstVersion == 0) & (ok == 1)') # Join on the two dataframe to obtain the rows that you are # looking for df = pd.merge(left=tab, right=d_fv0_ok1, on=['version', 'sheetNum', 'pages'], how='inner', copy=False) dfg = df.groupby('page').agg(['mean', 'std']) ## dfg is a multi index dataframe. # You can access them by dfg.loc[pageid, ('index0', 'mean')], etc. return dfg.join(d.pages, how='inner')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.