0

i'm a data scientist and python noob, but I seem to have found a bug with scipy.stats.percentileofscore. it may be "by design" but please bear with me.

if you have an array, you will get an upper bound of 100, but never a lower bound of 0. The lowest percentile you will get is 100/len so if I have an array of 30 items the lowest percentile I'll get is 100/30 ==> (3)

so I have a workaround to allow a zero percentile result. is my workaround valid? is there something I"m doing wrong? please suggest.

also note the following i put the numbers in the 0.0 to 1.0 range (I don't like using 0 to 100 for percentages)

stochState.kSlopePercentile = scipy.stats.percentileofscore([stochHist.kSlope for stochHist in stochState.history],stochState.kSlope)  / 100.0
invHistLen= 1.0/len(stochState.history)
if(stochState.kSlopePercentile < 1.0):
    stochState.kSlopePercentile -= invHistLen * (1-stochState.kSlopePercentile+invHistLen)

the above code subtracts up to 1/len from the percentile (more the closer the percentile is to zero)

  • try kind=strict, or try a value that is smaller than all existing ones. As dan1111 said, there are many possible definitions, scipy.stats has four of them. – Josef Mar 25 '14 at 18:30
  • thanks, i didn't know the kind parameter existed. i think this is exactly what i need. – JasonS Mar 27 '14 at 3:52
1

I think this is likely by design. In my opinion, calling something the 0th percentile when you have only 30 values is dubious. If something is the bottom value out of 30, that doesn't give you confidence that it would still be the bottom value if you had a much larger population. And in my mind, this is what percentile would be used for--saying something about the likely distribution of the population you are trying to represent, not the 30 values themselves.

However, there is no standard "correct" way of calculating percentile. There are several different ways, and their differences will be most apparent when used on small samples. Wikipedia has a good overview.

If you are not happy with the algorithm Python uses, I would recommend finding a package that implements your preferred algorithm, or implementing your own, rather than hacking the results at the end. This hack is likely to be problematic.

  • I think you're right, that it is by design: docs.scipy.org/doc/scipy/reference/generated/… – Zeroth Mar 25 '14 at 16:34
  • I need normalized distributions of rank, thus having 100 but not 0 is an issue. anyway, thanks for the feedback. i'll leave my hack as is. the hack isn't very problematic, if read my implementation carefully. it just normalizes to 0 instead of 1/n – JasonS Mar 27 '14 at 3:45
  • looking at the link from Zeroth i see the kind parameter. that's what i needed. – JasonS Mar 27 '14 at 3:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.