# Questions tagged [numeric-precision]

The tag has no usage guidance.

32 questions
Filter by
Sorted by
Tagged with
77 views

### What is the proper way to deal with rounding error?

I am currently working on a Physics Engine in C++. The Physics Engine is for a sandbox video game. I want to build a powerful physics engine, so therefore, it needs to be very precise. I have ...
441 views

### Is the sum or subtraction of two positive machine numbers exact?

If you have two positive numbers that can be represented exactly by the finite arithmetic of your computer, that is, two machine numbers, if you perform their sum or subtraction with perfect ...
56k views

### What causes floating point rounding errors?

I am aware that floating point arithmetic has precision problems. I usually overcome them by switching to a fixed decimal representation of the number, or simply by neglecting the error. However, I ...
292 views

### Which layer should have responsibility for rounding numbers?

I am developing a financial system and want to have a defined policy for rounding monetary values. Given the following layers: View API Entity Model Persistence If I am passing a monetary value ...
6k views

### Why do some languages round to the nearest EVEN integer?

Programming languages like Scheme (R5RS) and Python (see this Question) round towards the nearest even integer when value is exactly between the surrounding integers. What is the reasoning behind ...
490 views

### Unit test a generic floating point equality function

I've written a function that tests two floating point numbers for approximate equality (see this Code Review question). I'd like to unit test my function, but I'm not positive of the best way to do ...
695 views

### Is new object creation necessary with BigDecimal?

Problem description: I am running performance test of calculation of PI number with primitives and with BigDecimal class. Calculation of PI with primitives is around 5-6 times faster than the same ...
11k views

### Handling extremely large numbers in a language which can't?

I'm trying to think about how I would go about doing calculations on extremely large numbers (to infinitum - intergers no floats) if the language construct is incapable of handling numbers larger than ...
17k views

### Solutions for floating point rounding errors

In building an application that deals with a lot of mathematical calculations, I have encountered the problem that certain numbers cause rounding errors. While I understand that floating point is ...
1k views

### How can a computer round the last digit in a floating point representation?

I'm confused by how a computer rounds off the last digit in the floating point representation. For example, I'm told that x=1.24327789 is stored in a computer with a 6-digit capacity then it;s ...
58 views

### Historic (and prehistoric) dates in Ruby [closed]

In Ruby, is there a standard class or gem to work with dates that can be - but are not required to be - historical in nature? The dates have precision (to the nearest million years, decade, century, ...
321 views

### Floating point accuracy and conversions

I am trying to build a class where user can compute floating operations with a particular accuracy. For example, if the user wants to have up to 4 digit accuracy, everything less than 0.0001 is zero ...
243 views

### Floating point absorption phenomena and ULP

Numerical analysis text books talks about the absorption phenomena (Introduction to Numerical Analysis and Scientific Computing; Nabil Nassif; Dolly Khoueiri Fayyad; CRC Press; 2014) when adding ...
16k views

### Why SQL numeric precision is specified as the number of digits?

I understand that this might be a loaded question. For example, in Sql Server 2008, doing DECLARE @someNum NUMERIC(10); means that the range of the variable will be limited by 10 digits. In other ...
11k views

### Why do we still use floats? [duplicate]

I understand why floats served a purpose in the past. And I think I can see why they're useful in some simulation examples today. But I think those example are more exceptional than common. So I ...
57 views

### multiplication with insufficiently long registers [duplicate]

I want to multiply two numbers but since I don’t have long enough registers in the current architecture need to “break them down” into shorter ones and somehow perform the calculation. For example: ...
1k views

### Is order of arguments in an arithmetic expression important to achieve as most exact result as possible (speed is not necessary)?

Actually, in this question I do not ask about particular language or architecture, however I understand there might be some differences. In physics / engineering it is usually better to handle ...
330 views

### BCMath vs MySQL: performance

Scenario I am working on a web application that handles financial transactions, and my code will need to be reusable for everything from consumer carts to manufacturer/distributor shipments. The ...
350 views

### Avoid numeric errors in Naive Bayes method

I am implementing the Naive Bayes method with Gaussian distribution. The problem that I have is that the variance used on the Gaussian curve (calculated from a training set) is REALLY small. They ...
21k views

### How to handle large numbers? [duplicate]

Well, I've just started doing puzzles and it's so annoying to see puzzles which are easy to do but also need to handle very large numbers. That's the problem. Say I have to deal with numbers like 10 ^...
11k views

### Why are floating point numbers used often in Science/Engineering?

While investigating the accuracy of floating point numbers, I've seen in a few places a statement similar to "float and double are (designed for/used often in) engineering and scientific calculation"...
337 views

### Implied precision of floating-point functions

While reviewing another programmer's implementation of a function to calculate the normal distribution CDF, I made a suggestion to either replace the entire implementation with Python's built-in ...
235 views

### Difference between USP and IEEE rounding

I am trying to understand the difference between different rounding methods: Our application offers two different types of rounding: IEEE USP (GMP) rounding. http://www.usp.org/sites/default/files/...
342 views

### Trivially Sortable Encoding for Arbitrary-Precision Decimals

I'm looking for a reasonably(*) space-efficient way to encode arbitrary-precision decimals (e.g. BigDecimal), such that when sorting the bit-pattern of the encodings lexicographically, the numbers ...
4k views

### How would I change the precision of a variable in Python?

I'm working on a 2D-physics engine, and I need a certain variable to be precise only to the hundredths. Are there any methods for basically shaving off all of that unneeded precision? I have tried ...
27k views

### Is this a good way to compare two numbers?

If we have double numbers, let's say I want to see if some double parameter is equal to zero that is passed as double: public bool AlmostEqual(double x, double y) { double epsilon = Math.Max(Math....
2k views

### How to identify unstable floating point computations?

In numerics, it is very important to be able to identify unstable schemes and to improve their stability. How to identify unstable floating point computations? I am working on a very complex ...
1k views

### Why normalization improves numerical precision?

I was reading the following article: Polynomial interpolation of GPS satellite coordinates, Milan Horemuz and Johan Vium Andersson 2006 and it states the following: "The estimation procedure of the ...
6k views

### Addition of double's is NOT Equal to Sum of the double as a whole

I am aware of the floating point errors as I had gained some knowledge with my question asked here in SE Floating Point Errors. What I am finding it difficult to understand is, the output of the ...
533 views

### Arithmetic Coding vs. Machine Numeric Precision

When subdividing intervals for arithmetic coding, the worst case scenario is that the final interval will have size 2^(1-n), where n is the number of unique symbols you are encoding. This will reach ...