Questions tagged [numeric-precision]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
62
votes
7answers
56k views

What causes floating point rounding errors?

I am aware that floating point arithmetic has precision problems. I usually overcome them by switching to a fixed decimal representation of the number, or simply by neglecting the error. However, I ...
15
votes
4answers
11k views

Handling extremely large numbers in a language which can't?

I'm trying to think about how I would go about doing calculations on extremely large numbers (to infinitum - intergers no floats) if the language construct is incapable of handling numbers larger than ...
17
votes
5answers
17k views

Solutions for floating point rounding errors

In building an application that deals with a lot of mathematical calculations, I have encountered the problem that certain numbers cause rounding errors. While I understand that floating point is ...
3
votes
2answers
27k views

Is this a good way to compare two numbers?

If we have double numbers, let's say I want to see if some double parameter is equal to zero that is passed as double: public bool AlmostEqual(double x, double y) { double epsilon = Math.Max(Math....