Why are floating point numbers used often in Science/Engineering?

Question

While investigating the accuracy of floating point numbers, I've seen in a few places a statement similar to

"float and double are (designed for/used often in) engineering and scientific calculation"

From my understanding, the strength of floats and doubles are the amount of memory they use for their (good, but not perfect) precision.

I feel like I'm almost getting an understanding from this answer

"floating point numbers let you model continuous quantities"

I still am not convinced I understand. Engineering and Science both sound like fields where you would want precise results from your calculations, which, from my understanding, floating points do not give. I'm also not sure I follow what a "continuous quantity" is, exactly.

Can somebody expand on this explanation and perhaps give an example?

Answer 1

Computation in science and engineering requires tradeoffs in precision, range, and speed. Fixed point arithmetic provides precision, and decent speed, but it sacrifices range. BigNum, arbitrary precision libraries, win on range and precision, but lose on speed.

The crux of the matter is that most scientific and engineering calculations need high speed, and huge range, but have relatively modest needs for precision. The most well determined physical constant is only known to about 13 digits, and many values are known with far less certainty. Having more than 13 digits of precision on the computer isn't going to help that. The fly in the ointment is that sequences of floating point operations can gradually lose precision. The bread and butter of numerical analysis is figuring out which problems are particularly susceptible to this, and figuring out clever ways of rearranging the sequence of operations to reduce the problem.

An exception to this is number theory in mathematics which needs to perform arithmetic operations on numbers with millions of digits but with absolute precision. Numerical number theorists often use BigNum libraries, and they put up with their calculations taking a long time.

Answer 2

What alternative do you propose?

Continuous quantities are represented using real numbers in mathematics. There is no data type that can encode every possible real number (because reals are uncountable), so that means we can only pick a subset of those real numbers that we're most interested in.

• You can pick all computable reals, which is similar to what computer algebra systems (CAS) do. The problem is that it becomes rapidly unfeasible as your expression tree grows larger and larger. It's also very slow: try solving a huge system of differential equations in Mathematica symbolically and compare against some other floating-point based implementation and you'll see a dramatic difference in speed. Additionally, as Jörg W Mittag and kasperd have pointed out: you don't even have decidable equality/comparison operations.

• You could use exact rational numbers, but that doesn't really work for many applications because you need to calculate square roots or cosines or logarithms etc. Furthermore, there is also a tendency for rationals to become increasingly complex and thus requiring more space to store and time to process as you perform more and more calculations on them.

• You could also use arbitrary-precision decimals, but then even something as simple as division won't work because you get infinitely repeating digits. You can also run into the issue of increasing complexity as you perform more similar to rational numbers, though to a lesser extent.

So you'd be forced to use approximations at some point, in which case that's exactly where floating-point numbers do best. Floating-point numbers are also of fixed width (unlike all the other 3 data types mentioned earlier), which prevents the complexity increase as you perform more and more calculations on them.

Answer 3

Your proposition about science is wrong, Engineering and Science other then Math don't work with exact precise results. They work with a precision factor which is built into how many digits you show.

The key term you need to understand here is: significant figures. The significant figures of a number are those digits that carry meaning contributing to its precision.

Which basically means if I state that something is 12 centimeters longs, it can actually be somewhere between 11,5 and 12,5 centimeters long. If however I state that something is 12,00 centimeters long it can be somewhere between 11,995 and 12,005 centimeters long.

Just as an illustration, if you take a measurement tape and measure your living room. Even though you may find that it 6 meters 25 centimeters wide, you know that your tape measurement wasn't accurate enough to tell anything about the millimeter-accuracy or nano-meter-accuracy.

Answer 4

Note that floating point numbers are basically the same as scientific and engineering notation, the standard way for humans to write numbers in math and science. In these fields, there's not a great need for extreme precision, but there's often a huge range.

To pick a random example from my physics homework, I recently had to work with the mass of an electron, which is roughly 9.11 * 10^-31 kg. I don't care a lot about the precision; it could easily be 9.12 for all I care. But I care about the exponent and don't want to have to write out 0.0000...911 kg, so I use scientific notation.

Similar reasoning applies in scientific and engineering computing: there's a huge range, but we don't want to have to store and work with very large numbers, so we store a normalized value and an exponent, which is smaller and faster to work with.

Answer 5

Floating-point numbers also have several properties that lend themselves well to computing certain types of scientific results. Most notably, precision is inversely proportional to magnitude, just like in scientific notation, so you can represent both small differences close to zero and larger differences much farther away.

Goldberg's paper is probably the most famous analysis of the properties of floating-point numbers (and should be required reading if you care about this sort of thing), but Kahan's papers I think do a better job of explaining the rationale behind many of the subtle design issues.

In particular, Kahan's diatribe about Java's implementation of floating point, while quite inflammatory, makes several good points about why IEEE-754 semantics are useful, and Much Ado About Nothing's Sign Bit explores the rationale for signed zero in considerable depth.

Answer 6

TL;DR We do not know how to compute most functions with perfect precision, there is therefore no point representing numbers with perfect precision.

All of the answers so far miss the most important point: we cannot compute exact values of most numbers. As an important special case, we cannot compute exact values of the exponential function — to cite only the most important irrational function.

Naive answer to the naive question

It seems your question is rather “there is exact arithmetic libraries, why don't we use them in place of floating point arithmetic?” The answer is that exact arithmetic works on rational numbers and that:

• Archimede's number — the pedantic name of π — is not rational.
• Many other important constants are not rational.
• Many other important constants are not even known to be rational or not.
• For any non-zero rational number x the number exp(x) is irrational.
• Similar statements hold for radicals, logarithms, and a wealth of functions important to scientists (Gauss's distribution, its CDF, Bessel functions, Euler functions, …).

The rational number is a lucky accident. Most numbers are not rational (see Baire's theorem) so computing on numbers will always bring us out of the rational world.

What is computing and representing a number?

We may react by saying “OK, the problem is that rational numbers were not such a great choice to represent real numbers.” Then we roll up our sleaves fork Debian and devise a new representation system for real numbers.

If we want to compute numbers we have to pick a representation system for real numbers and describe important operations on them — i.e. define what computing means. Since we are interested in scientific computing, we want to represent accurately all decimal numbers (our measures), their quotients (rational numbers), values of the exponential functions and some funny constants, like Archimede's number.

The problem is that the only way to perfectly represent numbers in such a system is to use symbolic form, that is, not to compute anything at all and work with algebraic expressions. This is a rather crippled representation of real numbers, because we cannot reliably compare two numbers (which one is greater)? We cannot even easily answer the question “Is the given number equal to 0?”.

If you look for more precise mathematical definitions and problems, look for rational numbers, transcendental numbers, best approximations, and Baire's theorem, for instance.

Answer 7

Because

1) The authors make the assumptions that "engineering and scientific calculation" measure real-world physical quantities

2) Physical quantities are continuous, and exactly as you state "floating point numbers let you model continuous quantities"

.. and the rest of my answer is summed up nicely by Rufflewind, so I'm not going to repeat that here.

Answer 8

Floating point numbers provide relative accuracy: they can represent numbers that are at most a small percentage (if you want to call something like 0.0000000000001% a percentage) away from any accurate number over a wide range of numbers. They share this trait with a slide rule, though the latter does not get better than something like 3 digits of accuracy. Still, it was quite sufficient for working out the static and dynamic forces of large structures before digital computers became commonplace for that, and that's because the material constants also show some variation, and picking constructs that are reasonably benign against material and construction differences will tend to make the maximum loads and weak points reasonably identifiable.

Now "accuracy" is a useful feature for many numbers representing measurements and/or magnitudes of physical properties.

Not everything in science/engineering belongs in that category. For example, if you are using number theoretic transforms for multiplying large numbers or Galois fields for manipulating error correction polynomials, there is no such thing as a small error: any single bit error during processing will lead to results that are quite indistinguishable from completely random noise.

Even in those areas one can work with floating point numbers (like using complex FFTs for doing convolution) if one keeps track of the accumulation of errors and makes sure that the floating point errors do not accumulate enough magnitude to possibly even flip a single bit in the actual entities that they are an approximations of. For such approximations, fixed point processing would likely be more appropriate but floating point units in the field tend to provide faster operation and a larger number of usable bits.

Also programming languages like C or Fortran make it surprisingly hard to access basic operations like mixed precision multiplication and division or a carry bit for addition/subtraction, and those are basic building blocks for going beyond limited precision integers.

So if you can map operations to floating point numbers, you tend to have reasonably powerful hardware at your disposal these days and you can reasonably well specify your algorithms in one of today's general purpose programming languages.

Answer 9

I think this can be answered by addressing what application float/double data types are not suitable for.

When you need to make sure that you can represent a number accurately with a specific number of digits, then floating point numbers are inappropriate, because they represent the numbers as powers of 2, instead of powers of 10, as is how we represent numbers in the real world.

So one domain where floating point data types should not be used is that of finance*. For the core system of e.g. a bank, it would be completely unacceptable if an amount that should have been $100000.01 suddenly becomes $100000.00 or $100000.02.

Such a problem could easily occur when using floats, especially if the number was the result of one or more calculations, e.g. calculating the sum of all transactions in an account.

Engineering and scientific calculation are domains where these relatively small rounding errors are acceptable. Users are normally aware that all numbers have a limited precision, and they often work with a number of significant digits. But most importantly they have a well defined relative precision, i.e. they have provide the same number of significant digits, both for very large numbers, and for very small numbers.

* I did once work on a financial application where floats had been used to represent values, and as a consequence, rounding errors were introduced. Fortunately, this specific bug wasn't critical at all, the users did complain about calculation errors in the program. And this led to a different, far worse, effect: the users started loosing faith in the system.