Floating Point Expectations
Eliot Moss
moss at cs.umass.edu
Sat Jun 2 04:40:00 AEST 1990
Just a minor followup to the (45.0 / 100.0) == .45 question:
While the integers 45.0 and 100.0 are undoubtedly represented exactly, .45 is
a repeating form in binary (just as, say, 1/7 is in decimal notation). One
*would* expect (50.0 / 100.0) == .50 since .50 *does* have an exact
representation in binary. In general, when writing floating point algorithms
one must demand equality only within a tolerance; choosing that tolerance is a
complex issue, and one can have relative and absolute tolerances. Hope this
adds a little to the original posters understanding .... Eliot
--
J. Eliot B. Moss, Assistant Professor
Department of Computer and Information Science
Lederle Graduate Research Center
University of Massachusetts
Amherst, MA 01003
(413) 545-4206; Moss at cs.umass.edu
More information about the Comp.unix.wizards
mailing list