Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Problems with precision in numbers
Message
General information
Forum:
Visual FoxPro
Category:
Other
Miscellaneous
Thread ID:
00326555
Message ID:
00326744
Views:
19
>I'm using:
>
>set decimals to 18
>?5000.3 + 0.6
>
>The result is: 5000.900000000001000000 instead of simply 5000.900000000000000000
>
>Someone could explain me why?

Fractional numbers with the exception of currency, which uses a fixed precision BCD format, are stored as floating point numbers, which do not correspond precisely to fixed decimal representation. If you have very large numbers (on the order of 10^18 for example) the precision of floating point numbers won't allow you to discern the difference between two numbers where the units (the digit immediately to the left of the decimal place) is different. try the following to see how floating point representation is not adequate for very large numbers, or for extreme precsion:
a = 123456789012345678
for j = 1 to 100
   if a # a + j
      exit
   endif
endfor
? j
If you need extreme, extended precision, you need to use another language with a data type that can handle such extremes of precision - languages that have very large BCD representations, or use more precise floating point formats such as C, FORTRAN and COBOL (the PACKED DECIMAL representations can handle extreme precision if you make the fields big enough) are better choices.
EMail: EdR@edrauh.com
"See, the sun is going down..."
"No, the horizon is moving up!"
- Firesign Theater


NT and Win2K FAQ .. cWashington WSH/ADSI/WMI site
MS WSH site ........... WSH FAQ Site
Wrox Press .............. Win32 Scripting Journal
eSolutions Services, LLC

The Surgeon General has determined that prolonged exposure to the Windows Script Host may be addictive to laboratory mice and codemonkeys
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform