Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Problems with precision in numbers
Message
From
03/02/2000 13:57:54
 
 
To
03/02/2000 13:53:59
Dragan Nedeljkovich
Now officially retired
Zrenjanin, Serbia
General information
Forum:
Visual FoxPro
Category:
Other
Miscellaneous
Thread ID:
00326555
Message ID:
00326834
Views:
20
>>>I'm using:
>>>
>>>set decimals to 18
>>>?5000.3 + 0.6
>>>
>>>The result is: 5000.900000000001000000 instead of simply 5000.900000000000000000
>>>
>>>Someone could explain me why?
>>
>>Fractional numbers with the exception of currency, which uses a fixed precision BCD format,
>
>Wasn't the currency stored as a 8-byte integer, with decimal pointed shifted four places?

No, it's actually an 8 byte BCD format as I recall, with something like 15 or 16 places of precision. Same idea as BCD, though; Fixed size and decimal precision. If we really had it stored that way, we'd have convenient access to ULARGE structures.
EMail: EdR@edrauh.com
"See, the sun is going down..."
"No, the horizon is moving up!"
- Firesign Theater


NT and Win2K FAQ .. cWashington WSH/ADSI/WMI site
MS WSH site ........... WSH FAQ Site
Wrox Press .............. Win32 Scripting Journal
eSolutions Services, LLC

The Surgeon General has determined that prolonged exposure to the Windows Script Host may be addictive to laboratory mice and codemonkeys
Previous
Reply
Map
View

Click here to load this message in the networking platform