Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
INT() returns unexpected value
Message
From
14/09/2006 17:23:32
 
 
To
14/09/2006 12:16:10
General information
Forum:
Visual FoxPro
Category:
Other
Miscellaneous
Thread ID:
01153755
Message ID:
01153877
Views:
15
>Just wondering if anyone run into the following problem:
>
>INT(1333365.89000*100) return 133336588 this should be 133336589
>INT(333365.89000*100) return 33336589 expected
>INT(11333365.89000*100) return 1133336589 expected
>
>Does anyone have any ideas why the first scenario the last digit is different?
>
>Thanks !

vfp store data with IEEE floating point, on this format some values don't exists;
1333365.89 is an example.
? 1333365.89-1333365.0000000000000
then, INT() return correct value.

You can use ROUND() before INT(), but it have bugs, and return wrong values on some values.

Some other pretty result can happen:
CLEAR

* correct is .F.
? 1333365.89000000001396970 < 1333365.890000000013969690
Previous
Reply
Map
View

Click here to load this message in the networking platform