Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
INT() returns unexpected value
Message
 
To
14/09/2006 12:16:10
General information
Forum:
Visual FoxPro
Category:
Other
Miscellaneous
Thread ID:
01153755
Message ID:
01153880
Views:
15
Hi,

>Does anyone have any ideas why the first scenario the last digit is different?

You can write any number as a series of exponents. The number 13.4 is the same as 1*10^1 + 3*10^0 + 4*10^-1. Computers store numbers in a similar way, except that they use 2 as the base, rather then 10. The same number 13.4 in binary form looks like this:

? 2^3+2^2+2^0+2^-2+2^-3+2^-6+2^-7+2^-10+2^-11+2^-14+ ;
2^-15+2^-18+2^-19+2^-22+2^-23+2^-26++2^-27+2^-30+ ;
2^-31+2^-34+2^-35+2^-38+2^-39+2^-42+2^-43

or 1101.0110011001100110011001100110011001100110011

You probably noticed that the sequence 1100 repeats over and over. While 13.4 is a finite value in decimal notation, it's an infinite value in the binary system. There's no way to precisely store 13.4 as a binary value. Hence, you cannot store 13.4 on a computer, only its approximation. The number above is in decimal notation 13,399999999999980, which is close enough to 13.4 in most cases.

Since INT() just cuts off all digits following the comma, in cases like this you end up with a value one less than you expect. If you want to remove the fraction part of a decimal value, ROUND(...,0) is what you need, not INT().

I had an article in the November 2005 issue of FoxPro Advisor that explain such phenomens more in depth.
--
Christof
Previous
Reply
Map
View

Click here to load this message in the networking platform