Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Decimal bug
Message
From
13/03/2003 13:13:54
Todd Cottengim
Alpine Land Information Services
Redding, California, United States
 
 
To
All
General information
Forum:
Visual FoxPro
Category:
Troubleshooting
Title:
Decimal bug
Miscellaneous
Thread ID:
00765485
Message ID:
00765485
Views:
60
I have come across a problem in VFP 6. Maybe its just my machine but I thought I would post it.

Given two values nX and nY where nX = 40 and nY = 10, compute the following: INT(nX / nY)

With SET DECIMALS at anything less than or equal to 10, the answer is correct (4). With SET DECIMALS at 11 or higher, I get the result 3.99999999996 with the 6 repeating based on number of decimals.

My computer:
Pentium 4 2.4Ghz with 512M memory
Nvidia G-Force4 Ti 4600 video card
VFP 6 - SP5

Too weird for me!
Anybody have any thoughts on this

Todd Cottengim
Alpine Land Information Services
Anderson, CA
Next
Reply
Map
View

Click here to load this message in the networking platform