Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Rounding problem?
Message
From
21/03/2000 05:28:16
Paul Frost
Instem Computer Systems Ltd
Stone, United Kingdom
 
General information
Forum:
Visual FoxPro
Category:
Coding, syntax & commands
Miscellaneous
Thread ID:
00348122
Message ID:
00348234
Views:
28
Seems like the problem shows itself when the total number of trailing decimal zeroes in all numbers of the calculation exceeds 8.

So round( .3500000000 * 943 * .1, 2 ) = 33.01
round( .3500000000 * 943.0 * .1, 2 ) = 33.00
round( .350000000 * 943.0 * .1, 2 ) = 33.01
round( .35 * 943.00000000 * .1 ) = 33.01
round( .35 * 943.00000000 * .10 ) = 33.00

Playing around further while writing this, I've found the following :

.35 * 943 * .1 = 33.005
.350 * 943 * .1 = 33.0050
.3500 * 943 * .1 = 33.00500

It seems that the precision of the result of the calculation is based on the precision of the operands. It maybe that the value transferred to the function may not retain the required precision.

Another one to try is
.350000000000 * 943 * .2 = 66.0100000000000
.3500000000000 * 943 * .2 = 66.0099999999999

The extra information may not be of any help as to why the problem occurs, but it may help to avoid it.
Paul
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform