General information
Category:
Coding, syntax & commands
Seems like the problem shows itself when the total number of trailing decimal zeroes in all numbers of the calculation exceeds 8.
So round( .3500000000 * 943 * .1, 2 ) = 33.01
round( .3500000000 * 943.0 * .1, 2 ) = 33.00
round( .350000000 * 943.0 * .1, 2 ) = 33.01
round( .35 * 943.00000000 * .1 ) = 33.01
round( .35 * 943.00000000 * .10 ) = 33.00
Playing around further while writing this, I've found the following :
.35 * 943 * .1 = 33.005
.350 * 943 * .1 = 33.0050
.3500 * 943 * .1 = 33.00500
It seems that the precision of the result of the calculation is based on the precision of the operands. It maybe that the value transferred to the function may not retain the required precision.
Another one to try is
.350000000000 * 943 * .2 = 66.0100000000000
.3500000000000 * 943 * .2 = 66.0099999999999
The extra information may not be of any help as to why the problem occurs, but it may help to avoid it.
Paul
Previous
Next
Reply
View the map of this thread
View the map of this thread starting from this message only
View all messages of this thread
View all messages of this thread starting from this message only