Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Rounding of decimal places
Message
From
15/11/2003 07:34:44
 
 
To
27/10/2003 03:47:17
General information
Forum:
Visual FoxPro
Category:
Troubleshooting
Miscellaneous
Thread ID:
00842819
Message ID:
00850217
Views:
18
Hi Gregory,

-------------
I suspect that this has something to do with the internal precision.

the result of 3.900000000 * 55.15 is a bit less than 215.085


? 3.900000000 * 55.15 && 215.0850000000000

? (3.9 * 55.15-215.085)*10000 && -0.00000000028

The round() works ok.
-------------

You are sure ?

Run with attention this code, thanks :
CLEAR
? 'format to decimal 10'
SET DECIMALS TO 10
? 'literal round 2 is bad, round 3 is ok'
? 3.9=3.900000000,3.9*55.15=3.900000000*55.15,ROUND(3.9*55.15,2)=ROUND(3.900000000*55.15,2) ,ROUND(3.9*55.15,3)=ROUND(3.900000000*55.15,3)
a=55.15
y=3.900000000
x=3.9
rx=x*a
ry=y*a
? 'variables round 2 is bad, round 3 is ok'
? x=y, x*a=y*a ,ROUND(x*a,2)=ROUND(y*a,2),ROUND(x*a,1)=ROUND(y*a,1)
? rx=ry, ROUND(rx,2)=ROUND(ry,2) ,ROUND(rx,3)=ROUND(ry,3)

? 'format to decimal 11'

SET DECIMALS TO 11
? 'literal round 2 is ok, round 3 is ok'
? 3.9*55.15=3.900000000*55.15,ROUND(3.9*55.15,2)=ROUND(3.900000000*55.15,2) ,ROUND(3.9*55.15,3)=ROUND(3.900000000*55.15,3)

a=55.15
y=3.900000000
x=3.9
rx=x*a
ry=y*a
? 'variables round 2 is bad, round 3 is ok'
? x=y, x*a=y*a ,ROUND(x*a,2)=ROUND(y*a,2),ROUND(x*a,1)=ROUND(y*a,1)
? rx=ry, ROUND(rx,2)=ROUND(ry,2) ,ROUND(rx,3)=ROUND(ry,3)
On math rules, if Round(,n-1) are equals then Round(,n) are equal.

VFP uses floating point variables for do any numeric computation ( integer,numeric,float,date,
datetime ... ), and with SET DECIMALS TO ... try to solve all, but this is a illusion.

On VFP9 add list i have not see the top of the top item:
"Add to VFP correct functions for fixed decimal mathematic."

Fabio
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform