Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
What is a more accurate way to round numbers than Round(
Message
From
17/09/2003 17:09:51
 
 
To
17/09/2003 15:12:58
Hilmar Zonneveld
Independent Consultant
Cochabamba, Bolivia
General information
Forum:
Visual FoxPro
Category:
Coding, syntax & commands
Miscellaneous
Thread ID:
00830026
Message ID:
00830114
Views:
27
Hilmar,

All fields in the source table and resulting summary tables are N(18,6), and all variables check out to be type "N". NO currency data types to be found. Using VFP6 SP5.

I just find it puzzling that manually setting the variables to (supposedly) the same value as that returned by sum yields different results when using round() function. It must have something to do with the way that the numbers are internally calculated.
clear
Set decimals to 4
select EMISSIONUNITREPORT2
sum EMISBYAVG to m.lnTotalAVG
      *!* result = 58271.270050
select EMISSIONUNITREPORT3
sum EMISBYAVG to m.lnTotalAVG1
      *!* result = 58271.270050

?m.lnTotalAVG - m.lnTotalavg1
     *!* result = 0.000000

?Type("m.lnTotalAVG")
     *!* result = "N"

?Type("m.lnTotalAVG1")
     *!* result = "N"

?str(m.lnTotalAVG,30,18) 
     *!* result = "58271.270049999400000000"

?str(m.lnTotalAVG,30,18) 
      *!* result = 58271.270049999900000000

*!* this is how my test program was checking each total
? round(m.lnTotalAVG,4) = round(m.lnTotalAVG1,4)
      *!* result = .f.
? round(m.lnTotalAVG,4)
      *!* result = 58271.2700
? round(m.lnTotalAVG1,4)
      *!* result = 58271.2701  && Wrong!!


*!* now manually set to same values as returned by sum

m.lnTotalAVG = 58271.270049999400000000
?m.lnTotalAVG
     *!* result = 58271.270049999400000000

m.lnTotalAVG1 = 58271.270049999900000000
?m.lnTotalAVG1
      *!* result = 58271.270049999900000000

?Type("m.lnTotalAVG")
     *!* result = "N"

?Type("m.lnTotalAVG1")
     *!* result = "N"

? round(m.lnTotalAVG,4) = round(m.lnTotalAVG1,4)
      *!* result = .T. && ROUND WORKS AS EXPECTED HERE
? round(m.lnTotalAVG,4)
      *!* result = 58271.2700
? round(m.lnTotalAVG1,4)
      *!* result = 58271.2700
>Perhaps you can check whether any of the intermediate numbers are of type currency?
>
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform