Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Rounding problem?
Message
 
To
21/03/2000 09:34:16
Paul Frost
Instem Computer Systems Ltd
Stone, United Kingdom
General information
Forum:
Visual FoxPro
Category:
Coding, syntax & commands
Miscellaneous
Thread ID:
00348122
Message ID:
00348357
Views:
21
>>>Yeah, I found that out too. My solution is to ROUND my result to 5, then ROUND to whatever I really want, like 2. That seems to work, but sounds buggy to me!
>>
>>Janet,
>>
>>It has something to do with the literals you're using. As was mentioned elsewheres, modifying the literals by removing the unnecessary zeroes, produces the correct (and expected) result. If, for example, you were to assign the result of the math to a variable and round the variable, it works correct. Further, I wouldn't think that, in actual usage, you'd use this anyway. If you needed 33.01, then that's what you'd use.
>
>I know what you're saying here makes sense, & as a mathematician & engineer, I would agree there is a difference in meaning between 0.35 & 0.3500000. But as a database programmer, I would expect no difference in value between the two.
>
>It's just another symptom of the inaccuracies inherent in floating point calculations on digital computers, & highlights the need to know how the language of choice represents such numbers.
>
>There are still inconsistencies in operation though, try
>
>r = 0.35 * 943 * .1
>r1 = 0.350000000000 * 943 * .1
>? r = r1 (displayed result = .T.)
>? round( r, 2 ) = round( r1, 2 ) (displayed result = .F.)
>
>so the numeric results are equal, but rounding them leads to a difference ! Surely a bug ?
>
Paul,

The question isn't whether or not it's a bug, but where it's coming from. VFP? or FPU?
George

Ubi caritas et amor, deus ibi est
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform