Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Rounding problem?
Message
 
To
21/03/2000 10:30:47
Paul Frost
Instem Computer Systems Ltd
Stone, United Kingdom
General information
Forum:
Visual FoxPro
Category:
Coding, syntax & commands
Miscellaneous
Thread ID:
00348122
Message ID:
00348403
Views:
27
Paul,
>>
>>The question isn't whether or not it's a bug, but where it's coming from. VFP? or FPU?
>
>Without wanting to get into an argument here (I respect your views highly), as far as I am concerned it is a bug.

I appreciate the kind words, but we're not getting into an argument, just having a discussion. It's essential that in the course of such we respect each other's POVs.

>If I have 2 numbers which are numerically equal (according to the = operator) & I apply an operation such as rounding directly to those numbers, then it is reasonable to expect the result to be the same, according to the = operator. As a user it should not be my concern whether the source of the problem is VFP or FPU - I believe I have a reasonable cause to expect consistent results. And these results are not consistent.
>
>Between 2 different packages or languages, I might expect discrepancies & if working with the two together then I would expect to handle those dicrepancies myself. But working with a single package, any such discrepancies consitute a bug. I should need to know that = uses FPU & round VFP routines & that there are differences.
>
It has something to do with the number of decimal places that's being supplied. My hunch (and this isn't a SWAG it's more of a WAG) is that the number of decimals is exceeding the limit of 8 byte double precision floats. Remove any zero to the right of the decimal place and the result's correct.

Try this:

? TRANSFORM(.35000000 * 943.0 * .100, '###.###') && Correct
? TRANSFORM(.35000000 * 943.0 * .100, '###.###########') && Correct
? TRANSFORM(.35000000 * 943.0 * .100, '###.############') && Incorrect

That last one hits 12 decimals. I don't know if that's the limit of doubles or not. Someone more knowledgable in this area will have to answer that.

IAC, I don't really see this as a major issue in view of the fact that I can't fathom a reason to do this in this fashion in an application. You'd being using variables, and this doesn't exist there.
George

Ubi caritas et amor, deus ibi est
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform