Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Why not 1*(0.5-0.4-0.1) = 0?
Message
 
To
06/01/2005 19:04:56
Cetin Basoz
Engineerica Inc.
Izmir, Turkey
General information
Forum:
Visual FoxPro
Category:
COM/DCOM and OLE Automation
Environment versions
Database:
MS SQL Server
Miscellaneous
Thread ID:
00974881
Message ID:
00974913
Views:
30
>>>A friend of mine asked this to me today. I said yes,0 with confidence and he said OK then go and put that in Excel as a formula.
>>>I thought it was an Excel bug but he turned and said it happens in SQL server with float datatype! I couldn't yet see that myself but trust his experience.
>>>Any idea and solution?
>>>TIA
>>
>>Cetin,
>>
>>How're doing? We haven't communicated in a long time.
>>
>>I just have a SWAG, but I think it might be right. I tested this will both Excel and VFP 9.0 and got completely different results. IOW, Excel returned false and VFP, correctly, returned true.
>>
>>The only thing that I can think of that might explain it is the Pentium class process floating point bug.
>>
>>In Excel, when I entered the equation, 1*(0.5-0.4-0.1), it returned -2.77556E-17. Naturally, this isn't equal to zero.
>>
>>Sorry, Cetin, that's all I've got. BTW, Congrats on being a "UT Personality".:-)
>
>Hi George,
>I'm fine and busy all day:)

I expected as much.:-)

>I initially thought it as common roundation error but like you tested with VFP anyway. VFP5-8 returns it as a 0, C#, excel as that -2.7...E-17 value. Interestingly there are smaller numbers in float range close to 0.
>
>Thanks:)

You're welcome, but from you've said, it appears to be that .NET still has the same problems with rounding. Might be something to investigate.
George

Ubi caritas et amor, deus ibi est
Previous
Reply
Map
View

Click here to load this message in the networking platform