Plateforme Level Extreme
Abonnement
Profil corporatif
Produits & Services
Support
Légal
English
Why not 1*(0.5-0.4-0.1) = 0?
Message
De
06/01/2005 19:04:56
Cetin Basoz
Engineerica Inc.
Izmir, Turquie
 
Information générale
Forum:
Visual FoxPro
Catégorie:
COM/DCOM et OLE Automation
Versions des environnements
Database:
MS SQL Server
Divers
Thread ID:
00974881
Message ID:
00974904
Vues:
29
>>A friend of mine asked this to me today. I said yes,0 with confidence and he said OK then go and put that in Excel as a formula.
>>I thought it was an Excel bug but he turned and said it happens in SQL server with float datatype! I couldn't yet see that myself but trust his experience.
>>Any idea and solution?
>>TIA
>
>Cetin,
>
>How're doing? We haven't communicated in a long time.
>
>I just have a SWAG, but I think it might be right. I tested this will both Excel and VFP 9.0 and got completely different results. IOW, Excel returned false and VFP, correctly, returned true.
>
>The only thing that I can think of that might explain it is the Pentium class process floating point bug.
>
>In Excel, when I entered the equation, 1*(0.5-0.4-0.1), it returned -2.77556E-17. Naturally, this isn't equal to zero.
>
>Sorry, Cetin, that's all I've got. BTW, Congrats on being a "UT Personality".:-)

Hi George,
I'm fine and busy all day:)
I initially thought it as common roundation error but like you tested with VFP anyway. VFP5-8 returns it as a 0, C#, excel as that -2.7...E-17 value. Interestingly there are smaller numbers in float range close to 0.

Thanks:)
Cetin
Çetin Basöz

The way to Go
Flutter - For mobile, web and desktop.
World's most advanced open source relational database.
.Net for foxheads - Blog (main)
FoxSharp - Blog (mirror)
Welcome to FoxyClasses

LinqPad - C#,VB,F#,SQL,eSQL ... scratchpad
Précédent
Suivant
Répondre
Fil
Voir

Click here to load this message in the networking platform