Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Why not 1*(0.5-0.4-0.1) = 0?
Message
From
07/01/2005 08:24:56
Cetin Basoz
Engineerica Inc.
Izmir, Turkey
 
 
To
07/01/2005 04:15:42
General information
Forum:
Visual FoxPro
Category:
COM/DCOM and OLE Automation
Environment versions
Database:
MS SQL Server
Miscellaneous
Thread ID:
00974881
Message ID:
00975038
Views:
36
>>>>A friend of mine asked this to me today. I said yes,0 with confidence and he said OK then go and put that in Excel as a formula.
>>>>I thought it was an Excel bug but he turned and said it happens in SQL server with float datatype! I couldn't yet see that myself but trust his experience.
>>>>Any idea and solution?
>>>>TIA
>>>
>>>Cetin,
>>>
>>>How're doing? We haven't communicated in a long time.
>>>
>>>I just have a SWAG, but I think it might be right. I tested this will both Excel and VFP 9.0 and got completely different results. IOW, Excel returned false and VFP, correctly, returned true.
>>>
>>>The only thing that I can think of that might explain it is the Pentium class process floating point bug.
>>>
>>>In Excel, when I entered the equation, 1*(0.5-0.4-0.1), it returned -2.77556E-17. Naturally, this isn't equal to zero.
>>>
>>>Sorry, Cetin, that's all I've got. BTW, Congrats on being a "UT Personality".:-)
>>
>>Hi George,
>>I'm fine and busy all day:)
>>I initially thought it as common roundation error but like you tested with VFP anyway. VFP5-8 returns it as a 0, C#, excel as that -2.7...E-17 value. Interestingly there are smaller numbers in float range close to 0.
>>
>>Thanks:)
>>Cetin
>
>Hi Cetin,
>
>This might explain to why it is happening. As you see the value is -2.7..E-17 is because a floating point of Double data-type has maximum limit of 8-bytes = 64 bits = 16 digits only. In the old version of C, there is a Long Double (= max FPU size = 10 bytes), they drop this capability in the newer version of C/C++ (I don't know exactly what version it was dropped). In the new version the Long Double has been Casting into Double only. So, you can only work around this 64-bits limitation.
>
>So, if you type this in excel: =ROUND( 1*(0.5 - 0.4 - 0.1), 16 ) it will give you 0.
>I think VFP already use this workaround internally.
>
>Now, this one is interesting. Because of that limitation, if you type a number (with no decimal point) longer than 16 digits both VFP and Excel will return different results and they both incorrect <g>.
>Try this:
>
>x = 45036172356878295
>?x
>x = 15036172356878295
>?x
>x=117346456767843354
>?x
>
>
>Btw, CONGRATULATIONS for you :-))
>Regards

Yes I read that C double thing but couldn't dare to tell it to my friend who is a C veteran:)
Suggested money type already.
Cetin
Çetin Basöz

The way to Go
Flutter - For mobile, web and desktop.
World's most advanced open source relational database.
.Net for foxheads - Blog (main)
FoxSharp - Blog (mirror)
Welcome to FoxyClasses

LinqPad - C#,VB,F#,SQL,eSQL ... scratchpad
Previous
Reply
Map
View

Click here to load this message in the networking platform