Plateforme Level Extreme
Abonnement
Profil corporatif
Produits & Services
Support
Légal
English
Why not 1*(0.5-0.4-0.1) = 0?
Message
De
06/01/2005 23:22:20
Dragan Nedeljkovich (En ligne)
Now officially retired
Zrenjanin, Serbia
 
 
À
06/01/2005 18:43:54
Information générale
Forum:
Visual FoxPro
Catégorie:
COM/DCOM et OLE Automation
Versions des environnements
Database:
MS SQL Server
Divers
Thread ID:
00974881
Message ID:
00974953
Vues:
35
>With ANY decimal numbers you will have some rounding errors. Sometimes the programs masks the error, and you get the result you want, and sometimes you get the wrong value, there's simply no way to avoid that. You can calculate with a million decimals and minimize the error, but you simply can not eliminate it. Everyone knows that (1/3)*3=1, but still all the calculators and programs I have seen shows 0.99999. In practice it's the same problem, it's only easier to see it.

Fox seems to have some internal tricks in place, probably rounding to the 15th significant digit... I've tried
(1/3+1/3+1/3-1)*1e46
with the last number varying between 1e9 and 1e46 arbitrarily, and the result was consistently 0.00, even when I set decimals to 12.

My favorite trick with fixed (non-float) calculators was to take square root of 8 and square that. The difference was amazing - sometimes I'd get two last digits different from 9. Of course, never got an eight, not in fixed-point.

back to same old

the first online autobiography, unfinished by design
What, me reckless? I'm full of recks!
Balkans, eh? Count them.
Précédent
Répondre
Fil
Voir

Click here to load this message in the networking platform