Plateforme Level Extreme
Abonnement
Profil corporatif
Produits & Services
Support
Légal
English
Why not 1*(0.5-0.4-0.1) = 0?
Message
De
06/01/2005 18:43:54
 
 
À
06/01/2005 18:35:01
Information générale
Forum:
Visual FoxPro
Catégorie:
COM/DCOM et OLE Automation
Versions des environnements
Database:
MS SQL Server
Divers
Thread ID:
00974881
Message ID:
00974892
Vues:
45
With ANY decimal numbers you will have some rounding errors. Sometimes the programs masks the error, and you get the result you want, and sometimes you get the wrong value, there's simply no way to avoid that. You can calculate with a million decimals and minimize the error, but you simply can not eliminate it. Everyone knows that (1/3)*3=1, but still all the calculators and programs I have seen shows 0.99999. In practice it's the same problem, it's only easier to see it.

>>Hi Cetin,
>>
>>There is no sulution as such. The problem is that NO decimal numbers has an accurate binary value, so it's always a matter of rounding. To get accurate calculations with decimals, you must multiply all the numbers by a 10, 100, 1000 or another power of 10, so that you only calculate the integers (whole numbers) which have accurate binary values. Then you must divede the result with the same factor. I am suprised you did not know that?
>>
>
>
>Is that what the fox is doing internally? because in vfp 6 it comes out to 0.
>
>
>
>
>
>>>A friend of mine asked this to me today. I said yes,0 with confidence and he said OK then go and put that in Excel as a formula.
>>>I thought it was an Excel bug but he turned and said it happens in SQL server with float datatype! I couldn't yet see that myself but trust his experience.
>>>Any idea and solution?
>>>TIA
>>>Cetin
Précédent
Suivant
Répondre
Fil
Voir

Click here to load this message in the networking platform