Plateforme Level Extreme
Abonnement
Profil corporatif
Produits & Services
Support
Légal
English
Rounding problem?
Message
De
21/03/2000 09:34:16
Paul Frost
Instem Computer Systems Ltd
Stone, Royaume Uni
 
Information générale
Forum:
Visual FoxPro
Catégorie:
Codage, syntaxe et commandes
Divers
Thread ID:
00348122
Message ID:
00348335
Vues:
21
>>Yeah, I found that out too. My solution is to ROUND my result to 5, then ROUND to whatever I really want, like 2. That seems to work, but sounds buggy to me!
>
>Janet,
>
>It has something to do with the literals you're using. As was mentioned elsewheres, modifying the literals by removing the unnecessary zeroes, produces the correct (and expected) result. If, for example, you were to assign the result of the math to a variable and round the variable, it works correct. Further, I wouldn't think that, in actual usage, you'd use this anyway. If you needed 33.01, then that's what you'd use.

I know what you're saying here makes sense, & as a mathematician & engineer, I would agree there is a difference in meaning between 0.35 & 0.3500000. But as a database programmer, I would expect no difference in value between the two.

It's just another symptom of the inaccuracies inherent in floating point calculations on digital computers, & highlights the need to know how the language of choice represents such numbers.

There are still inconsistencies in operation though, try

r = 0.35 * 943 * .1
r1 = 0.350000000000 * 943 * .1
? r = r1 (displayed result = .T.)
? round( r, 2 ) = round( r1, 2 ) (displayed result = .F.)

so the numeric results are equal, but rounding them leads to a difference ! Surely a bug ?

Paul
Précédent
Suivant
Répondre
Fil
Voir

Click here to load this message in the networking platform