Plateforme Level Extreme
Abonnement
Profil corporatif
Produits & Services
Support
Légal
English
Inaccurate results when using large numbers
Message
De
12/06/2002 18:21:22
Paul Dorrington
Dorrodata Computing Pty Ltd
Carlton, Australie
 
Information générale
Forum:
Visual FoxPro
Catégorie:
Codage, syntaxe et commandes
Divers
Thread ID:
00667226
Message ID:
00667694
Vues:
19
>Paul,
>
>I did this and it seems to work.
>
>
>SET DECIMALS TO 6
>? (800000000 / 9382080) * 10000000
>
>
>It gave a result of 852,689,382.311811
>
>It is not clear which result you want to get.

Jim,

Thanks for your reply. Sorry if my message was unclear.

The correct answer is 852,689,382.311811 and VFP gets this no matter what the SET('DECIMALS') setting. But if I use the VFP ROUND() function as follows, I get inconsistent results.

ROUND(800000000 / 0.938208, 6) = 852,689,382.311812 [incorrect]
ROUND((800000000 / 938208)*1000000, 6) = 852,689,382.311812 [incorrect]
ROUND((800000000 / 9382080)*10000000, 6) = 852,689,382.311811 [correct]

FYI, the reply by Hilmar Zonneveld provides a solution.

Regards
Paul
Précédent
Répondre
Fil
Voir

Click here to load this message in the networking platform