Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Inaccurate results when using large numbers
Message
From
13/06/2002 09:49:05
Dragan Nedeljkovich (Online)
Now officially retired
Zrenjanin, Serbia
 
 
To
12/06/2002 01:28:34
Paul Dorrington
Dorrodata Computing Pty Ltd
Carlton, Australia
General information
Forum:
Visual FoxPro
Category:
Coding, syntax & commands
Miscellaneous
Thread ID:
00667226
Message ID:
00667915
Views:
20
>In our VFP application to support Unit Trusts (Mutual Funds) we have the occassional need to record reasonably large transactions and we note that sometimes VFP appears to get it wrong in the sixth decimal place. (We are required to record 6 decimal places for some clients.)
>
>Have other people experienced this problem and are there any known work arounds other than the obvious one of not using the VFP ROUND() function.
>
>For example:
>
>Buy amount = $800,000,000.00
>Unit Price = 0.938208
>Units issued = 800,000,000.00 / 0.938208 = 852,689,382.311811
>
>In VFP command window 800000000 / 0.9382080 = 852,689,382.311811
>but ROUND(800000000 / 0.9382080,6) = 852,689,382.311812
>and ROUND(800000000 / 0.9382080,7) = 852,689,382.3118114
>
>In Excel 800000000 / 0.9382080 = 852,689,382.311811
>and ROUND(800000000 / 0.9382080,6) = 852,689,382.311811
>
>Any suggestions would be much appreciated.

You're hitting the limit of VFP's numeric precision here. From what I remember of the trillion percent inflation we had nine years ago, it's using some IEEE format which is precise to 15.5 significant digits, which means that you are using the 15 already, and the 16th may be off by 1 (which is the probable reason it rounds the 4 up instead of down).

So if you really need this precision, go for Hilmar's solution, or do your calculations using Excel automation.

back to same old

the first online autobiography, unfinished by design
What, me reckless? I'm full of recks!
Balkans, eh? Count them.
Previous
Reply
Map
View

Click here to load this message in the networking platform