Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Inaccurate results when using large numbers
Message
From
12/06/2002 01:28:34
Paul Dorrington
Dorrodata Computing Pty Ltd
Carlton, Australia
 
 
To
All
General information
Forum:
Visual FoxPro
Category:
Coding, syntax & commands
Title:
Inaccurate results when using large numbers
Miscellaneous
Thread ID:
00667226
Message ID:
00667226
Views:
66
In our VFP application to support Unit Trusts (Mutual Funds) we have the occassional need to record reasonably large transactions and we note that sometimes VFP appears to get it wrong in the sixth decimal place. (We are required to record 6 decimal places for some clients.)

Have other people experienced this problem and are there any known work arounds other than the obvious one of not using the VFP ROUND() function.

For example:

Buy amount = $800,000,000.00
Unit Price = 0.938208
Units issued = 800,000,000.00 / 0.938208 = 852,689,382.311811

In VFP command window 800000000 / 0.9382080 = 852,689,382.311811
but ROUND(800000000 / 0.9382080,6) = 852,689,382.311812
and ROUND(800000000 / 0.9382080,7) = 852,689,382.3118114

In Excel 800000000 / 0.9382080 = 852,689,382.311811
and ROUND(800000000 / 0.9382080,6) = 852,689,382.311811

Any suggestions would be much appreciated.

Thanks
Paul Dorrington
(Melbourne, Australia)
Next
Reply
Map
View

Click here to load this message in the networking platform