Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Whole numbers
Message
From
26/01/2000 06:58:06
 
 
To
26/01/2000 06:45:37
General information
Forum:
Visual FoxPro
Category:
Coding, syntax & commands
Title:
Miscellaneous
Thread ID:
00322702
Message ID:
00322705
Views:
18
>a=123456789012345678
>? a
>result is :
>123456789012345700
>why or how can i see the true result?

SET DECIMALS may help, but there is a limit to the precision of a number. Numbers of this magnitude are stored as floating point values and are intherently subject to rounding errors because they are not integer representations. You don't have 18 digits of precision, since this is going to be stored as a floating point number. And in all probability, and in all probability, unless you are computing astronomical distances, you don't have a real requirement of 18 digit precision. And if you do, you want a language with something bigger than the available float representations in VFP. The way the number is stored, it probably can't resolve the units in the least significant digit position accurately.

FORTRAN or one of the other scientific/mathematical languages, or LISP, would be far more appropriate for this type of precision, and would offer stronger numerical support.
EMail: EdR@edrauh.com
"See, the sun is going down..."
"No, the horizon is moving up!"
- Firesign Theater


NT and Win2K FAQ .. cWashington WSH/ADSI/WMI site
MS WSH site ........... WSH FAQ Site
Wrox Press .............. Win32 Scripting Journal
eSolutions Services, LLC

The Surgeon General has determined that prolonged exposure to the Windows Script Host may be addictive to laboratory mice and codemonkeys
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform