Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
On VFP 4000 sums are not reliable!
Message
From
16/12/2005 06:44:09
 
 
To
16/12/2005 06:10:53
General information
Forum:
Visual FoxPro
Category:
Databases,Tables, Views, Indexing and SQL syntax
Environment versions
Visual FoxPro:
VFP 9
Database:
Visual FoxPro
Miscellaneous
Thread ID:
01078586
Message ID:
01078593
Views:
8
>this is not a bug, but a VFP feature by design
>
>CLEAR
>CLOSE TABLES ALL
>
>CREATE CURSOR table1 (id I, coeff1 I)
>INSERT INTO table1 VALUES (0,32767)
>INSERT INTO table1 VALUES (1,32767)
>
>CREATE CURSOR TEST (fk I,n N(9,6))
>
>FOR K=1 TO 4000
>	INSERT INTO TEST VALUES (m.k%2,0.100000)
>NEXT
>
>SELECT	SUM(n*coeff1) VFP,'13106800,000000' Exact FROM test JOIN TABLE1 ON FK=ID
>
>* of course ROUND() is useless
>SELECT	ROUND(SUM(ROUND(ROUND(n,6)*ROUND(coeff1,6),6)),6) VFP,'13106800,000000' Exact FROM test JOIN TABLE1 ON FK=ID
>
>
>this is a simple example, but it is possible to build a combination of numbers
>what it produces smaller values but however wrong.
>
>A workaround exists, but it is orrible

Fabio,

I think that this is due to the fact that all N(..) arithmetic is done in/by Double

You know that
- max precision of a double = 15.95 digits (log10(2^53))
- max precision of a Currency = log10(2^63) = 18.96 = 15 integer + 4 fractional
- the floating point exponent is a power of two, not 10

If you can do with max 4 digits after the decimal point, use Currency data type
CLEAR 
CLOSE TABLES ALL

CREATE CURSOR table1 (id I, coeff1 I)
INSERT INTO table1 VALUES (0,32767)
INSERT INTO table1 VALUES (1,32767)

CREATE CURSOR TEST (fk I,n Y)

FOR K=1 TO 4000
	INSERT INTO TEST VALUES (m.k%2, $0.1)
NEXT

SELECT	SUM(n*coeff1) VFP,'13106800,000000' Exact FROM test JOIN TABLE1 ON FK=ID
Gregory
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform