Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Rounding of decimal places
Message
From
15/11/2003 08:34:42
 
 
To
15/11/2003 07:34:44
General information
Forum:
Visual FoxPro
Category:
Troubleshooting
Miscellaneous
Thread ID:
00842819
Message ID:
00850223
Views:
18
Hi Fabio,

What can I say ?

Apart from a double, vfp also keeps the number of decimal places for any numeric. I suspect that the number of decimal places is taken into account when arithmetic is done. Hence.

Have a look at the vfp api

Numeric
ev_type ‘N’
ev_width Display width
ev_length Decimal places
ev_real Double precision

look at my download which retrieves a 'SAVE TO' file. Watch the T_NUMERIC part carefully

excerpt
case inlist(VariableType, T_NUMERIC)
	declare	VOID	RtlMoveMemory in win32api Double @Dest, Char @Src, Size_T BytesToCopy
	&& 1		: 0a 00
	&& 1.1		: 0c 01
	&& 1.12		: 0d 02
	&& 1.123	: 0e 03
	&& 1.1234	: 0f 04
	&& 1/3		: 0d 02 depends on set decimals when division was done
	c  = fread_at(fd, FileOffset +32 + OffsetValue, 8)
	set decimals to (VariableDecimals)
	VariableValue  = 10^(-VariableDecimals) 
	VariableLength = 8
	=RtlMoveMemory(@VariableValue, @c,  VariableLength  )
>Hi Gregory,
>
>-------------
>I suspect that this has something to do with the internal precision.
>
>the result of 3.900000000 * 55.15 is a bit less than 215.085
>
>
>? 3.900000000 * 55.15 && 215.0850000000000
>
>? (3.9 * 55.15-215.085)*10000 && -0.00000000028
>
>The round() works ok.
>-------------
>
>You are sure ?
>
>Run with attention this code, thanks :
>
>
>CLEAR
>? 'format to decimal 10'
>SET DECIMALS TO 10
>? 'literal round 2 is bad, round 3 is ok'
>? 3.9=3.900000000,3.9*55.15=3.900000000*55.15,ROUND(3.9*55.15,2)=ROUND(3.900000000*55.15,2) ,ROUND(3.9*55.15,3)=ROUND(3.900000000*55.15,3)
>a=55.15
>y=3.900000000
>x=3.9
>rx=x*a
>ry=y*a
>? 'variables round 2 is bad, round 3 is ok'
>? x=y, x*a=y*a ,ROUND(x*a,2)=ROUND(y*a,2),ROUND(x*a,1)=ROUND(y*a,1)
>? rx=ry, ROUND(rx,2)=ROUND(ry,2) ,ROUND(rx,3)=ROUND(ry,3)
>
>? 'format to decimal 11'
>
>SET DECIMALS TO 11
>? 'literal round 2 is ok, round 3 is ok'
>? 3.9*55.15=3.900000000*55.15,ROUND(3.9*55.15,2)=ROUND(3.900000000*55.15,2) ,ROUND(3.9*55.15,3)=ROUND(3.900000000*55.15,3)
>
>a=55.15
>y=3.900000000
>x=3.9
>rx=x*a
>ry=y*a
>? 'variables round 2 is bad, round 3 is ok'
>? x=y, x*a=y*a ,ROUND(x*a,2)=ROUND(y*a,2),ROUND(x*a,1)=ROUND(y*a,1)
>? rx=ry, ROUND(rx,2)=ROUND(ry,2) ,ROUND(rx,3)=ROUND(ry,3)
>
>
>On math rules, if Round(,n-1) are equals then Round(,n) are equal.
>
>VFP uses floating point variables for do any numeric computation ( integer,numeric,float,date,
>datetime ... ), and with SET DECIMALS TO ... try to solve all, but this is a illusion.
>
>On VFP9 add list i have not see the top of the top item:
>"Add to VFP correct functions for fixed decimal mathematic."
>
>Fabio
Gregory
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform