Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Decimal precision
Message
From
23/05/2012 03:14:01
 
 
To
22/05/2012 22:25:46
General information
Forum:
ASP.NET
Category:
Coding, syntax and commands
Environment versions
Environment:
VB 9.0
OS:
Windows 7
Network:
Windows 2003 Server
Database:
MS SQL Server
Application:
Web
Miscellaneous
Thread ID:
01544101
Message ID:
01544116
Views:
56
>In VFP, the default is rounded at two decimal places such as:
>
>? 25/49.88
>
>...will result at 0.50.

Yes and no. It displays only two decimals since set('decimals') = 2, but has more than 2 decimals

see
? empty(25/49.88-.50)  && false
? (25/49.88-.50)==0.00  && False
?100000000/(25/49.88) && 199520000.00
>However, changing the decimal precision to 10 will give 0.5012028869.



>
>In .NET, if I have defined a variable to be a Double. If I do the same calculation, will I obtain right away a better precision? If yes, what would be the default number of decimals place that would be considered?


That depends on the context. If the context is currency, you will mostly only consider two decimals

In your tax problem
base for federal tax = 500.00
round(500.00 * .05, 2) = 25.00 && federal tax

base for  provincial tax = 500.00 + 25.00 = 525.00

round( ( 500.00 + 25.00) * 9.5 / 100, 2) = 49.88 && provincial tax

Total = 500.00 + 25.00 + 49.88 = 574.88
So, if you want to reverse the process, you have to apply the steps backwards - one at a time - do not combine them - this is no algebra
After each division you have to round to 2 decimals
Total = 574.88
(1) 
Base for provincial tax = round( 574.88 / 1.095, 2) = 525.00

provincial tax = 574.88 -  525.00 = 49.88

(2) 
base for federal tax = Base for provincial tax / 1.05
base for federal tax = round( 525.00 / 1.05, 2)  = 500.00
federal tax = 525.00 - 500.00 = 25.00
iow, you need two decimals since the taxes were calculated with two decimals

You apply the steps backwards
- dividing and rounding to 2 decimals to obtain the base
- subtracting to obtain the tax ( do not use a division here )
Gregory
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform