Plateforme Level Extreme
Abonnement
Profil corporatif
Produits & Services
Support
Légal
English
Decimal precision
Message
De
23/05/2012 03:14:01
 
 
À
22/05/2012 22:25:46
Information générale
Forum:
ASP.NET
Catégorie:
Code, syntaxe and commandes
Versions des environnements
Environment:
VB 9.0
OS:
Windows 7
Network:
Windows 2003 Server
Database:
MS SQL Server
Application:
Web
Divers
Thread ID:
01544101
Message ID:
01544116
Vues:
58
>In VFP, the default is rounded at two decimal places such as:
>
>? 25/49.88
>
>...will result at 0.50.

Yes and no. It displays only two decimals since set('decimals') = 2, but has more than 2 decimals

see
? empty(25/49.88-.50)  && false
? (25/49.88-.50)==0.00  && False
?100000000/(25/49.88) && 199520000.00
>However, changing the decimal precision to 10 will give 0.5012028869.



>
>In .NET, if I have defined a variable to be a Double. If I do the same calculation, will I obtain right away a better precision? If yes, what would be the default number of decimals place that would be considered?


That depends on the context. If the context is currency, you will mostly only consider two decimals

In your tax problem
base for federal tax = 500.00
round(500.00 * .05, 2) = 25.00 && federal tax

base for  provincial tax = 500.00 + 25.00 = 525.00

round( ( 500.00 + 25.00) * 9.5 / 100, 2) = 49.88 && provincial tax

Total = 500.00 + 25.00 + 49.88 = 574.88
So, if you want to reverse the process, you have to apply the steps backwards - one at a time - do not combine them - this is no algebra
After each division you have to round to 2 decimals
Total = 574.88
(1) 
Base for provincial tax = round( 574.88 / 1.095, 2) = 525.00

provincial tax = 574.88 -  525.00 = 49.88

(2) 
base for federal tax = Base for provincial tax / 1.05
base for federal tax = round( 525.00 / 1.05, 2)  = 500.00
federal tax = 525.00 - 500.00 = 25.00
iow, you need two decimals since the taxes were calculated with two decimals

You apply the steps backwards
- dividing and rounding to 2 decimals to obtain the base
- subtracting to obtain the tax ( do not use a division here )
Gregory
Précédent
Suivant
Répondre
Fil
Voir

Click here to load this message in the networking platform