Plateforme Level Extreme
Abonnement
Profil corporatif
Produits & Services
Support
Légal
English
Will SET PRECISION fix odd math?
Message
De
24/02/2006 10:10:56
 
 
À
24/02/2006 09:00:31
Information générale
Forum:
Visual FoxPro
Catégorie:
Problèmes
Versions des environnements
Visual FoxPro:
VFP 6 SP5
OS:
Windows XP SP2
Network:
Novell 4.x
Database:
Visual FoxPro
Divers
Thread ID:
01098837
Message ID:
01098867
Vues:
12
>This matter have been discussed to a point of boredom before...! <bg> No offence intended.
>


None taken... hey... where's Sergey w/ the links to the old threads where this was discussed before? Sick today? <g>


>A decimal number can not be presented correctly binary, it is simply impossible, so they have to choose the best of several evils. If you want real precision, calculate the data by a large number, and divide the end result by the same number.
>


Does it make sense to you though, Tore, how it performs the incorrect calculation within my loop, and it performs the calculation correctly when I type just that line in the command window? It would seem like if I can isolate what the difference is, then I could get more consistant performance and avoid the proposed fact of "A decimal number can not be presented correctly binary, it is simply impossible".


>
>
>>Hi Everyone,
>>The purpose of this code is to use math to determine how many decimal places are in a floating point number. I have used this algorithm when importing unknown/unpredictable data to help determine what data type and size I would like the field to be. The odd thing is that although the algorithm is mathematically sound, Fox does some creative math when I feed in numbers with a lot of decimal places. Here is the code:
>>
>>
>>nValue = 234.8234721345
>>LOCAL decPortion, itterationCtr
>>
>>itterationCtr = 0
>>decPortion = nValue - INT(nValue)
>>DO WHILE INT(decPortion) < decPortion
>>	decPortion = decPortion * 10
>>	itterationCtr = itterationCtr + 1
>>	?decPortion
>>ENDDO
>>? itterationCtr
>>
>>
>>... and the output starts like ...
>>
>>
>>8.2347213450
>>82.3472134500
>>823.4721345000
>>8234.7213450001
>>82347.2134500008
>>823472.1345000083
>>8234721.4560000830
>>
>>
>>You can run it & see how it ends from there, but the interesting part begins on the 4th element of output. All I am doing is multiplying the number by 10. Why does 823.4721345000 * 10 get me 8234.7213450001? I'm assuming it has SOMETHING to do w/ mathematical precision, but if I just type in
?823.4721345000 * 10
outside of this looping logic, it returns the CORRECT value w/o that '1' tacked onto the end of it. Does anyone have any thoughts or insights on this?
Paul A. Busbey
Victoria Insurance
Précédent
Suivant
Répondre
Fil
Voir

Click here to load this message in the networking platform