Plateforme Level Extreme
Abonnement
Profil corporatif
Produits & Services
Support
Légal
English
Will SET PRECISION fix odd math?
Message
De
24/02/2006 11:11:46
Cetin Basoz
Engineerica Inc.
Izmir, Turquie
 
 
À
24/02/2006 08:53:34
Information générale
Forum:
Visual FoxPro
Catégorie:
Problèmes
Versions des environnements
Visual FoxPro:
VFP 6 SP5
OS:
Windows XP SP2
Network:
Novell 4.x
Database:
Visual FoxPro
Divers
Thread ID:
01098837
Message ID:
01098919
Vues:
15
>Hi Everyone,
>The purpose of this code is to use math to determine how many decimal places are in a floating point number. I have used this algorithm when importing unknown/unpredictable data to help determine what data type and size I would like the field to be. The odd thing is that although the algorithm is mathematically sound, Fox does some creative math when I feed in numbers with a lot of decimal places. Here is the code:
>
>
>nValue = 234.8234721345
>LOCAL decPortion, itterationCtr
>
>itterationCtr = 0
>decPortion = nValue - INT(nValue)
>DO WHILE INT(decPortion) < decPortion
>	decPortion = decPortion * 10
>	itterationCtr = itterationCtr + 1
>	?decPortion
>ENDDO
>? itterationCtr
>
>
>... and the output starts like ...
>
>
>8.2347213450
>82.3472134500
>823.4721345000
>8234.7213450001
>82347.2134500008
>823472.1345000083
>8234721.4560000830
>
>
>You can run it & see how it ends from there, but the interesting part begins on the 4th element of output. All I am doing is multiplying the number by 10. Why does 823.4721345000 * 10 get me 8234.7213450001? I'm assuming it has SOMETHING to do w/ mathematical precision, but if I just type in
?823.4721345000 * 10
outside of this looping logic, it returns the CORRECT value w/o that '1' tacked onto the end of it. Does anyone have any thoughts or insights on this?



That creativity is not foxbased < g >.
nValue = 234.8234721345
cValue = trim(trans(m.nValue))
? iif(occurs('.',m.cValue),len(m.cValue)-at('.',m.cValue),0)
Why don't you simply use:
lnSignificantdecimals = 10
create table ... (myMathField b(m.lnSignificantDecimals), ...)
(or Y)
Cetin
Çetin Basöz

The way to Go
Flutter - For mobile, web and desktop.
World's most advanced open source relational database.
.Net for foxheads - Blog (main)
FoxSharp - Blog (mirror)
Welcome to FoxyClasses

LinqPad - C#,VB,F#,SQL,eSQL ... scratchpad
Précédent
Suivant
Répondre
Fil
Voir

Click here to load this message in the networking platform