Plateforme Level Extreme
Abonnement
Profil corporatif
Produits & Services
Support
Légal
English
Will SET PRECISION fix odd math?
Message
De
24/02/2006 13:16:22
 
 
À
24/02/2006 12:12:00
Cetin Basoz
Engineerica Inc.
Izmir, Turquie
Information générale
Forum:
Visual FoxPro
Catégorie:
Problèmes
Versions des environnements
Visual FoxPro:
VFP 6 SP5
OS:
Windows XP SP2
Network:
Novell 4.x
Database:
Visual FoxPro
Divers
Thread ID:
01098837
Message ID:
01098995
Vues:
12
Hey, I'm not asking for AI here. :) I guess here's what I would have preferred to happen. Let me use your example.
? STR(234.8234721345-234,16,15)
This outputs the mathematically correct answer with some extra 0's and an 8 on the end, indicating this whole floating point math issue that computers inherantly have. The unfortunate thing is that if you don't convert it to a string & you just use the numeric answer it provides, it shows you the mathematically correct answer, while secretly remembering it's inherant margin of error which effects your result in a compound fashion when used in a loop, secretly building up until an incorrect result is returned. (incorrect based on expected base 10 mathematics, and also based on looking at the numeric values it's actually returning to you).

I'll figure out which way I want to go about circumventing this. It looks like this is some age old thing that has been hashed out repeatedly, and I just seldom to never have needed to use so many decimal places that I would have ever noticed it before. In fact, in application, this program may never reach that program either, but in my testing I saw this calculational anomale & I'd prefer to use a more airtight method in the final product.

Thanks for your time & comments.

>Hey Paul,
>We're only at year 2006 and you're comparing the brilliant human brain with computers:)
>Cetin
>
>>Hi Fabio,
>> Thanks for taking the time to respond. Is it too much to ask for mathematical function that I believe the schools are teaching around the 3rd grade to just 'work right'? 823.4721345000 * 10 = 8234.721345000 except when I situationally decide to tack a 1 on the end of it? I think that's not how they're teaching it <g>.
>>
>>Thanks again... I'll be toying w/ a few ideas & see what works best for me.
>>
>>>
>>>VFP store not integer numbers into a IEEE floatig point format;
>>>with this you cannot store exactly number.
>>>Then, is impossible make exact computation with a specific number of decimals.
>>>
>>>example
>>>
>>>? STR(234.8234721345-234,16,15)
>>>
>>>You can use currency as a surrogate.
>>>
>>>Uses this for you problem
>>>
>>>? LEN(RTRIM(CHRTRAN(PADR(m.nValue % 1,16),'0',' ')))-2
>>>
Paul A. Busbey
Victoria Insurance
Précédent
Répondre
Fil
Voir

Click here to load this message in the networking platform