Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
What's wrong with VFP 5.0a's math (or is it me)?
Message
General information
Forum:
Visual FoxPro
Category:
Coding, syntax & commands
Miscellaneous
Thread ID:
00032141
Message ID:
00032337
Views:
37
>>>I have a program with the following line of code:
>>>
>>>lPayment_Amount = lpct * tpd_payments_lte.weekly_ttd_rt
>>>
>>>When I run my program and check the debugger, the following values are used:
>>>
>>>10.8539997 = 0.04998 * 217.18
>>>
>>>When I ROUND(lPayment_Amount,2) I get 10.85.
>>>
>>>But when I do the math using the Windows 95 calculator (0.04998 * 217.18) I get 10.8546564 which should round to 10.86.
>>>
>>>Any ideas on how I can fix this?
>>>
>>>Thanks!
>>>
>>>Jeff Rusch
>>>UW-Madison
>>
>>If I multiple in VFP I get 10.8546564. BTW, rounding 10.8546564 should give you 10.85 not 10.86. So it looks like you are getting the right results (10.85) but where does the 10.8539997 come from ???
>>
>>-Doug-
>
>If I multiply using the numbers directly, I also get the correct answer. But in my program I get an incorrect answer and debugger indicates the values of the variables to be as I've indicated. So I'm guessing that VFP isn't using the exact numbers the debugger has indicated.
>
>Also, how come 10.8546564 rounds to 10.85. Shouldn't the 6 in 10.8546 make it 10.855 and then the 5 make it 10.86?
>
>Jeff

Sure not. The rounding operation is not recursive.

Vlad
Previous
Reply
Map
View

Click here to load this message in the networking platform