Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
What's wrong with VFP 5.0a's math (or is it me)?
Message
General information
Forum:
Visual FoxPro
Category:
Coding, syntax & commands
Miscellaneous
Thread ID:
00032141
Message ID:
00032225
Views:
43
>>>I took a look since seeing this got me concerned (I do lots of statistical calculating). The bug that's shown in the KB article is too many (unneeded) parentheses in an equation, so we just have to watch it when we use them...that's not your problem, is it?
>>
>>Are you sure you're looking at the same KB article? In the steps to reproduce they list a formula that containsa number of parenthesis, but it doesn't say that this is the only situation that produces the problem.
>>
>
>You could be right, I'm inferring that since removing the extra parens gives correct solution, that's the only problem. I would hope that my inference is right, but I think I'll test it later to check it.
>
>>My problem is this: when my program runs, I have a line of code that is something like:
>>
>>variable1 = variable2 * table.field
>>
>>If variable2 = 0.04998, the value of table.field is 217.18, then I should get variable1 = 10.8546564. Instead I get variable1 = 10.8539997.
>>
>>While my program is running, if I go to the command window and type variable2 = variable1 * 217.18 I also get the incorrect answer. But without the program running, if I type variable1 = 0.04998, variable2 = variable1 * 217.18, I get the correct answer.
>>
>>I am unable to reproduce this by creating a new program.
>>
>
>I used to have this trouble in PL/I, they used binary processing to compute and odd results were common. Anyway, did you try setting a variable 3 to the table.field before evaluating to see if result is good? And also maybe try the KB doc's idea of using ROUND?


check your set decimals....
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform