Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Strange results in COM DLL
Message
General information
Forum:
Visual FoxPro
Category:
COM/DCOM and OLE Automation
Miscellaneous
Thread ID:
00467419
Message ID:
00467463
Views:
16
>>George,
>>This could be a problem with rounding and how many decimal places are displayed. I think outputting the values to text respects SET DECIMAL so the values may look exact. However, internal processing may have the values off by .000001 (not equal). My suggestion would be to use ROUND() where you need to to make the number of places decimal explcit to what you want.
>>
>>HTH.
>
>Larry,
>
>Thanks for the response.
>
>I considered that issue (and the bug regarding very small numbers). DECIMALS is set to 4. However, in this case there are a couple of things that make me wonder if this applicable to this case. First off, it would seem to me that 145.6 divided by 0.1000 shouldn't yield this when compared to 1456 and against other data it hasn't. Second, in VFP, working with the same data, using the same DLL it works fine.
>
>Frankly, in this case I've got to approaching rounding very carefully. For example, the nTuftedWidth property may be divided by a value as small as 0.078125. The if the result of this is off by as little as 0.00032, then the underlying values are incorrect.

George,

There is a bug in TRANSFORM function.
According to KB Article Q247494 "You might see that the results of the TRANSFORM() function appear off by approximately .0000000000001. This behavior may occur when COMPATIBLE is OFF."
--sb--
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform