I have the following trivial code where mMaxX and mMinX are doubles, while N is integer. In one calculation it happens that mMaxX is 0.8, mMinX is -1.2 and N is 10.
Look at the corresponding output. Instead of 0 I have -2.220446e-16 !!!
It’s not the first time I see this kind of problem. Ideas on how to solve it?
for i as integer = 0 to N
xt(i) = i * (mMaxX-mMinX)/N + mMinX
PrintOutput Str(i * (mMaxX-mMinX)/N + mMinX)
It has to do with how base-10 doubles are represented by base-2 arithmetic. Unfortunately, you will see things like that, but they are usually meaningless. The number you got is so vanishingly small that it might was well be zero.
However, if you know you will be dealing with no more than two decimal points, you can try Currency instead of Double, or you can test your values for a certain tolerance within zero and set it manually.
Hi Kem, thanks for the answer. Even though that number if vanishingly small it hurts me because I use those values in a chart as grid values and a value like that compromises the appearance of the chart. I should start thinking about how to solve this.
be aware that this is an issue with how DOUBLE is stored and is not unique to XOJO… any language that bases its internal storage of floating non-integer values on IEEE standards and not a BCD (binary coded decimal) format will suffer this to some degree.