issue with rounding


I have the following trivial code where mMaxX and mMinX are doubles, while N is integer. In one calculation it happens that mMaxX is 0.8, mMinX is -1.2 and N is 10.
Look at the corresponding output. Instead of 0 I have -2.220446e-16 !!!

It’s not the first time I see this kind of problem. Ideas on how to solve it?


    for i as integer = 0 to N
      xt(i) = i * (mMaxX-mMinX)/N + mMinX
      PrintOutput Str(i * (mMaxX-mMinX)/N + mMinX)



It has to do with how base-10 doubles are represented by base-2 arithmetic. Unfortunately, you will see things like that, but they are usually meaningless. The number you got is so vanishingly small that it might was well be zero.

However, if you know you will be dealing with no more than two decimal points, you can try Currency instead of Double, or you can test your values for a certain tolerance within zero and set it manually.

Hi Kem, thanks for the answer. Even though that number if vanishingly small it hurts me because I use those values in a chart as grid values and a value like that compromises the appearance of the chart. I should start thinking about how to solve this.

be aware that this is an issue with how DOUBLE is stored and is not unique to XOJO… any language that bases its internal storage of floating non-integer values on IEEE standards and not a BCD (binary coded decimal) format will suffer this to some degree.

If it’s just the printed representation you could use Format

PrintOutput Format(i * (mMaxX-mMinX)/N + mMinX), “0.0”)


You’ll see these kind of things everywhere. You can try to keep these things to a minimum by formatting the representation but the issue will always be there.

Test this in your favorite spreadsheet program:

Put 0.2 in field A1
Now put =A1*6-1 in field A2. This should be 0.2 again right?
Now copy A2 from A3 downwards to A50 or so.

Thanks for all the replies. The solution I currently found is the following:
I check the number of decimals in the sequence. If I see an outlier I round it to have the same precision of the neighbors.

When comparing floating point numbers for equality, the usual strategy is to subtract one from the other and take the absolute value. If it’s smaller than some epsilon (e.g. 10^-7), they are equal.

One thing to remember: if you want to represent money, use Currency. Not Double.

Or scaled 64 bit integers when you need decimal precision greater than 4 places :stuck_out_tongue:

Isn’t that what the Xojo function, “Equals” is for?