Rounding dilemma

We ran into an issue with rounding. And it’s not because our rounding function, or even MBS’s rounding function doesn’t work, it’s because of the way XOJO is storing the value.

If you enter a value 32.175 in a text field or a variable, XOJO stores the value as 32.174999999. If you then try to round it two places, you ALWAYS get 32.17, which is incorrect.

Oddly, if you enter the value 32.165, and then round it to two placed, it round correctly to 32.17.

Anyone know a workaround and how many other numbers don’t round correctly because of the way xojo is storing these numbers?

This happens because you’re using a Double. If you want more exact rounding, use Currency or an Integer that gets divided when it’s displayed.

1 Like

I think I found a better solution which I will post tomorrow after further testing.

Using currency does NOT solving the rounding issue. How do you convert 32.175 to rounded 2 digits?

And what I thought would work did not. Still looking for a solution.

An old trick is to add 0.005 (one more decimal place than you want), then multiply by 100 and round that, then divide by 100 to get your final answer.

dim d1 as double = 32.175
dim n as integer = round((d1 + .005) * 100)
dim d2 as double = n / 100

But this is why you never use doubles for apps that deal with money. It introduces creeping error when adding up monetary amounts. It is much better to use integer values and divide by 100 for display purposes.

3 Likes

Thanks Tim. If anyone had the answer, I knew you would.

But doesn’t this also point out the flaw in the way xojo is storing doubles? I don’t see how it makes sense to store 32.175 as 32.17499999

It’s nature of floating point real numbers stored in binary code… Certain numbers can not be stored exactly in the number of bits used…

-karen

You really should read this:

https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

2 Likes

Xojo is using the IEEE standard format. It’s not a shortcoming of Xojo, it’s just the nature of representing a floating point number on a computer.

3 Likes

This seems to work:

Var d As Currency = 32.175
Var e As Currency = Round(d*100)/100
Break

image

If I change to double:
image

Can 32.175 be represented exactly as a floating point number? If not, then its being represented as 32.174999999 is inherent in the hardware. Further, for it then to become 32.17 is not rounding, it’s truncation. How are you doing the rounding?

If you want to store non-whole numbers accurately you cannot use the Double (or Single) data type. This is a technical limitation of the data type in all languages and not a Xojo issue. There are various tricks which can help you avoid the limitation in some places but I have found from experience that you eventually hit a dead end.

The better solution is to use fixed point arithmetic.

You can roll your own using the Int64 data type, use the Xojo Currency data type or if you need something more complex you can try Bob Delaney’s fp plugin which is now being maintained by Björn Eiríksson at Einhuger: Einhugur Software - Open source projects

This is not a Xojo issue; it’s an issue related to the limited precision of floating point numbers. For example, in:

we see that:

“In computing, floating-point arithmetic is arithmetic that represents real numbers approximately, using an integer with a fixed precision, called the significand, scaled by an integer exponent of a fixed base.”

Forgive me for expecting rounding a floating point number to work correctly. You see, I came from Visual Foxpro and we are converting a financial application that used floating point numbers with 6 digits of precision and it handled floating point numbers correctly. Perhaps because they included their own rounding function that handles this without any problems.

I also wondered why our rounding function worked fine with 32.165 but did not work with 32.175? Of course, we only stumbled on this and in this particular case there was only 3 digits of precision, but in most cases we need 6 digits.

Also, correct me if I’m wrong, but since we need 6 digits of precision, the suggestion to use currency is a moot point since it only allows for 4 digits of precision.

Not trying to be cantankerous, just trying to find a solution for an issue that we need to resolve.

Again - how are you doing the rounding? For 32.17499999 to become 32.17 suggests truncation, not rounding.

Well, I know nothing of any FoxPro but if it’s a financial application perhaps they have their own arithmetic library which handles numbers in software. Arithmetic done with Doubles will use the built-in floating point hardware, which is IEEE 754 compliant.

1 Like

There seems to be a contradiction here.

Nonetheless, several have explained to you why this is not a Xojo-only issue and offered potential solutions. As a programmer who has identified a problem, it’s now up to you to make it work.

If you still think that Xojo can make an improvement here, you should file an issue in tracker.xojo.com.

They are behaving the way floating point numbers are supposed to.

Are you sure that Foxpro application was using doubles as a quick Google shows me Foxpro documentation that recommends using their Currency data type.

From Visual Foxpro’s command window I created two floating point numbers with 6 digits of precision and then used their internal ROUND function to round them to 2 decimals.

floats and doubles have this issue in all languages. There is no precise way to represent irrational numbers in floating point. This applies to IEEE floating point and to the older Microsoft floating point formats.

Because I write mostly business software, I avoid these issues in .NET using the Decimal type, which stores any number precisely without such issues, at the expense of a bit of memory and performance. Xojo lacks such a type; although there is a freeware plugin that implements a decimal type, I haven’t yet needed to use if for Xojo projects, so can’t vouch for it. Ominously, last I looked at it, the release notes mention fixing memory leaks (it’s written in C++) and the author “thinks” he got “all or most of them”. Not something you ever really want to read about a lib that will be frequently called.

1 Like