Fun with precision

And by fun, I don’t mean fun at all.

Try this one in any version of Xojo: (32 and 64 bit tested)

dim d as double = 27.45
dim s as string = format(d, “-#0.0”)

The result for s? 27.4 truncated, rather than rounded

Watch it in the debugger and you’ll see that d is 27.4499999999999993

I get it, precision. The machine can only store so many numbers, some fall off the edges a bit a get stored as a fairly close number.

My problem is that the str and format functions don’t handle it in a sane manner. I’m not certain it’s their fault, but what a pain it can be,

If I do this:

dim d as double = 27.45
dim r as double = Round(d*10.0)/10.0
dim s as string = format(r, “-#0.0”)

I get 27.5 for r and s - so I’m able to round it myself but there’s the risk that I might clip a large number by multiplying it by 10

So what are we to do? I can add a very small number like 0.000001 before I use format but I suspect that’s going to introduce the reverse problem on other numbers.

Anyone have any suggestions?

From my old computer class days, adding 5 one position to the right of where you want to round, and then truncating, was the standard way of doing it.

You could incorporate a function like this:

Public Function RoundN(value As Double, decimalPlaces As  Integer) as String
  dim adjusted As Double = value+exp(-(2+decimalPlaces)*Log(10))*5
  dim formatPattern As String = "-########."+left("##################",decimalPlaces)
  Return format(adjusted,formatPattern)
End Function

Thrown together quickly and not exhaustively tested.

thanks for that… not sure how it works yet.

It does round 27.45 to 32.5 though, so we’re not quite there…

I noticed an error after I uploaded, and then edited it. You may have tested the earlier buggy version. I’ll check it again to make sure I didn’t do something silly.

Arrrgh! Still not right. I was trying to make things far more complicated than necessary.

Basically, if you want to round to n decimal places, this function will add 5 in the (n+1)th position, and then the formatting would chop off everything after n decimal places. I didn’t need all the convoluted log and exp functions to do this; I was overcomplicating it. This one should make more sense.

Public Function RoundN(value As Double, decimalPlaces As  Integer) as String
  dim adjusted As Double = (abs(value)+5*10^-(decimalPlaces+2))*sign(value)
  dim formatPattern As String = "-########."+left("##################",decimalPlaces)
  Return format(adjusted,formatPattern)
End Function

Still no guarantees, but this one should make a bit more sense.
It’s been a long day, and I should probably quit for the night.

several options

  1. never round anything - always use int64 and deal in “pennies” or “tenths of pennies” and use an implied decimal point

  2. use a currency (which is kind of like option 1)

  3. grab bob delaney’s Decimal Plugin and skip all this

  4. use doubles and perform self-flagellation repeatedly for having done so :stuck_out_tongue:

1 Like

I guess it depends on the application. If the OP is dealing in currency, then the currency data type is certainly the way to go. He mentions concern about the risk of overflow when multiplying by 10. If this is about money, then that’s a problem I’d like to have. :slight_smile:

For my own work, I’m dealing with engineering calculations, and the double data type is the only suitable type for speed, range and precision. We need the full precision for all of the intermediate calculations in order to prevent round off error, but the final result should be presented with the amount of precision that is appropriate for the precision of the original input data. That may be only 4 or 5 digits, and so it would be misleading to output a number that shows 15 digits.

I’d look into bob delaney’s plugins as he has ones that are basically “infinite precision”
And since they are plugins they should perform decently

Yes, I’ve used Bob Delaney’s Decimal plug-in in the past when I’ve needed more precision than what the double will give. He has several very useful math plug-ins on his site, including a complex number plug-in which I haven’t tried yet, but expect it’s going to come in very handy at some point.