Problem with Floor()

Hi to all.

I do have problems using ceil(), floor() or round().

I tried to create a method to round to a certain number of decimal places. Most of the time it works (I solved it like described in the help to ceil() ) - but sometimes it just does not work as expected.

[code]wert As Double, stellen As Integer

Dim dResult As Double
Dim Potenz As Integer

Potenz = 10^stellen

dResult = wert*Potenz
dResult = Floor(dResult)
dResult = dResult/Potenz

Return dResult
[/code]
For wert = 0,6333333333333333 and stellen = 5 the situation is as follows:

dResult = wert*Potenz => 63333,3333333333284827
dResult = Floor(dResult) => gives 63333 as expected but the next line
dResult = dResult/Potenz => gives 0,6333299999999999 // in most cases I do get the expected 5 digits on the right side - but not every time.

Can anybody give me a hint what I’m doing wrong? Thank you!

Best regards
Peter

Nothing - a double simply cannot, in this case, represent .633333 accurately enough.
It all comes down to how to represent that value as the sum of powers of 2.
Every integer can be expressed as some sum of 2^0, 2^1, and so on.
The fractional portion in a double is also powers of 2 as 2^-1 (1/2), 2^-2 (1/4), etc.
And if you try to generate .633333 you’ll find the best you can do is get VERY close - but not exactly that value.
see http://en.wikipedia.org/wiki/Floating_point#Representable_numbers.2C_conversion_and_rounding

Use Bob Delaney’s maths plugins to do it right.

http://delaneyrm.com/

The good news is, you can force it to display properly to your users so they won’t know that anything is “wrong”, and that value will always be represented internally the same way, so a comparison will work. In other words:

  p = 10. ^ 5.
  d1 = 0.6333333333333333333
  d1 = Floor( d1 * p ) / p
  d2 = 0.63333
  // d1 = d2 because both are represented by 0.6333299999999999