I’m not sure if this is new in the latest release, but the following [simple] equation is returning what I consider the wrong answer:
Dim w, h As Integer
Dim aspect As Single
aspect = 640 / 480
w = 640
h = w / aspect
This is a simplification of the code in my actual app, but in either case h is calculated as 479 not 480, which doesn’t make any sense. Even knowing that the result is getting assigned to an integer, it should still be 480 since 640!/1.333333333333333! is actually 480.000000000000012
What is going on?
Generally there is no reason to use Single. Using Double instead seems to calculate h as 480. Probably related to type conversions of some kind, but maybe Joe R. has a better answer.
It happens also with 2015r1.
Dim w, h As Integer
Dim aspect As Single
aspect = 640 / 480
w = 640
h = Round(w / aspect)
With "Round(w / aspect) " it works. So it could have something to do with the rounding of the assignment.
I believe that the code is doing what is expected.
Dim w, h As Integer
Dim aspect As Single
aspect = 640 / 480
Dim s as string = Format(aspect,"#.############") // s = "1.33333337307"
w = 640
h = w / aspect // 640. / 1.33333337307 = 479.999985694 (truncated is 479)
The last statement probably promotes w to a single to match aspect and the result 479.999… and is truncated to an integer. If h is a single versus integer then it will compute to 480.
Thanks Paul. Switching to a double did the trick though I would like to know why (i.e. what is the compiler doing?)
@William Jones: Interestingly, looking at the value in the “Variables” pane at runtime shows aspect = 1.333333333333333 (15 3’s) regardless of whether it’s defined as a single or a double. I’d call that a bug.
When I look at the variable aspect (as a single) in debugger with 2015r2 I am seeing 1.333. A variable dimmed as a single can only have a precision of about 6-9 decimal digits. You can see that using the format statement with gives “1.33333337307” which is only accurate to 8 digits.
not really 
Singles and Doubles only have a certain amount of resolution. Mathematically the exact answer to 640/480 is 4/3, not 1.333333333333. If you put a bar above the last 3 to mean ‘extend 3’s forever’ then it would represent the exact answer, but Singles and Doubles don’t/can’t do that. Instead they get set to the closest value they can represent. Sometimes this is a bit lower, sometimes a bit higher, sometimes exactly.
When the value is a bit off from being mathematically exact, each time you use it in an operation it’s using a slightly wrong value, plus the result using this wrong value may not have an exact representation, wiggling more. So this error margin grows as you use floating point numbers but usually it’s low enough it’s still accurate enough.
Doubles have a much greater resolution than Singles, but neither can represent 4/3. It appears Doubles are accurate enough that after the calculation the result gets wiggled back to exactly 480 (or maybe above). But not always. You’ll see it less with Doubles but you still have this problem of error margin. Try this code that uses a Double for aspect and tests heights from 1 to 1000; there’s 40 heights that don’t return to the same value. (Change Double to Single and 536 mismatch)
dim aspect As Double, height, result, count As Integer
for height = 1 to 1000
aspect = 640 / height
result = 640 / aspect
if result <> height then
count = count + 1
TextArea1.AppendText Str(height) + " -> " + Str(result) + EndOfLine
end
next
TextArea1.AppendText "counted mismatches: " + Str(count)
Each mismatch happens because the result is calculated as a floating point number that is just below the input height, and when assigned to an integer it doesn’t round, it just drops the fractional part.
The correct thing to do is Round. What you’re after is the closest integer to a floating point result, assigning that floating point number to an integer doesn’t do that, you have to call Round yourself. Use “result=Round(640/aspect)” above and there’s no mismatches, even using a Single.