Xojo Math badly broken: 32bit problems in a 64bit environment

For a forthcoming article in xDev I had to use the Mod function and came across an interesting problem:

From the Xojo documentation:

result = number1 Mod number2

The Mod operator divides number1 by number2 and returns the remainder as result. If either number1 or number2 is a floating-point type, it is first coerced to Int32. If either number1 or number2 is an Int64, then both values are promoted to Int64.

What happens with result = doubleValue mod intValue?

result and intValue are Integer, and should be a 64 bit integer on a 64 bit system. Testing revealed that the coercing to 32bit Integer rule takes precedence over promoting to 64bit Integer on the right side.

But why would the double be coerced into a 32bit integer on 64 bit systems? That just screams overflow error! Have Xojo forgotten to update this when they moved to 64 bit?

Basically you are running into 32 bit limitations on a 64 bit system.

Christian dug out the over 10 year old bug report turned feature request 11536 - mod fails for doubles with divisors greater than [ ( 2 ^ 31 ) - 2 ] by Kem Tekinay that has been stuck in “reviewed” for over 10 years. It concludes “Please change this to a feature request for mod to coerce the values to Int64 instead.”

But it is MUCH worse than that: the same seems to apply to a number of other Math functions - it seems the underlying structure never got updated to 64 bit, and therefore with larger numbers you run into 32 bit limitations because of a systematic problems in Xojo.

The biggest problem is that your code might seem to work just fine - until it hits a not very large number that Int64 should easily handle and instead it fails. Which it would not if Xojo used Int64 on 64bit systems instead of coercing to Int32…

And now Xojo don’t seem to have a compiler guy anymore to fix it?

:man_facepalming:

What kind of programming language has a systematic Maths error like this?

2 Likes

Marcus, why do you keep saying this about our team? We’ve done several compiler updates over the last few years since Joe Ranieri left. In the future, please check your sources before making statements like that.

10 Likes

Does that mean, the Mod function does not work properly with Int64 compiled to 32 bit and 64 bit?

Has anybody done any tests to see what actually happens?

Int64 works. It’s just that double is truncated to Int32 for doing mod (and others).

Travis, Greg or William may be perfectly able to do a fix here buy checking where the Double to Int promotion is in the compiler’s frontend for creating the code and change it to use Int64 instead of Int32.

OK. So, changing it to follow the build settings rather than being hard-wired to int32.

… change it to use Int64 instead of Int32.

+1

As a workaround you can use CType (< double value >, integer).

BTW: I do this whenever it’s useful to tell the compiler, yes, I know that this could cause a loss of precision and I don’t want to see a warning about doing this.

1 Like

For giggles I tried to enable the compiler warnings on the numbers stuff. With over 10k warnings I didn’t see any need to check each one.

1 Like

But Xojo’s official stance is it’s not a bug so… :man_shrugging: I guess.

Seriously? THAT’s what you want to focus on?

For one thing because Geoff said you no longer have a dedicated compiler guy.

(see https://forum.xojo.com/t/something-funky-about-uint32-comparisons/57447/163 )

Whether your answer is correct depends a LOT on what you define as “compiler” (eg whether you count fixes in the pre-processing steps as “compiler fixes” or not).

But I’m glad to hear that Xojo can do compiler fixes so it should be easy to fix this … :roll_eyes:

The basic problem is that under a 32 bit system a Double can represent a MUCH larger range of integers accurately than an Integer can. So a LOT of code used Double instead of Integer to avoid integer overflows:

The maximum value of “whole numbers” that can be accurately represented:

32 bit 64 bit
Integer 2,147,483,647 9,223,372,036,854,775,807
Double 9,007,199,254,740,992 9,007,199,254,740,992
unsigned Integer 4,294,967,295 18,446,744,073,709,551,615

On 32 bit systems it made some sense to cast to a 32 bit integer even though I would argue it would have been better (and more future proof) to cast to a 64 bit integer (and yes, you CAN use 64 bit integers on 32 bit systems though it will probably slow down the calculation).

However on 64 bit systems it is simply not acceptable to cast a Double to 32 bit Integers.

Of course. I’m not a theoretical physicist … :wink:

Something that causes int32 overflow errors where none should be is a bug, not a feature.

Basic functionality should be rock solid. Especially where computers and Maths are concerned.

Xojo’s Maths implementation leaves a LOT to be desired.

Btw the two Boeings that fell out of the air because their engines stalled? That was due to a 32bit Integer overflow error.

6 Likes

P.S. d.TotalSeconds is a double with current value 3,686,809,196, so performing a mathematical operation with it might cause it to be cast to a 32bit Integer and consequently a 32 bit integer overflow without users being aware of it.

3 Likes

Maybe that’s why I get Int32 when using DateInterval:

1 Like

I tried your code. At first I thought I could not repeat the problem but then I realized that I needed to add another column to the Listbox :man_facepalming: and I wonder if the reviewers had the same problem - it is better to submit an example project!

Now your code SHOULD better be

Dim date1 As xojo.core.date = xojo.core.date.now
Dim date2 As xojo.core.date = xojo.core.date.now
Dim date3 As xojo.core.date = xojo.core.date.now

Dim interval2 As New Xojo.Core.DateInterval
Dim interval3 As New Xojo.Core.DateInterval

interval2.NanoSeconds = 2147483647
interval3.NanoSeconds = 2147483648

Listbox1.AddRow("Date Now", date1.ToText)
Listbox1.AddRow("Date 1", date2.ToText)
Listbox1.AddRow("Date 2", date3.ToText)

date3 = date2 + interval2
Listbox1.AddRow("Add 2147483647 nanosec to date 2", date3.ToText)

date3 = date2 + interval3
Listbox1.AddRow("Add 2147483648 nanosec to date 2", date3.ToText)

and I do see it: adding interval3 leads to a subtraction:

screenshot_208

So YES, that seems to be the same “cast to Int32 overflow error”

1 Like

The case is old, the forum post is newer (I didn’t know that my case was closed for several months).

You can use seconds instead of nanoseconds with DateTime to see the big difference:

Var d1 as DateTime = DateTime.Now
Var d2 as DateTime = DateTime.Now
Var d3 as DateTime = DateTime.Now

Var interval2 as new DateInterval
interval2.Seconds = 2147483647

Var interval3 as new DateInterval
interval3.Seconds = 2147483648

d2 = d2 + interval2
d3 = d3 + interval3
//d1.SQLDateTime = 2020-10-28 16:53:53
//d2.SQLDateTime = 2088-11-15 19:08:00
//d3.SQLDateTime = 1952-10-10 12:39:45

Interesting that the IDE uses different green for the seconds:
image

Yep - I see that too:

As you can see TotalSeconds / SecondsFrom1970 is a double, and adding an integer will cause it to be cast to an Int32. The sum of the two will be then done as two Int32 so interval 3 is added as a negative value.

That simply should not happen on a 64 bit system.

1 Like

Int64 and uint64 are present in 32bit systems, just the processing is a bit slower emulating instructions, so it should not happen anywhere. As for when to promote values to larger types and demote to smaller, it needs a better smarter expression evaluator with better type evaluation of the expression and finally promoting or demoting the results to fit the destination (when demoting, i.e. casting to a inferior type losing some information, a warning must be raised unless you have an explicit cast to say you did it in a purpose, or compiler directive to say it; a pragma in Xojo terms). The Xojo expression evaluation engine needs a full review, new types were introduced and it’s not able to work correctly with them. Also, bitwise operations should be written direct in such expressions using the CURRENT bitwise operators and not having a module with methods to emulate them in a slower fashion just because the values are in a 64bit range.

Nothing wrong with being a theoretical physicist (my brother was one). But they do rely on experimental results and not just what the documentation says. :thinking:

So

date3 = date2 + interval2

becomes

Dim sec As Double = CType( date2.secondsFrom1970, Integer ) + CType( interval2.seconds, Integer )
Dim date4 As New xojo.core.date( sec, date2.TimeZone )

as you can’t assign a value to SecondsFrom1970 but you need to create a new instance and set it from the seconds you have …

:flushed:

[quote=“TimStreater, post:17, topic:58288, full:true”]

Actually that was my “The Big Bang Theory” reference :wink:

I like sprinkling my writing with movie references … makes it more fun … well, for me …

1 Like

I know - I said so further up :wink: