Pure XOJO code works differently for MacOS intel / arm

[Pawel_Soltysinski]

XOJO 2023 r.4, MacBook Pro M1

There is nice example of simple FTP client/server:
https://github.com/charonn0/RB-FTP

The author said it was tested only for Windows but it works OK for MacOS, too - but with strange behaviour. When connecting to simple old-ways FTP using Demo client, this happens:

• if compile architecture is set to Intel - it can log and authorise with user password correctly
• if compile architecture is set to arm/universal (so it is arm in my case) - you can log in into ftp but user password is not being sent.

I have searched trough the code and I couldn’t find anything intel/arm related - but it is enough to just switch between intel/arm when building - to see this strange difference.

something works differently between architectures, apparently. Any hints?

[Emile_Schwarz]

That is 12 years old code : (old) API.

Upgrade the code to API2 and see how it works.

I will not rely on so old code for a commercial application.

[Sam_Rowlands]

@Pawel_Soltysinski

something works differently between architectures, apparently. Any hints?

There are several things that I know of that work differently when compiling between the two different architectures. In some cases, Apple’s header files even says to use iOS values with macOS on ARM instead of traditional Mac values, in other places, it just is and I’ve not seen any comments as to why.

However, if you’re writing pure Xojo code, Xojo should have handled these situations automatically for you.

[Pawel_Soltysinski]

anyway, old API or not - should work the same

[AlbertoD]

@Pawel_Soltysinski

old API or not - should work the same

Maybe something is broken and fixed with API 2 that will never be fixed for API 1.

[Mike_D]

I just tried it, and it worked fine for me.

Test 1 running on Intel i9:

Test 2 running as universal build on M1 mac.

I’m using this FTP test server:
https://dlptest.com/ftp-test/

(note that if you use this test server, the password has an uppercase J which is hard to see on the website due to the Font used)

[Pawel_Soltysinski]

…it works, indeed. So it is server dependent. I mean there is subtle difference which triggers problem. The server I am experiencing the error is

[Mike_D]

Interesting - testing with your server ( serwer1469173.home.pl) without the correct username/password) does show a behavior difference.

Left: Intel Right: M1

Looks as if the ARM version is not reading the 331 password required… response correctly?

Even more strange, if I run the app in the Remote Debugger (from Intel mac to M1 mac) then it works normally. Hmm. Edit to add: The remote debugger seems to running the Intel version even though I’m remote debugging to ARM.

So there does seem to be an Intel/ARM difference here.

[Jean-Yves_Pochez]

@Emile_Schwarz

That is 12 years old code : (old) API.

FTP is … 40 years old ?
just kidding…

[Mike_D]

Interesting bug.

More logging reveals a key difference:

Intel

log

11:06:39 AM : My Application Launched
11:06:44 AM : FTP.Client.DataAvailable bytesAvailable=80 lookahead='220-Idea FTP Server 5.4.0 (serwer1469173.home.pl) [188.128.169.216]
220 Ready
’
: FTP.Client.ParseResponse 220-Idea FTP Server 5.4.0 (serwer1469173.home.pl) [188.128.169.216]

        : FTP.Client.ParseResponse 220 Ready

        : FTP.Client.DataAvailable bytesAvailable=39 lookahead='331 Password required for 'Username'.

’
: FTP.Client.ParseResponse 331 Password required for ‘Username’.

        : FTP.Client.DataAvailable bytesAvailable=42 lookahead='530 User name and password do not match.

Arm

log

11:07:01 AM : My Application Launched
11:07:04 AM : FTP.Client.DataAvailable bytesAvailable=80 lookahead='220-Idea FTP Server 5.4.0 (serwer1469173.home.pl) [188.128.169.216]
220 Ready
’
: FTP.Client.ParseResponse 220-Idea FTP Server 5.4.0 (serwer1469173.home.pl) [188.128.169.216]

        : FTP.Client.ParseResponse 220 Ready

11:07:05 AM : FTP.Client.DataAvailable bytesAvailable=39 lookahead='331 Password required for ‘Username’.
’
: FTP.Client.ParseResponse 331 Password required for ‘Username’.

        : FTP.Client.DataAvailable bytesAvailable=39 lookahead='331 Password required for 'Username'.

’
: FTP.Client.ParseResponse 331 Password required for ‘Username’.

        : FTP.Client.DataAvailable bytesAvailable=39 lookahead='331 Password required for 'Username'.

’
: FTP.Client.ParseResponse 331 Password required for ‘Username’.

        : FTP.Client.DataAvailable bytesAvailable=39 lookahead='331 Password required for 'Username'.

’
: FTP.Client.ParseResponse 331 Password required for ‘Username’.

Notice how on the M1 mac, the socket.bytesAvailable and socket.lookAhead values appear to be incorrect.

My hunch is that the bug is something like:

• In Xojo 2023r4

• on an ARM build
• in the TCPSocket.DataAvailable event
• TCPSocket.lookahead and TCPSocket.bytesAvailable are wrong.
  

EDIT: I think the bug is actually somewhere else.

[Mike_D]

I was totally wrong - the TCPSocket is behaving normally on ARM.

What is different? The FTP server is sending back different responses. Why? because the FTP client is sending different commands on Intel vs. ARM.

Why is this happening? Looks like there’s a bug in ARM with popping a Structure.

This code is not behaving the same on Intel and ARM:

If Not TransferInProgress And UBound(PendingVerbs) > -1 Then
  Dim nextverb As FTPVerb = PendingVerbs.Pop

Note that FTPVerb is a Structure and I suspect that’s important.

Here is what happens step-by-step on ARM:

First, notice that there are two FTPVerbs in the array. These are correct:

Next, we step through the command:
Dim nextverb As FTPVerb = PendingVerbs.Pop

And now there is only one item in the array, as expected. Item 1 was popped, leaving item 0.

What is the value of the nextVerb variable? It should be equal to the item that was popped “PASS Password”.

But it’s not:

I’ve seen some cases where adding extra logging to the app fixes the bug, so I think it’s some sort of compiler bug.

I’ll see if I can isolate it to a test case and report it as an Issue.

[Mike_D]

I started a new thread: Array.Pop with a Structure is buggy on ARM builds (Apple Silicon)

[Pawel_Soltysinski]

just thank You!

[Mike_D]

To circle back to the original issue: I’m not sure why the app uses a structure for FTPVerb, but I just did a test where I replaced FTPVerb with class called cFTPVerb, and after adding a few lines, it all works normally on Arm now, as the bug with Array.Pop() doesn’t seem to be triggered when the array is holding class instances instead of Structures.

Class cFTPVerb
  verb as string
  arguments as string
end Class

since Classes don’t instantiate automatically you also have to add “new” in a couple of places:

// dim nextverb as FTPVerb // structure
Dim nextverb As new cFTPVerb // using class

[Mike_D]

There is a separate bug which is that your FTP server sends two 220 responses in a row, and the FTP client then responds with the USER command twice. I’m not sure if this is bug in your FTP server, or a bug in this Xojo FTP client:

[Andrew_Lambert]

@Mike_D

I’m not sure if this is bug in your FTP server, or a bug in this Xojo FTP client:

It’s a quantum superposition of both and neither.

RFC 959#5.4 states:

The communication between the user and server is intended to be an alternating dialogue. As such, the user issues an FTP command and the server responds with a prompt primary reply. The user should wait for this initial primary success or failure response before sending further commands.
[…]
One important group of informational replies is the connection greetings. Under normal circumstances, a server will send a 220 reply, “awaiting input”, when the connection is completed. The user should wait for this greeting message before sending any commands. If the server is unable to accept input right away, a 120 “expected delay” reply should be sent immediately and a 220 reply when ready. The user will then know not to hang up if there is a delay.

Arguably, the server is violating the “alternating dialog” by sending more than one response in a row.

But, section 4.2 allows multi-line responses!

There will be cases however, where the text is longer than a single line. In these cases the complete text must be bracketed so the User-process knows when it may stop reading the reply (i.e. stop processing input on the control connection) and go do other things. This requires a special format on the first line to indicate that more than one line is coming, and another on the last line to designate it as the last. At least one of these must contain the appropriate reply code to indicate the state of the transaction. To satisfy all factions, it was decided that both the first and last line codes should be the same.
Thus the format for multi-line replies is that the first line will begin with the exact required reply code, followed immediately by a Hyphen, “-” (also known as Minus), followed by text. The last line will begin with the same code, followed immediately by Space , optionally some text, and the Telnet end-of-line code.

So, arguably the server is sending a properly formatted multi-line reply. The client erred in treating it as two separate responses.

But, arguably 220 shouldn’t ever be a multi-line reply. The other multiline replies are all direct responses to a previous command (“STAT”, “FEAT”, etc.), so the client is expecting it. The server erred by sending a multiline initial response.

But, arguably the client should be prepared for any response to be multi-lined since that’s not explicitly forbidden. The client erred in making multiline responses a special case.

Etc. It’s a bottomless pitt.