SSLSocket buggy?

Hi all!

I have a server and a client app. The server uses the ServerSocket with SSLSockets. I set the SSLSocket to TLSv12 and assign a certificate (self-signed). All fine, works great.

The problem is, if I send much data (larger files) over such a secured connection, the client says he sent all data (has written all data to the socket buffer) but on the server side sometimes not all packages are received. In this case the server’s socket has only 80 % of the data.
I first write a small, fixed sized header that contains the expected number of bytes (the file size). This way the server knows how much data should be come in for a single file.

I’m pretty sure that my code is fine so far, because if I use the SSLSocket as a normal TCPSocket (simply set SSLEnabled to false) it all works fine. Files from a few bytes to many hundred megabytes are transferred fine w/o any erros.

So is SSLSocket buggy (in conjunction with ServerSocket)? Any ideas or are there known issues?

Please read the server socks. You won’t get all data in one dataAvailable event.
An SSLSocket is not as a TCPSocket since it has to do encryption/decryption of blocks (mostly) which causes even more dataAvailable events to happen, even sometimes without data.

The problem is, there are DataAvailable events missing, otherwise my data where complete :wink: As I wrote, it works sometimes, sometimes not.
As I understand this whole SSL stuff, I have to set properties correctly and whole encryption/decryption process is completely transparent to my app.
I simply have no idea where the problem could be (in my code).

Did you add your own buffer? That’s a thing we always do:

DataAvailable event:

Var incoming As String = ReadAll(Nil)
Self.mBuffer = Self.mBuffer + incoming

Self.ProcessData

Process data is a method that checks for data and reads if possible, this allows for re-entry (can happen in sockets) but makes sure all data is in order in your buffer.

Note that XojoWeb also uses the SSLSockets and SeverSocket, and i’ve never had any issues with missing data. You can check the Express app (based of aloe express) and use the code there:

You may also want to try URLConnection instead of SSLSocket. The URLConnection class uses HTTP 1.1 while I think the SSLSocket uses HTTP1.0 as the protocol. In general, I recommend URLConnection now.

I don’t recall having the issue you state, but in my docs for SSL there includes this text:

Writing to a socket is done asynchronously. This means each time the Write method is called, the data passed goes into a buffer in memory before actually being sent and then removed from the buffer. Once the socket has finished sending the data in the buffer to the computer at the other end of the socket connection, the SendComplete event handler is executed. This allows you to know when all of the data has really been sent.

Calling Read, ReadAll, or Lookahead may not fetch all of the data in the internal buffer. This is because SSL needs to read data in blocks (due to the cryptography), and it may not have a complete block in the buffer. For example, there may be 700 bytes available in the buffer, but SSL can only decrypt 512 bytes due to the remainder being an incomplete block. What occurs in this case is some data may remain stagnant in the buffer. When more data comes in, the DataAvailable event handler is called. If there are no more DataAvailable events, then upon disconnection, additional DataAvailable event will be issued to let you pick up any stagnant data that SSL can give us back. There are two things to watch out for because of this:

  1. If there is not sufficient data for SSL to decrypt, you may get a DataAvailable event but no data.
  2. Calling SSLSocket.Close may execute DataAvailable events.

Thanks @DerkJ !

This is good to know. But maybe the issue is on the client side. I think I create a sample app and try to reproduce the issue.

BTW: In this case I don’t need a buffer; I write the data directly to disk via a BinaryStream. And I’ve never had a DataAvailable event with no data in Xojo. As I wrote, this all works fine on any file sizes with SSLEnabled = false.

Thank you @Douglas_Handy, but I don’t want to use URLConnection.

Keep in mind that writing to an sslsocket does not mean that the data has been sent. When you write, it’s written to a local buffer and sent as the sending and receiving sockets deem necessary.

I suspect what’s going on here is that you’re letting the sending socket go out of scope before the buffer completely empties. Make sure you hold onto a reference until the SendComplete event fires.

1 Like

Also, keep in mind that the DataAvailable event is reentrant. Derk’s suggestion to use a buffer may very well be necessary except that I’d suggest using a Timer to indicate when to do the processing instead of calling the processing method directly.

Because Strings can hold binary data, I usually use a string array for this. In DataAvailable I simply do this:

Buffer.Add ReadAll(Nil)
ProcessTimer.RunMode = Timer.RunModes.Single
ProcessTimer.Reset

Then in the Action handler of the timer you can loop through the buffer from bottom to top to write the data to disk.

ProcessTimer can have a relatively low Period, but enough that if more data comes in, that it does get delayed a bit. There are other optimizations you can do to make this operate more smoothly, but I can’t do them on my phone.

1 Like

Thanks @Greg_O !
My problem is, that sometimes simply DataAvailable events are missing and the socket don’t get all data. But the client completes with a SendComplete event.

I’ve a created an issue #69235 with two sample projects to demonstrate it and hopefully someone is running into it.

Did you try my suggestions though? I suspect that your “lost” DataAvailable events will be solved by not disconnecting too early and by using a cache.

Yeah, you’re going out of your way to shoot yourself in the foot on that DataAvailable event. I’ll modify that and upload a version illustrating my suggestions.

I don‘t disconnect anywhere.

Shoot in the foot? I simply write the data to disk instead of storing it in internal buffer. And there is no processing needed. I simply want data in a file, what mostly works fine. The question is „why mostly“ and not always. Anyway, I would be happy if you could help.

What am I missing? What is the downside of URLConnection (if on at least 2018R4 where it was introduced)?

Always store received packets in fast buffers as soon as received and free the system to exchange more packets right after. Then forward processing such data to another lower priority task, even if such task is just to write raw data to disk (slow and blocking place to store packets).

That’s simply because of the re-entry, now is just the first time you noticed. It’s a thing that offcially not exist in xojo, actually some platforms look like they call something on a parallel thread. What happens if you test with only:

In DataAvailable:
System.debuglog currentmethodname + ", " + Me.ReadAll

Do you mis any data?

As I wrote earlier, there’s no problem at all if I don’t use the SSL stuff.

Yes, I miss data and the socket gets stuck even if I don’t store the data anywhere. You can check yourself with the sample files provided.

It doesn’t matter. SSL connections are slower by nature as they encrypt/decrypt while doing its job. Don’t expect the same timing and exact behaviors.

1 Like