Xojo.Core.TCPSocket Sending too much binary data

I’ve got some basic code for sending an SQLite database from an iOS app to a Xojo-based desktop app. It sends it via binary data using Xojo.Core.TCPSocket.

The following code works fine in the debugger but for some reason is sending an additional ~4kb of data when used on the device. On the iOS side:

[code] if SendFileStream<>nil then ///a binary stream that is reading the database from disk
dim byteChunkSize as integer =20000 ///i’ve tried various sizes

///write the db to the socket
do until SendFileStream.EOF
  if byteChunkSize > SendFileStream.Length - SendFileStream.Position then
    byteChunkSize = SendFileStream.Length - SendFileStream.Position
  end if
  
  SendFileSentAmount=SendFileSentAmount + byteChunkSize
  
  me.WriteData(SendFileStream.Read(byteChunkSize))
  
loop

SendFileStream.Close

end if[/code]

On the desktop side I have a classic TCPSocket with the data available event:

[code] dim len as double = me.BytesAvailable

  if len+BinaryCurrentReceived>BinaryTotalToReceive then
    len = BinaryTotalToReceive - BinaryCurrentReceived
  end if
  
  
  if len <= 0 then
    ///we're done
    DoneReceivingFile
    
  else
    
    
    if me.IncomingFileStream<>nil then
      me.IncomingFileStream.Write(me.Read(len))
      BinaryCurrentReceived = BinaryCurrentReceived+len
    end if
    
    if BinaryCurrentReceived>=BinaryTotalToReceive then
      DoneReceivingFile
    end if
    
  end if[/code]

On the desktop and the iOS app, it stores and correctly shows the amount of bytes for the file that have been sent/received at 711,680 bytes. That is the correct file size.

However the file that ends up on the desktop is ALWAYS 715,776 bytes. The SQLite database only partially works and some tables are corrupted in the database.

Any ideas?

[quote=209623:@Tom Iwaniec]I’ve got some basic code for sending an SQLite database from an iOS app to a Xojo-based desktop app. It sends it via binary data using Xojo.Core.TCPSocket.

The following code works fine in the debugger but for some reason is sending an additional ~4kb of data when used on the device.[/quote]

Have you examine the contents of those extra bytes to get an idea of what might be going wrong? And is the data that precedes it actually what you expect it to be? If you use a differently sized file to transfer, does the size difference change?

Thanks Joe.

Yes the two seem to go out of sync in random places. I’ve done a hex dump of both files and they start out at the same spot, then deviate for only a couple of lines, then they are fine for the next several hundred.

Should I open a private feedback request and submit the hex dumps or what’s the best way to respond to this?

Separately when sending a file that is 1,444,864 bytes, it appears on the desktop as 1,444,864 bytes, but the SQLite database is still unusable as the binary seems to be messed up and it’s giving a “database disk image is malformed” error.

It’s strange because it works flawlessly from the debugger but does not seem to work with any file from the device itself.

Here’s the example from the first file of the very beginning, notice lines 00000010-00000050 have differences.

[quote=209629:@Tom Iwaniec]Thanks Joe.

Yes the two seem to go out of sync in random places. I’ve done a hex dump of both files and they start out at the same spot, then deviate for only a couple of lines, then they are fine for the next several hundred.

Should I open a private feedback request and submit the hex dumps or what’s the best way to respond to this?[/quote]

A small sample project that sends predictable data (e.g. 0-9 over and over) that reproduces the problem would be best. While putting that together, you might notice trends in the errors, which would be helpful to note with the bug report.

One other thing to look at is how latency affects the transfer. Apple has a handy ‘network link conditioner’ tool, which can introduce arbitrary levels of latency or bandwidth constraints.

I’m not on my computer right now, so I don’t have a link to that handy and I can’t rule out that the receiving code is incorrect. Where does the BytesTotalToReceive value come from?

One last thought is to use a packet sniffer to verify what actually makes it from the device to the network. It would definitively rule out any bugs on the receiving side.

Joe, on the off chance that there’s code similarity between this and good old IPCSocket, have you had a chance to look at the memory leak bug?
<https://xojo.com/issue/34107>

I’ve done the same thing with a simple file using only characters 0-9 and then did a hex dump & comparison in BBEdit and on a ~7 mb file there were about ~20 discrepancies. Will try to make a simple sample project for this.

this reminds everyone to include a header before sending data, so the receiver knows how much to expect and maybe a Hash (SHA512 for example), so receiver can check if file is right.

Yes the file size is sent beforehand which is how the other machine knows the size to look for.

Joe, if I remember correctly, on iOS devices there is a max send-buffer size that the sockets can deal with. Might the issue be that the Xojo.Core.TCPSocket isn’t handling the buffer size internally and this is causing issues? I can’t find a way to access this information through the class, whereas in the classic TCPSocket we could access BytesLeftToSend.

I’ve created a secondary way of sending the binary data that uses a timer to send chunks at set intervals since the issue seems to be just related to the send buffer. The problem is this won’t solve the issue if there is network latency, and we have no way of knowing (from what I can see) if the send buffer is full.

I’ve created sample desktop/iOS networking apps and created a new Feedback case:
<https://xojo.com/issue/40595>

Am I missing something about the class that would let us know about the send buffer limit or the current buffer size?

Thanks!

shameless bump

Any idea if we can properly/safely send files from iOS sockets?

If as I suppose you are the developer of the program on both ends, why not proceed by chunks ?

The point is that we don’t know how quickly we can send the data, and that speed will be variable based on the bandwidth of the current connection, which is varies widely on a mobile device. It’s not that there is a max file size, it’s that there is a maximum amount that can be added to the socket at any one time and if it goes over that amount it becomes corrupt. The amount in the socket’s outgoing buffer will depend on the speed of the connection, and we have no way of knowing any of that, so we wouldn’t know how large to make the chunks.

It’s very inefficient to try sending chunks at different speeds and then verifying it sent correctly, because the same chunk could continually fail (especially on a non-wifi connection), and the send process could take much, much longer than it should need to.

If I understand right, you get some sort of padding data. That looks like a bug to me, that should be reported.

Until that is fixed, could you not prefix the data with a couple bytes that indicate the length of the data, and in the receiving app cut the relevant part ?

The bug was reported and linked earlier in the thread. Knowing the size of the chunk doesn’t prevent the sending socket from corrupting the data if it exceeds the send buffer limit. I think for now it’s not possible to reliably send data from the iOS socket.

HTTPSocket ?