Yes. Doing all of the work in the DataAvailable event is what’s causing you trouble. Here’s an example of why this is a problem for you:
DataAvailable fires, the code begins to run. Lets say it gets as far as setting the bytes_avail, but when it calls System.DebugLog, another DataAvailable event occurs. Because all of these events occur on the main thread, now the second event is the primary running event. Because the bytes didn’t get read in the first event, bytes_avail is now twice the length it should be. System.DebugLog is called and the context switches back to the first event. This is called reentrancy.
This is why I’m saying that you should only write data into a buffer using DataAvailable and then process in a method called from a timer.
Right. SSL is a little different. But to be fair, the reason you’re seeing such variability in your results is that it’s dependent on how busy the CPU is doing other things and how large the transfers are… The busier the CPU is and the larger the transfers are, the worse this problem will get, regardless of whether you’re using a TCPSocket or an SSLSocket with Secure set to True or False.
Maybe it’s me but this could/should have been avoided by design, say the DataAvailable event should have been blocked from context switches (some kind of pragma?).
Anyway the only way to get around this annoyance is to go around it or use URLConnection (uses system libraries).
No, using a pragma will not prevent the data from coming in… and that’s a great way to miss a DataAvailable event anyway. If you’ve told the system not to switch contexts for anything, you could easily miss the last event.
I think his problem is on the server side though, in which case URLConnection doesn’t fix the problem.
i meant something different. More like the context switch is prevented but the incoming data will be in the buffer for the next event call since this is system based buffering (i hope?). Then internally there is an counter that one or more DataAvailable was skipped and is called (context is switched) after the first actually was handled.
To be clear, the Xojo web framework uses a ServerSocket and TCP/SSLSockets for talking to browsers. It uses the same techniques that I described here, except that if the headers indicate that the request is above a certain size (like for a file upload) the request is actually written to disk instead of to memory. A lot of transmission issues like this just went away when that was changed.
The real server app runs as a Windows Service and never on a Mac. I’ve tested now with the sample app under macOS Big Sur and you guess what, I never run into this issue.
So, on Mac it works w/o any problems and also on a physical machine with Windows Server 2022. The tests that showed the issue of missing DataAvailable events are done on a Windows Server 2019 in a Parallels-VM on an MacBook Pro. Tests go on… Thanks to all and sorry for the trouble!
Thats awful news. You are just confirming that you are building a house of cards, an app that depends on the speed of the file system instead of a propper buffer implementation
Just wait to the server being busy with another proccess and there is where you Shoot in the foot with this. Having random chunks lost, corrupted files in production.
A final word from me, because I’m the OP and can’t believe that the code that handles the DataAvailable event is a time critical thing to receive all data correctly. I doubt it for the simple reason because my code works for years and has transferred hundreds of gigabytes of data w/o a single problem for 10+ users.
We have a dedicated application for stress testing that sends different files ranging from a few KB to several hundred MB to the server at random intervals. The file content is preceded by a header that contains, among other things, a checksum of the file, which the server verifies after receiving the data. This is software that has been in use for years and works without any problems.
For example, I can put a delay in a DataAvailable event and still get all the data without errors, just slower:
// Before using ReadAll() or Lookahead().
var ___t as double = System.Microseconds() + (1000 * 1000) // one second
do
App.DoEvents(1) // You can also use Thread.YieldToNext().
if ___t < System.Microseconds() then exit
loop
I didn’t say it was time critical, I said it was reentrant. What you were running into here is not a bug in Xojo, and needs to be planned for regardless of even if it seems to work in your simple single-connection testing environment. It is going to fail for you in the future on real hardware, just like it does in a VM now. You will see the effects of this more often as you have more and more of simultaneous connections to your app and will then need to rethink your approach. I’m talking from experience here, and am simply trying to save you time later.
What exactly are these effects, data loss? That was not my main problem, by the way, but an effect of the stuck socket.
If what happens in the event is not time-critical, what is the point of referring to a fast buffer?
As I wrote above, I can reproduce the problem in the Parallels VM - exclusively in the VM - without buffering any data in the DataAvailable event.
Well you said “stuck socket” but you also said you were not getting the last bit of data, that the data received was smaller than you expected. What I’m saying is that what you’re encountering is data loss. Again, as your app gets more and more busy, because socket events always fire on the main thread, you’re going to have more and more incidences of the next DataAvailable event for a particular socket firing before you’ve finished processing the data in the current one. If your code doesn’t guard against that you’re going to get caught by it.
And now I’m done explaining this. I’ve tried three different ways and while I did want to help you avoid the brick wall you’re driving towards, you seem fine with ignoring all the advice you’ve gotten here. I only suggest that you bookmark this thread so when things do start acting oddly when your app gets overloaded, that you can come back and get the info you need to solve it.
That is NOT what they are suggesting. READ carefully.