UrlConnection download a file using range

Hi,
while I download a file from scratch, Activity Monitor > Memory for my app remains pretty stable.

Yet, after stopping a download process, after I start downloading the remaining portion of the same file, Activity Monitor > Memory keeps increasing more or less according to the amount of “newData” that I write to file in URLconnection > ReceivingProgressed (below).

So I think that I’m doing something wrong.
Suggestions appreciated. Thank you.

In a pushButton:
[codeSub Action() Handles Action
dim i as integer = theFile.Length - 200
dim pSt as string = “bytes=” + format(i, “#”) + “-”
mURLConn.RequestHeader(“User-Agent”) = mAgent
mURLConn.RequestHeader(“Accept”) = “/
mURLConn.RequestHeader(“Range”) = pSt
resumingDownload = true
mURLConn…send(“GET”,theFile)
End Sub
[/code]

In the following snippet, instead of “dim gg as BinaryStream” I tried also using gg as a property, so not to continuously open and close the stream. But the memory still kept increasing.

Sub ReceivingProgressed(bytesReceived As Int64, totalBytes As Int64, newData As String) Handles ReceivingProgress if resumingDownload then dim gg as BinaryStream = BinaryStream.Open(theFile, true) gg.Position = gg.Length gg.Write newData gg.close end if

Carlo, what version of xojo are you using and on what version of the os?

Using 2019r3.1 I see the memory increase both when using and not using range on macos and on windows it instantly allocates the memory based on the expected size of the download. This is probably a bug in the urlconnection async code as it only really needs to be storing the small amount of data received between the firing of the ReceivingProgressed events.

If this isn’t what you’re seeing, could you post a simple demo project showing the problem as our code might be different and that might be causing the problem not to show.

Here’s a link to some online test files that you can point to during testing http://ovh.net/files/ (the top half of this page) if you don’t want to host the files yourself.

Julian,
I’m using 2019r1.1 and r.2.1 on Catalina (10.15.4).
I uploaded on dropbox a sample, with some verbose explanation.
https://www.dropbox.com/s/jw581gq7k55zyja/URLConn%20Test.xojo_binary_project?dl=1

From what I do and what I say there, you’ll see that the issue is that, when I download without interruptions, the code in Start button is: urlSocket.send(“GET”,TextField1.text,theFile)
In this case, ReceivingProgressed event does not have to write anything.

But if I stop the process and resume it, then the code in Resume button becomes: urlSocket.send(“GET”,TextField1.text); i.e. no more “theFile”.
Therefore I have to use ReceivingProgressed in order to add/write newData to the file. And by this, memory increases.

Do you know if --when resuming a download process-- there is a way to use urlSocket.send(“GET”,TextField1.text,theFile) in such a way that newData get added at the end of the file?

Thanks.

Thanks Carlo the demo was perfect, I see what you mean now, yes Send with a FolderItem seems to behave differently internally to Send without a FolderItem specified.

When a FolderItem is specified the file is sent directly to disk and the framework doesn’t allocate a block of memory that increases in size to store the file internally. When you resume and don’t specify the FolderItem, the framework is using a different internal mechanism to download the file and seemingly incorrectly slowly allocating the full size of the file (on mac) before its downloaded even though the mechanism is async and this imho shouldn’t be happening.

I can’t think of a way around this at the moment as Send with a FolderItem will always overwrite the file (feature request for an optional Append parameter?), the only solution I can think of is when you resume and the file already exists increment the filename e.g. “flags.zip-1”, “flags.zip-2” etc. for each resume, when you have the complete file, check for the existence of files with -1 -2 etc and append them onto the end of the original file. This would mean you could continue to use Send with FolderItem and keep your memory footprint to a minimum.

Failing that you could download only using the Send method without supplying a FolderItem and both actions would act the same by slowly increase memory usage (on mac). This shouldn’t really be a problem if the downloaded files aren’t crazy big.

I think that the slow memory increase on mac (instant on windows) is a bug when calling Send without a FolderItem so I’ll make a demo and put a ticket in about it. If/when it might be fixed is anyone’s guess.

Thanks Julian for answering.

Yes, I had come to the same conclusion, but at the end I thought (and I still think) that one may feel reasonably happy with things as they are now.
I had also thought of submitting a feature request as you yourself mention (optional Append parameter); but I’m pretty sure I won’t be able to see it in my lifetime (smiley).
As for the memory issue, maybe a socket.flush could have made a difference; but urlconnection does not have a flush or purge method as other sockets have.

I’ve just put the ticket in regarding the memory issue <https://xojo.com/issue/59626>, posted here for future search.

Sorry for the delay Carlo, I don’t know if you know, but when moving from a paint even to a picture buffered system you can simply move the code from the paint event and into a method where you dim g as as graphics and point the picture’s graphics property to the g variable there by making all your existing code work as it did before.

Julian, thank you for putting the ticket in Feedback. Let’s hope…

If/when you have time, would you slightly expand on this topic? I’m a little confused about the relationship among the two topics (Urlconnection and paint event). But I always appreciate new insights. Thanks again.

Sorry Carlo, ignore the comments about paint, my brain is wonky, that was meant for another thread :slight_smile:

Nothing to be sorry about. It was an out-of-topic interlude.