HTTPSecureSocket instances *NOT* being destroyed when reference count is theoretically ZERO

General
1

Hello:

I have been dealing for some time with frustrating memory consumption in programs that use HTTPSecureSocket. To eliminate causes such as design flaws, unresolved circular references, etc., I have put together a primitive Console project (XOJO 2014, but should work in RealStudio 2012, too) to demonstrate the problem in simplest terms.

The project file is available from HERE . In short, this program’s output should demonstrate (at least on Windows, which is where I write and deploy my software) that HTTPSecureSockets operating with .secure=TRUE NEVER seem to have their Destructor() methods called when the instances should be considered out of scope. The result is an proliferation of detritus in the Runtime (as measured via Runtime.ObjectCount) and therefore memory “leakage”.

If I am “doing it wrong”, I would absolutely love to hear this – please tell me if you see that that is the case! I would be grateful to learn that there is some correct method that I am missing for releasing HTTPSecureSocket instances. However, I believe that the three scenarios in my test program belie that possibility.

Here’s a summary of what scenarios my XML project tests, when compiled/run, and what its results are (at least for me!):

  • TESTING METHOD:
    • a subclass is created for HTTPSocket, and one is also created for HTTPSecureSocket
    ••• the subclasses add only one thing: Destructor() methods that use StdOut.writeLine() to indicate that they are being called

• a URL is synchronously pulled via GET three times in a row
••• Runtime.ObjectCount is checked before and after each iteration and then compared to see if there is a net increase or decrease in ObjectCount (where an increase would indicate a memory leak)
••• is the Destructor for the socket object called when it is finished its work, purged and closed, and then set to NIL?

  • TEST 1: HTTPSocket accessed the HTTP version of the URL 3 times in a row
    •••••• RESULT: Destructor() methods ARE called, and there is no net increase in Runtime.ObjectCount with each iteration

  • TEST 2: HTTPSecureSocket is used [with .secure=true and .connectionType=TLSv1] to access the HTTPS version of the same URL 3 times in a row
    •••••• RESULT: Destructor() methods are NOT called, and there is a marked net increase (of 18) in Runtime.ObjectCount with each iteration

  • TEST 3: HTTPSecureSocket is used [with .secure=false] to access the HTTP version of the same URL 3 times in a row
    •••••• RESULT: Destructor() methods ARE called, and there is no net increase in Runtime.ObjectCount with each iteration (besides an increase in 1 with the first iteration)

Thoughts?

2

Feedback case ID?

3

This is literally the first time I have ever talked to anyone (either in the Forum or at Real/Xojo) in a decade.

Is a feedback ID a pre-requisite for posting this kind of question?

4

no, but it helps engineers. Without a case number, nothing will happen…

SSLSocket has leaks which are reported. This may be related.

5

I to think this is related to feedback #31706

6

I agree with Jordan - this is definitely related to <https://xojo.com/issue/31706>. Please sign on and add your example project.

7

Timothy, did you try to set secure = false on the end of the socket usage?
Does that help?

8

Dang it, guys. Sorry… yeah… My case is basically a clone. I’ve never even ever had to install Feedback. I just did and found case 31706.

PS: As noted in my original message, my test project conducts three tests: 1 with HTTPSocket on its own with no leaks; 1 with HTTPSecureSocket with .secure=TRUE that DOES have leaks; and then 1 with HTTPSecureSocket with .secure=FALSE that doesn’t really leak (maybe just one object).

So do I just reply and upload my XML file?

9

Click the Pencil icon on the bottom of the Feedback app if you want to add notes and then click the paperclip to attach your file to the case.

10

I have updated case 31706 with my project file and my notes.

WOW. I see now that this has been going on for some time, but apparently doesn’t register as important enough to track down and fix with haste. This is surprising, given holes like Heartbleed coming to the fore, and that these these are the days of Edward Snowden’s revelations shaking everyone out of their complacency with Web security.

Because of all this, these days, pretty much every service has to be consumed with HTTPS, and pretty much every service you might make has to be delivered in HTTPS, too.

I’m not sure what it says that XOJO’s implementation of HTTPSecureSocket has been provably unscaleable for quite a long time and that noone has considered it important enough to track it down and slay it once and for all.

This is going to have to be fixed, or it is going to be impossible for high-quality, high-throughput, HTTPS servers and clients to be written in XOJO without resorting to proxies or outside libraries !

Sorry to have wasted your time, guys. I see now that I’m sadly just one voice in the choir.

11

Or, the more likely scenario is that not so many of us are bothered by this. In my case, I open the socket when the user logs in and then it’s destroyed on the user’s exiting the app. No problem.

OTOH - have you tried Christian’s idea of setting Secure = False just before you close the socket?

12

Please forgive my tone. I vented some frustration.

Also, like a dummy, I didn’t read Christian’s note clearly enough, apparently. That is an intriguing idea, setting the .secure to FALSE before trying to close. I’ll try that and advise.

Thanks

13

No such luck, unfortunately.

I modified my “socketTest.xml” project to set the socket to .secure=FALSE just before calling .close() and setting it to NIL. I still have Destructor() methods not being called and ObjectCounts still increasing.

I am using secure HTTPSecureSocket in TLSv1 to synchronously .get(“https://www.apache.org/404.html”,2), in this iteration of the program – and I call .secure=FALSE right before closing. Here is the same output still:

Every line that says “SHOULD have just called a Destructor” is normally preceded by a line like “Calling Destructor / TestSocketHTTPS.Destructor” if Destructor() was actually called. In the HTTPS cases below, it was not.

.

.

[code]Getting the same URL via HTTPSecureSocket for 3 cycles… note that NO destructors are called and ObjectCount increases with each iteration

==============================

404 Not Found

Not Found

The requested URL /404.html was not found on this server.

Apache/2.4.9 (Unix) mod_wsgi/3.4 Python/2.7.5 OpenSSL/1.0.1g Server at www.apache.org Port 80

==============================
SHOULD have just called a Destructor

==============================

Leak: ObjectCount increased by 24

==============================

==============================

404 Not Found

Not Found

The requested URL /404.html was not found on this server.

Apache/2.4.9 (Unix) mod_wsgi/3.4 Python/2.7.5 OpenSSL/1.0.1g Server at www.apache.org Port 80

==============================
SHOULD have just called a Destructor

==============================

Leak: ObjectCount increased by 18

==============================

==============================

404 Not Found

Not Found

The requested URL /404.html was not found on this server.

Apache/2.4.9 (Unix) mod_wsgi/3.4 Python/2.7.5 OpenSSL/1.0.1g Server at www.apache.org Port 80

==============================
SHOULD have just called a Destructor

==============================

Leak: ObjectCount increased by 18

==============================[/code]

14

Well, while SSL is important to us, the SSLSocket implementation is slow (32231), insecure (2820), uses lot of memory (32087) and leaks memory (31706).

15

Any idea why these Feedback cases are Reviewed but none are Verified or Scheduled?
<https://xojo.com/issue/32231> sslsocket slow
<https://xojo.com/issue/2820> Certificates with sslsocket, httpsecuresocket
<https://xojo.com/issue/32087> sslsocket uses tons of memory
<https://xojo.com/issue/31706> SSLSocket leak with ServerSocket

16

Because there’s not a lot of pressure on Xojo to get them fixed? Snarkiness aside, I suspect it needs a rewrite and there’s just not a whole lot of extra bandwidth for it right now. iOS, LLVM, 64 bit are all in motion with iOS presumably about ready to hit alpha.

17

Yes, I had to give up using SSL sockets on my app as well. I have a webserver I wrote specifically for transferring large files and Chrome will crash it almost immediately.

<https://xojo.com/issue/31800> ssl socket errors with Chrome browser

I also had problems with an FTP tool I wrote where several newer clients wouldn’t connect because of TLS compatibility problems of some sort.

<https://xojo.com/issue/31126> Add TLS v1.1 and TLS v1.2 Support to Xojo

18

You’re probably right, otherwise they would get more attention.

As much as I like to see forward progress, I’m a big fan of addressing known issues in what’s already available. Hopefully Xojo becomes profitable enough to bring on more engineers in the not too distant future so there is enough bandwidth to react quickly to existing issues while also moving the product family forward.

19

I have now re-written the part of my product that was communicating via HTTPS… and now I have ZERO detectable memory loss or CPU usage… AND the performance of my outbound HTTPS requests is now noticeably faster than it was in the original format!

How can this be possible? Have I discovered a magical way of making the XOJO HTTPSecureSocket behave itself?Sadly NO… but for those who even care, here is what I ended up doing to resolve my woes:

  • I wrote a Class that routes all of my HTTPS requests (GET’s and POST’s) to the WGET binary via an asynchronous Shell.

Yes, you have read this all correctly: adding the convolution of proxying through an external hidden shell session to launch an external WGET process, collect its StdOut data, and then return the data to the requesting method, is demonstrably faster and more memory-efficient than using the native HTTPSecureSocket implementation.

In case it helps put into context why I care so much about being able to create and dispose of so many sockets with no room for resource wastage or unnecessary delay, here is an overview of the networking element of my application: it listens on insecure HTTP (via ServerSocket pool) for connections from trusted clients on a private LAN and then, for every connected session, can possibly be instructed to GET and/or POST up to dozens of times to a SOAP service published on the public Internet and accessible via HTTPS only.

Many thanks to many notable people for their comments here and, in general, their ideas and overall effort relating this long-festering deficiency.