I’ve had to change a lot of HTTPSockets to URLConnection lately due to issues with some servers. Might want to switch over to URLConnection since it’s easy and more up to date. Also http/https could be a problem.
The version number within a HTTP request is the version the server should use for the response.
The version number within a HTTP response is the maximum version the server supports.
A server responding with HTTP1.1 must support all of HTTP1.0
A server responding with HTTP1.1 does not have to support all of HTTP1.1
HTTPS encryption protocols are a different matter entirely. All versions of SSL and TLS prior to TLSv1.2 are considered broken. HTTPS enabled sites have been steadily turning off the earlier SSL/TLS versions. The main reason I upgraded my Real Studio 2012 license to Xojo was to get access to TLSv1.2
Testing with user agent string left empty still gets results from the server, so you would seem to be okay there. (Although as Michel implies, some of these services do expect something to be set in there.)
Looking at the server, SSLv2, SSLv3, TLS 1.0 and TLS 1.1 are all disabled, TLS 1.2 is enabled. So quite possibly the version of Xojo (Real Studio) you are using has become the issue, ie: now outdated. Might have to upgrade!
Certainly not how its supposed to work IF the RFC is implemented correctly
But experience has shown me that not every server follows the RFC faithfully and that some just refuse to respond if you make a request with HTTP/1.0. I have encoutered some wher I had to do nothing but switch the header to say HTTP/1.1 and away things went.
Much in the line with Norman’s experience, we do know that many servers don’t respond to HTTPSocket. We’ve seen the question come up a number of times here in the forum, and switching to URLConnection almost always resolves the issue. Though it’s supposed to work as described, if you think about it, not responding to bad requests helps protect the server from malicious attacks.
It’s easy to communicate with the servers that require HTTP 1.1 in most cases. URLConnection offers functionality that handles things automagically. Xojo has implemented SendSync on URLConnection which should make the transition much easier. Though, if you can use this opportunity to update your systems to the event driven nature of URLConnection I would recommend doing so.
[quote=477265:@Norman Palardy]Certainly not how its supposed to work IF the RFC is implemented correctly
But experience has shown me that not every server follows the RFC faithfully and that some just refuse to respond if you make a request with HTTP/1.0.[/quote]
Protocols are open to interpretation but this is core functionality. The vast majority of the web is served by Apache, Nginx, Express and IIS, they are all extremely well tested and respond to HTTP/1.0 requests perfectly fine.
Are you sure the problem is/was the server not responding? Perhaps it was the Xojo framework failing to respond.
The HTTPSocket class suffers from a rookie mistake that I documented back in January. Whoever coded HTTPSocket appears to have relied on the content-length header to raise the PageReceived event.
In the first captured response the server does not include a content-length header, due to the semantics of the cgi script adding the content body after the headers have been assembled in the server’s response buffer. The server does not need to tell the client how long the content will be because the end of transmission is inferred: i) By inclusion of the connection:close header. ii) By the server closing it’s socket immediately after sending the response. This is perfectly normal behaviour when HTTP/1.0 is used to serve dynamic content.
When the server ends the conversation, HTTPSocket needs to raise the PageComplete event, to allow the application to pull the content payload from the input buffer. What actually happens is the HTTPSocket sits there, waiting to count bytes that can never arrive, because the server has already closed the connection. Similar to forgetting to clear the last bytes out of serial buffer. A rookie mistake.
Yes because the HTTP/1.1 default is persistent connection. The server leaves it’s socket open unless it receives connection:close in the request header or decides the connection has timed out.
But thats just the thing that Norman is talking about. The original HTTPSocket was written when http/1.0 was what was available, and that spec says that Content-Length MUST be provided by the data sender. We have said numerous times that the old HTTPSocket is only http/1.0 compliant and will never be upgraded.
Now in terms of servers… a lot of time has passed since our sockets were written and the web servers around the world are also moving forward. In fact Ive seen formal declarations of the removal of http/1.0 support from servers over the last few years because the requests are just too inefficient. Its one of the reasons that the new web framework has been upgraded to http/1.1 and will eventually include http/2.0 support… and yes, supporting both 1.0 and 1.1 requests is a royal pain.
My feelings on this are that you are going to either start using a newer version of Xojo (you are 8 years out of date), you will need to use a 3rd party plugin or you will need to write your own client socket.
Let me drive this thread back onto the original topic: Why doesn’t the HTTPSocket get anything?
Answer: Because HTTPSocket doesn’t play well with today’s servers (which, unrelated, may or may not respect HTTP specifications)
Solution: [quote=477333:@Greg O’Lone]My feelings on this are that you are going to either start using a newer version of Xojo (you are 8 years out of date), you will need to use a 3rd party plugin or you will need to write your own client socket.[/quote]
[quote=477326:@Matthew Stevens]Protocols are open to interpretation but this is core functionality. The vast majority of the web is served by Apache, Nginx, Express and IIS, they are all extremely well tested and respond to HTTP/1.0 requests perfectly fine.
Sure - but not every site uses one of those
Some use custom written back end servers that do not respond per the RFC
Just reporting the things I have run into
To keep the show on the road, I know run a cURL GET script with output to a file. I have changed the app to read this file into the same variable that used to receive the output from the http get and all works fine (and much quicker than before)
Is it possible to call cURL straight from the app?
Dim sh As New Shell
Dim cmd As String, outPut As String
cmd = "curl -s --tlsv1.2 http://worldtimeapi.org/api/ip"
//RS2012 sh.Mode = 0
sh.ExecuteMode = shell.ExecuteModes.Synchronous
outPut = sh.Result
Dim rx As New RegEx, rm As RegExMatch
rx.SearchPattern = "(?mi-Us)""dst_offset"":([0-9]?[0-9]).*""dst"":(true|false)"
Dim dstOffset As Integer, dstActive As String
If rm.SubExpressionCount >=2 Then
dstOffset = rm.SubExpressionString(1).ToInteger
dstActive = rm.SubExpressionString(2)
//RS2012 TextArea1.AppendText cstr(dstOffset) + EndOfLine
If dstActive.Lowercase = "true" Then
TextArea1.AddText "DST Offset " + dstOffset.ToString + EndOfLine
TextArea1.AddText "DST is not in operation" + EndOfLine
Good idea using curl. I keep forgetting it is cross platform these days.