HTTP Get

I have a production app which uses HTTP get and has been working fine for years. Now it has just stopped returning any date.

I am using Real Studio 2012 R2 and the code is

Dim http As HTTPSecureSocket
Dim Data, GetData as string
Data = http.Get (GetData,30)

If I paste the url (ie the contents of the Getdata string) into a web browser it returns the xml data with out any problems.

whats the URL ?
maybe the server that response expects http1.1 which the httpsecuresocket isnt using ?
maybe the server expects a different ssl / tls version which the secure socket isnt using ?

Thanks i will check and see if the server has recently changed what it expects from the Http get

I’ve had to change a lot of HTTPSockets to URLConnection lately due to issues with some servers. Might want to switch over to URLConnection since it’s easy and more up to date. Also http/https could be a problem.

I don’t think that I will be able to use URLConnection as it was not introduced until 2018r4 and I am on 2012r2.

Is there anyother way that I can use HTTP 1.1 GET from my version Real Studio?

The URL as an example call is ;-
https://bb.ringingworld.co.uk/export.php?fmt=xml&place=birmingham&length=peal&pagesize=1&submitted_since=2019-11-01

Perhaps you need to set the user agent string.

That is not how HTTP versions works.

The version number within a HTTP request is the version the server should use for the response.
The version number within a HTTP response is the maximum version the server supports.
A server responding with HTTP1.1 must support all of HTTP1.0
A server responding with HTTP1.1 does not have to support all of HTTP1.1

HTTPS encryption protocols are a different matter entirely. All versions of SSL and TLS prior to TLSv1.2 are considered broken. HTTPS enabled sites have been steadily turning off the earlier SSL/TLS versions. The main reason I upgraded my Real Studio 2012 license to Xojo was to get access to TLSv1.2

Testing with user agent string left empty still gets results from the server, so you would seem to be okay there. (Although as Michel implies, some of these services do expect something to be set in there.)

Looking at the server, SSLv2, SSLv3, TLS 1.0 and TLS 1.1 are all disabled, TLS 1.2 is enabled. So quite possibly the version of Xojo (Real Studio) you are using has become the issue, ie: now outdated. Might have to upgrade!

Regards

My thanks for everyone’s help with this problem. It looks like as Matthew said that I will need to upgrade to Xojo to get this resolved.

Depending on your use case you may be able to avoid the upgrade by utilising a forward proxy,

These two are small enough to be embedded with an application.
tinyProxy
sTunnel

Node.js provides an even easier option.
https://github.com/http-party/node-http-proxy/issues/304

The drawback is yet more dependencies to manage.

Certainly not how its supposed to work IF the RFC is implemented correctly
But experience has shown me that not every server follows the RFC faithfully and that some just refuse to respond if you make a request with HTTP/1.0. I have encoutered some wher I had to do nothing but switch the header to say HTTP/1.1 and away things went.

Much in the line with Norman’s experience, we do know that many servers don’t respond to HTTPSocket. We’ve seen the question come up a number of times here in the forum, and switching to URLConnection almost always resolves the issue. Though it’s supposed to work as described, if you think about it, not responding to bad requests helps protect the server from malicious attacks.

It’s easy to communicate with the servers that require HTTP 1.1 in most cases. URLConnection offers functionality that handles things automagically. Xojo has implemented SendSync on URLConnection which should make the transition much easier. Though, if you can use this opportunity to update your systems to the event driven nature of URLConnection I would recommend doing so.

If you’re developing for modern systems, you will absolutely want to upgrade from Real Studio. There have just been so many changes and so much progress in the past 8 years. This upgrade isn’t always straight forward, and I do offer services to ease the transition.

[quote=477265:@Norman Palardy]Certainly not how its supposed to work IF the RFC is implemented correctly
But experience has shown me that not every server follows the RFC faithfully and that some just refuse to respond if you make a request with HTTP/1.0.[/quote]
Protocols are open to interpretation but this is core functionality. The vast majority of the web is served by Apache, Nginx, Express and IIS, they are all extremely well tested and respond to HTTP/1.0 requests perfectly fine.

Are you sure the problem is/was the server not responding? Perhaps it was the Xojo framework failing to respond.

The HTTPSocket class suffers from a rookie mistake that I documented back in January. Whoever coded HTTPSocket appears to have relied on the content-length header to raise the PageReceived event.

Take a look at the packet captures here, https://forum.xojo.com/57942-httpsocket-and-apache-2-4-42/p1#p471568

In the first captured response the server does not include a content-length header, due to the semantics of the cgi script adding the content body after the headers have been assembled in the server’s response buffer. The server does not need to tell the client how long the content will be because the end of transmission is inferred: i) By inclusion of the connection:close header. ii) By the server closing it’s socket immediately after sending the response. This is perfectly normal behaviour when HTTP/1.0 is used to serve dynamic content.

When the server ends the conversation, HTTPSocket needs to raise the PageComplete event, to allow the application to pull the content payload from the input buffer. What actually happens is the HTTPSocket sits there, waiting to count bytes that can never arrive, because the server has already closed the connection. Similar to forgetting to clear the last bytes out of serial buffer. A rookie mistake.

Yes because the HTTP/1.1 default is persistent connection. The server leaves it’s socket open unless it receives connection:close in the request header or decides the connection has timed out.

But that’s just the thing that Norman is talking about. The original HTTPSocket was written when http/1.0 was what was available, and that spec says that Content-Length MUST be provided by the data sender. We have said numerous times that the old HTTPSocket is only http/1.0 compliant and will never be upgraded.

Now in terms of servers… a lot of time has passed since our sockets were written and the web servers around the world are also moving forward. In fact I’ve seen formal declarations of the removal of http/1.0 support from servers over the last few years because the requests are just too inefficient. It’s one of the reasons that the new web framework has been upgraded to http/1.1 and will eventually include http/2.0 support… and yes, supporting both 1.0 and 1.1 requests is a royal pain.

My feelings on this are that you are going to either start using a newer version of Xojo (you are 8 years out of date), you will need to use a 3rd party plugin or you will need to write your own client socket.

Hmmm…

https://www.w3.org/Protocols/HTTP/1.0/spec.html#BodyLength

Let me drive this thread back onto the original topic: Why doesn’t the HTTPSocket get anything?

Answer: Because HTTPSocket doesn’t play well with today’s servers (which, unrelated, may or may not respect HTTP specifications)

Solution: [quote=477333:@Greg O’Lone]My feelings on this are that you are going to either start using a newer version of Xojo (you are 8 years out of date), you will need to use a 3rd party plugin or you will need to write your own client socket.[/quote]

[quote=477341:@]Hmmm…

https://www.w3.org/Protocols/HTTP/1.0/spec.html#BodyLength[/quote]
So it’s a bug in how we implemented the spec, but we’re still not fixing it.

[quote=477326:@Matthew Stevens]Protocols are open to interpretation but this is core functionality. The vast majority of the web is served by Apache, Nginx, Express and IIS, they are all extremely well tested and respond to HTTP/1.0 requests perfectly fine.
[/quote]
Sure - but not every site uses one of those
Some use custom written back end servers that do not respond per the RFC
Just reporting the things I have run into

To keep the show on the road, I know run a cURL GET script with output to a file. I have changed the app to read this file into the same variable that used to receive the output from the http get and all works fine (and much quicker than before)

Is it possible to call cURL straight from the app?

Yes, you can use the shell class

Dim sh As New Shell
Dim cmd As String, outPut As String

cmd = "curl -s --tlsv1.2 http://worldtimeapi.org/api/ip"

//RS2012 sh.Mode = 0 
sh.ExecuteMode = shell.ExecuteModes.Synchronous
sh.Execute cmd
outPut = sh.Result

Dim rx As New RegEx, rm As RegExMatch
rx.SearchPattern = "(?mi-Us)""dst_offset"":([0-9]?[0-9]).*""dst"":(true|false)"
rm =rx.Search(outPut)

Dim dstOffset As Integer, dstActive As String
If rm.SubExpressionCount >=2 Then
  dstOffset = rm.SubExpressionString(1).ToInteger
  dstActive = rm.SubExpressionString(2)
End If

//RS2012 TextArea1.AppendText cstr(dstOffset) + EndOfLine
If dstActive.Lowercase = "true" Then
  TextArea1.AddText "DST Offset " +  dstOffset.ToString + EndOfLine
Else
  TextArea1.AddText "DST is not in operation" + EndOfLine
End If

Good idea using curl. I keep forgetting it is cross platform these days.

[edit] Added RS2012 equivalent syntax.