HTTPSocket- redirect to Self?

General
1

I had some code that worked about a year ago to scrape a website bit have not used it recently…

The code to get the HTML is simply:

HTML = DefineEncoding(Get(theURL, kTimeOut).Trim, Encodings.UTF8)
It used to work… But now I get a redirect…

So I changed the code to:

HTML = DefineEncoding(me.Get(theURL, kTimeOut).Trim, Encodings.UTF8) Select Case me.HTTPStatusCode Case 301, 302 theURL = me.PageHeaders.CommaSeparatedValues("Location") HTML = DefineEncoding(Get(theURL, kTimeOut).Trim, Encodings.UTF8) End Select

But the redirected URL is the same as the original one and I get getting a redirect!!! This is the URL I have been testing with:

http://webbook.nist.gov/cgi/cbook.cgi?ID=71-43-2&Units=SI&cTP=on&cGC=on

Yet when I paste that exact URL in a browser, the page shows correctly and the address bar still shows the original URL … I have no idea what the problem is … any ideas how to get this to work again?

Thanks,

  • karen
2

It’s redirecting HTTP to HTTPS.

3

That’s not it… I had accidentally changed that while trying to figure it out…

Here is the input URL:

https://webbook.nist.gov/cgi/cbook.cgi?ID=71-43-2&Units=SI&cTP=on&cGC=on

Here is the URL i get from the location header when there is a 301 error code:

https://webbook.nist.gov/cgi/cbook.cgi?ID=71-43-2&Units=SI&cTP=on&cGC=on

And the returned HTML from the socket is:

[code]

301 Moved Permanently

Moved Permanently

The document has moved here.

[/code]

I really need this to work… any ideas would be appreciated!

  • karen
4

Ok even though the URL addressed had HTTPS , as i was using the HTTPSocket it cited as if the URL as just an HTTP address…

I then switched to using an HTTPSecureSocket and got a 102 error…

It was only when I used HTTPSecureSocket AND set

HTTPSecureSocket .Secure = True
And HTTPSecureSocket.ConnectionType = SSLSocket.TLSv12

That it started working again…

So Andrew was right it WAS redirecting to HTTPS … what confused me was that I WAS using an HTTPS Url … but with an HTTPSocket.

A year ago that did not matter… now it does so something changed on teh server side… I am a happy camper!

Thanks Andrew

  • karen