Help with URL and http socket

General
1

I have a textfield that a user can enter a url into and it will load the html of that website into a textfield.
It works fine as long as someone enters http or https in the beginning, but returns nothing if someone just types “www.psu.edu” for example. So I put a regEx in to check what is entered and it will append “http://” to the url if it’s missing. That seems to work fine as evidenced in the msgBox I get after the regEx search. Even though the url has now been ‘fixed’, it still returns nothing. But if I type “http://www.psu.edu” i get the html. I don’t understand why it’s not working when I fix the url. Here is the code. BTW, I’m using XOJO 2018 r2 on an older Mac.

Dim socket1 As New HTTPSocket

if urlField.text <> "" then

  myURL = urlField.text  

 rg3.searchPattern = "http"

  myMatch3 = rg3.search(myURL)

  if myMatch3 = nil then
    myURL = "http://" + myURL
  end if

 msgBox myURL

  Dim data As String = socket1.Get(myURL, 300)

  HTMLViewer1.LoadURL(myURL)

  textField1.text = HTMLViewer1.EvaluateJavaScriptMBS("document.body.innerText")

else 

 msgBox "Please enter a valid URL."

end if
2

Most websites will not accept an http address. You need to move to https addresses

3

adding an ‘s’ still doesn’t help. it does work fine if I enter “http://www.psu.edu” into the textfield.

4

It may be that HTMLViewer.LoadURL is happening asynchronously, and when you go to fetch the HTMLViewer’s body contents the page has not loaded yet. You should try your Javascript to get the contents in the HTMLViewer’s DocumentComplete event.

5

I don’t understand how it could get the url asynchronously if myURL is not yet established. I tried moving
Dim socket1 As New HTTPSocket
to after the url is established and that didn’t help either.

Not sure how to do this with javascript.

6

Your code is loading myURL to the HTMLViewer, not whatever you’re fetching with the HTTPSocket.

Dim data As String = socket1.Get(myURL, 300)

HTMLViewer1.LoadURL(myURL)

I’m building a demo project now for fun, why don’t you explain what you’re fully trying to do?

7

I’m just loading the html to search for certain things, like email addresses.

8

You shouldn’t need a HTMLViewer if you use URLConnection

var oURL as new URLConnection
var sResult as String = oURL.SendSync("GET", "https://psu.edu")
sResult = sResult.DefineEncoding(Encodings.UTF8)
break

Alternatively, here’s the demo project I was working on that shows how to work with a URL Bar / HTMLViewer: html_test.xojo_xml_project

9

Doesn’t work with XOJO 2018 r2. Getting errors with var and when I change to dim is says can’t find type with this name urlConnection

Thanks for the project. I opened your project in 2020 version (didn’t download the latest yet) that I have and it runs, but won’t allow a url without https. I could do that, but I thought it might be more user-friendly if someone was able to just type “www.psu.edu” to get the website data.

10

You could write a function to prepend the https:// when necessary. I just built the project for fun, I didn’t know your needs.

Newer versions of macOS require that your connections are secure (https). There’s an info.plist flag you can add to get around this (and load http), but I’m not certain what it is or if it still works.

If you can use the 2020r2.1, that would be the edition I recommend. HTTPSocket from the older IDEs just isn’t equipped for today’s web.

11

okay thanks. I’ll give that a shot.