Web Navigation and retrieval

Getting Started
1

I would like help on determining which web service requests I need to put together to accomplish the following:

  1. Open a web page in a browser or HTML Viewer (I need to visually inspect the results)
  2. Send a number to an input field on the page(HTML field name “ID”)
  3. Execute the Find button (“'ctl00_findButton”)
  4. Wait for the page to be returned
  5. Read 10 different data fields (HTML names “Data1”, “Data2”, etc.) into an array WData(10)

Is it faster to download the content of the page and then retrieve the HTML fields by name or just parse the data as a string?

If I need to do a similar process on a second website, would I open a second URLConnector with a different name?

Code is executed from a Mac Desktop app with up-to-date versions of OSX and XOJO.

Thank you for any assistance.

2

It sounds like the website you’re automating is running on some kind of framework. While you technically can do everything with URLConnection it’s going to be challenging and I’d instead recommend you do it all in a DesktopHTMLViewer using Javascript.

You’d load your page, and then come up with a script to fill in the number in your ID field and click the find button, then run it with .ExecuteJavascript. Once the page has updated you prepare another javascript to extract the 10 fields you want and send them back to xojo using executeInXojo and capturing them with the JavascriptRequest event.

3

Or ask the website provider if they have an official API to get those values…

4 Likes
4

No API available - I only need a few of the many fields. Thank you Christian! I was floundering for a starting point. There are also a couple examples of XOJO with javascript that I can work from. Thank you for your detailed answer.

5

Not sure what OS you’re on, but this is the kind of thing that Microsoft Power Automate is pretty good at accomplishing with no coding required. You can then export the data into a csv or database to pass along to a companion Xojo app.

6

Mac OS - I have already been successful in running a javascript to log into the website, so I am just going to copy from that example.
Thanks.

7

Christian, I now have an HTMLViewer window and have successfully logged in and copied the text of the page to parse into values. But I can’t figure out how to get individual values more efficiently from the HTMLViewer.
I have HTML of:
<a id=“ctl00_MainContentId_EmailLink” … >xxx@yyy.com
but:
Var c as New Clipboard
c=View.ExecuteJavaScriptSync(“document.getElementById(‘ctl00_MainContentId_EmailLink’).textElement”)

Is returning nothing. What am I missing?

8

It’s tough to say for sure without actually seeing the code.

But one thing I’ve noticed that happens a lot is that your javascript ends up in a race condition with the page’s javascript that populates the values (whether it’s AJAX calls or some other framework, etc).

Your script may be working correctly, but it’s reading the values before they are set by the website.

You could use a timer to keep testing those elements until they are not empty, or maybe try a MutationObserver.

9

My code is running a minute or more after the page opens. What I just realized that is happening is that it is opening a new window of the same type and maybe the script is running against that window. Even the code “UWBrowser.SetFocus” initiates a new window instead of running the java against the original UWBrowser window. How can I address the original window?