force HTTPSocket Downloader to write UTF-8

General
1

All,

I’m trying to download an HTML page to a local file. I’m using the HTTPSocket (see HTTPDownloader example). The downloaded page I’d like to show in a HTMLViewer. This is working fine.

If I load https://www.rentalcars.com/ar in the HTMLViewer, it loads fine even though is NOT in the HEAD of the page.
A downloaded version of the same page, does not contain this information either, obviously, but the downloaded file does NOT render correctly anymore.
I tested this with the Native and the WebKit rendering engine.
If I manually add to the HEAD, all is fine.

This triggers 2 questions:

  1. Is there a way to force the HTML page to be encoded as UTF-8 in case it wouldn’t be UTF-8 while it gets downloaded?
  2. Can I add the to the HEAD while the file is downloaded? If not, the only option I have is to open the HTML file, search for “” line and replace it with " to the HEAD"

I’m sure I’m not the first one running into this issue.

Can anyone help me with good advice, or with sample code that solved the problem for you?

Thank you.

gert

2

It sounds like the page is already encoded with UTF-8 but that Xojo thinks it’s encoded with a different encoding.

Use the DefineEncoding method to tell Xojo to use UTF-8:

[code]
Dim h As New HTTPSocket
Dim page As String = h.Get(“https://www.rentalcars.com/ar”, 0)

page = DefineEncoding(page, Encodings.UTF8)[/code]

3

Thanks Andrew, this solves one part of the problem. But I still have the issue that inside the downloaded HTML file the encoding is missing and therefor the HTMLViewer does not know how to render it. Do you know if there is a way to force the HTMLViewer to read and show the file correctly?

– thanks

4

You need a request header http:

Content-Type
text/html charset=utf-8

https://www.w3.org/International/articles/http-charset/index

5

Derk, thank you for replying. Where should I request this?
I’m setting
DownloaderTarget.SetRequestHeader(“Content-Type:”, “text/html charset=utf-8”)
before I start the download, but this does not change the HEAD in the downloaded file and the HTMLviewer is still not showing the downloaded page correctly. Only the online page is show correctly (what is strange because the encoding info is also missing in the HEAD there…

6

Try this exact value as the value of Content-Type:

text/html; charset=utf-8

7

Browsers try to render iso- blahblah.

Not sure of the rendering has anything to do with encoding. Can you send a PM to me with your mail?
Then i can review the exact source your getting

8

The server is sending the encoding in the Content-Type header, but because you’re viewing a downloaded copy of the file there are no headers for the HTMLViewer to read. If you load the URL in the HTMLViewer directly then it displays correctly.