External API asks for ISO 8859-1 through a POST request, characters always get wrong!

Hello,

I’m polishing up an e-commerce module for PagSeguro as shown in this post .

Everything is all and well but an annoying encoding issue. Every accented character gets corrupted after transmission:

As an example, the string [quote]Gerao, Comunicao e Gesto - Rua Baro do Amazonas, 412, Petrpolis[/quote] is turned to [quote]Gerao, Comunicao e Gesto - Rua Barão do Amazonas, 412, Petrópolis[/quote]

Pagseguro API docs states that any request to their site should use Contet-Type: application/x-www-form-urlencoded; charset=ISO-8859-1.

The way I’m making the request on Xojo side is like this:

All variables to be passed in the web request are set on the dictionary Session.saleitem (property of Session). This dictionary is appended/handled in several parts of the app to produce valid information as per PagSeguro docs.

When on the sending part, code (partial, Session.saleitem is a rather long dictionary…) is shown below:

Dim pagador As New HTTPSecureSocket
pagador.requestHeaders.DeleteAllHeaders
pagador.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded; charset=ISO-8859-1")

'For testing purposes, "shippingAddressStreet" is defined as:
Session.saleitem.Value("shippingAddressStreet") = "Gerao, Comunicao e Gesto - Rua Baro do Amazonas, 412, Petrpolis"

'The result is the same whether the below line is
'Session.saleitem.Value("shippingAddressStreet") = Session.saleitem.Value("shippingAddressStreet").StringValue.ConvertEncoding(Encodings.ISOLatin1)
'or
Session.saleitem.Value("shippingAddressStreet") = Session.saleitem.Value("shippingAddressStreet").StringValue.DefineEncoding(Encodings.UTF8)

pagador.SetFormData(Session.saleitem)

pagador.Secure = True
pagador.ConnectionType = HTTPSecureSocket.TLSv12

AddHandler pagador.PageReceived, WeakAddressOf Venda_Confirmada
AddHandler pagador.Error, WeakAddressOf Erro_Venda_Confirmada

pagador.Post("https://ws.sandbox.pagseguro.uol.com.br/v2/transactions")

Any ideas on how to correct that?

Thanks!

Did you do EncodeURLComponent as mentiond with application/x-www-form-urlencoded?

Hello @Marius Dieter Noetzel !

Thanks for your input. I just tried ‘EncodeURLComponet’ to no avail. Still getting strange characters. The data is sent as a POST form, not in the URL. If I understood correctly what ‘EncodeURLComponet’ does, it will only affect the URL line on a GET request.

Your answer tipped in a new direction. I looked the Language Reference and found out about ‘HttpSecureSocket.EncodeFormData’.

I changed a bit of the code to:

Dim mycontent As String
mycontent = pagador.EncodeFormData(Session.saleitem)
pagador.SetRequestContent(mycontent, "application/x-www-form-urlencoded; charset=ISO-8859-1")

pagador.Secure = True
pagador.ConnectionType = HTTPSecureSocket.TLSv12

pagador.Post("https://ws.sandbox.pagseguro.uol.com.br/v2/transactions")

Unfortunately the result was not as expected. :frowning:

set something like

Session.saleitem.Value(“shippingAddressStreet”) = EncodeURLComponent(“Gerao, Comunicao e Gesto - Rua Baro do Amazonas, 412, Petrpolis”)

Where do you see this: Gerao, Comunicao e Gesto - Rua Barão do Amazonas, 412, Petrópolis ??
Is it send back to your socket or do you get that outside of your xojo-app? If not, are you defining the encoding for that returned string?

Thanks again, @Marius Dieter Noetzel !

I get this line from the socket. It is the answer sent by PagSeguro.

I made a test with Postman and I have the same results. Very sad.

I just made a workaround (an ugly one) by replacing all accented characters before sending out the message. Not perfect, but at least the resulting page will not sting users’ eyes.

Read up on the exciting topic of Mojibake. At first glance your data looks like something mixed up encodings.

Normally, you do a DefineEncoding followed by a ConvertEncoding. When fixing Mojibake you do that the other way around. Here is a blog post from Christian to fix Mojibake for most cases:

https://www.mbs-plugins.de/archive/2018-12-04/Fixing_broken_text_encodings_i/monkeybreadsoftware_blog_xojo

If you know the wrong encoding then you don’t need to do the test for the encoding.