Parse URLConnection String into JSON error

General
1

I have a URLConnection that has an Event Handler - ContentReceived. I am attempting to parse the returned string ‘Content’ into a Variant named json and I have attempted to do this as followings with this error

// In the returned JSON, you will see it is an array
Dim json As Variant
json = ParseJSON(Content)

This errors as lexical error: invalid bytes in UTF8 string.

I have also tried this without success.

// Looking at the returned JSON, you can see it is an array
Dim json As Auto
json = Xojo.Data.ParseJSON(content.ToText)

and this

// Looking at the returned JSON, you can see it is an array
Dim json() As Auto
json = Xojo.Data.ParseJSON(content.ToText)

Both error with The data could not be converted to text with this encoding.

Here is the returned String I am attempting to put into JSON

[{“id”:“2”,“login_name”:“aperson”,“passwordSalt”:“jKAfSvcs10awESaqvsME”,“passwordHash”:“\r\u0003�/R>~�7��m#�c��\u001C���V��A!�W�b�S”,“privilege”:“Super”,“company”:“”,“db_to_use”:“somecounty”,“db_to_use_username”:“somecounty”,“db_to_use_password”:“anotherpassword”,“table_to_use”:“tabletouse”,“table_to_use_username”:“ausername”,“apassword”:“a1password”,“MyDBPersonalHost”:“10.8.32.12”,“MyDBDataHost”:“10.8.32.12”,“sql_str_to_use”:“SELECT * FROM tabletouse JOIN Contact_Info ON tabletouse.VRCNUM = Contact_Info.VRCNUM JOIN Lawn_Signs_2019 ON tabletouse.VRCNUM = Lawn_Signs_2019.VRCNUM JOIN Contributions_2019 ON tabletouse.VRCNUM = Contributions_2019.VRCNUM WHERE (Town_Code = ‘060’) Or (note LIKE ‘%supporter%’ ) ORDER BY (Town_Code+0) ASC, (Election_District+0) ASC, Street ASC, (House_Number+0) ASC”,“LD”:“15”,“ED”:“37”,“Party”:“party”,“Table_Year”:“2021”,“City”:“acity”,“TownCode”:“095”}]

I found threads where I need to change this Dim json As Variant to this Dim json As Variant or this Dim json As Auto but each fail.

I had this working on 2019r1.2 with that rerun Content was xojo.CoreMemoryBlock but do not see how to make this work on 2021r2 when content is returned as String. I need to get this into JSON so the

response can be read appropriately as I follow this up with the following

Our_Table = Xojo.Core.Dictionary(json(0)).Value("table_to_use")

2

Looks like you need to define the encoding of your content before parsing the json data.

3

I also attempted this
json = Xojo.Data.ParseJSON(content.DefineEncoding(Encodings.UTF8).ToText)
and that returns same error The data could not be converted to text with this encoding.

I have also changed the json VAR to

Dim json() As Auto
json = Xojo.Data.ParseJSON(content.DefineEncoding(Encodings.UTF8).ToText)

and that provides the same error

I think what the error is saying is that the encoding of content is not correct. Looking at Language Reference for URLConnection.ContentReceived and content states could be binary or text data.

Also tried this

Dim json() As Auto
json = ParseJSON(content.DefineEncoding(Encodings.UTF8))

and received
lexical error: invalid bytes in UTF8 string.

4

So I added this test

If Not Encodings.UTF8.IsValidData(content) Then
  // do whatever you want here...
  MsgBox "Error: Encodings.UTF8.IsValidData"
  Return
End

And it failed

so what encoding does URLConnection return?

5

Looking at your data I can see that there is a mixture of curly & standard quotes. Also the password hash really needs to be base64 encoded.

6

The real question is, What encoding does the site you’re connecting to return? URLConnection has no way of knowing.

7

It looks like the passwordHash has a value of binary data…?

Best to make sure the server rturns UTF-8 and without linebreaks (that could break the json)

8

The site returns UTF8 and is written in XOJO 2019r3.2 which is a modified LUNA web service API.

Previously this solution worked on both sides - the modified LUNA (Web API) and the iOS app that is generating the login request. At this time I am still using the modified LUNA in 2019 XOJO and am using that to test with.

The Web Service API returns ‘Response’ which is defined as a Dictionary. If all is good we pass the 200 check and get to these final lines of code

// Return the "200" response with the data.
Response.Value("ResponseStatus") = 200
Response.Value("ResponseBody") = Records.ToString
Return Response

This is where the response is returned to the iOS app.

I am using the iOS app with appropriate changes to get the app to compile and run. It is in here that reading the response does not work. I have attempted several modifications and so far I cannot get data that I can see in the debugger.

9

Some background
Our_Table is defined as Text and will be used to display text in this iOS app

To address encoding and get the info I now do the following

Var mystr As String = content.ConvertEncoding(Encodings.UTF8)

// Looking at the returned JSON, you can see it is an array
Var jsonData() As Variant
jsonData = ParseJSON(mystr)

Looking at TypeMismatch I try the following
Our_Table = Dictionary(jsonData(0)).Value("table_to_use").TextValue
and that errors as a TypeMismatchException
So I do this

Var mystr1 As Variant = Dictionary(jsonData(0)).Value("table_to_use")
Our_Table = mystr1

and Our_Table errors as a TypeMismatchException
So I do this

Var mystr1 As Variant = Dictionary(jsonData(0)).Value("table_to_use")
Our_Table = mystr1.TextValue

and Our_Table errors as a TypeMismatchException
Next I change Our_Table to String and then change all other code that uses Our_Table to be able to use a String verses a Text i.e. (Our_Table.ToText)

Var mystr1 As Variant = Dictionary(jsonData(0)).Value("table_to_use")
Our_Table = mystr1
Our_Table = Dictionary(jsonData(0)).Value("table_to_use")

and all is good.

If I have to make these changes to variables i.e Text to String then they are significant. Seems as though Variant works however casting to Text does not.

So what am I missing here?

10

You keep saying that all of the data is UTF-8, but it’s clearly not. The password hash is pure binary. All of those characters that show up as diamonds should be encoded at the very least, but it’d be easier to just use EncodeHex on that whole string when creating it and DecodeHex when reading it.

11

Please stop using Variant.TextValue use string instead. It will help you alot. And i think the same as greg as i’ve already noted. There seems to be binary (nil-encoding) data which cannot be combined. All must be UTF-8 or it will fail

use datatype.ToString (not .ToText)
use Variant.StringValue (not Variant.TextValue)

12

Yes I think your right. To check I made a sql req[quote=“Greg_O_Lone, post:10, topic:64663, full:true”]

I think you’re right on that. I had narrowed this down to the hash and was working on how to modify parts of the return. When I came up for air I saw your response and that is how I will proceed.

Thanks

13

You are correct the hash is not UTF-8. I will encode/decode the entire string ■■■ mentioned to Greg. The interesting thing is that this all worked in XOJO 2019. I never caught this when 2020 came out because I did not need to touch the code until now. So just like WE there are other changes that affect old code for iOS as well.

I am still caught in the String to Text piece from years ago. I think I am going to move to Tim Dietrich last piece of work for XOJO with his ALOE Web Service API. Then make whatever changes I need to and start from there. At the same time clean up xfer of data between Web Service API and iOS device using HEX encode/decode. In the end it just might make life simpler.

Thanks to both

1 Like