Listbox Malforms Korean sometimes

Arthur_Gabhart · June 22, 2018, 6:51pm

I have 2 images with file names on the leftmost column. The first cell in the top row should have this text:

This is the code which adds it to the array.

CollList.AddRow nwPRow.daita

CollList is the listbox and nwPRow is a class and .daita is a String array.

In the array daita it is correctly formed, and in my current version and in the stored text it is correctly formed.

In the lower version it is not correctly formed.
What is happening?

Correctly formed

Malformed

Michel_Bujardet · June 22, 2018, 7:02pm

Looks like an improper encoding. It can happen if you stored the text in a database. You should verify the encoding when entered. If I am not mistaken, Korean should be UTF16. Then make sure when you retrieve the text from the database, to defineencoding to the same encoding.

kevin_g · June 22, 2018, 7:12pm

From experience, UTF8 is fine for storing Korean. Like Michel said, I would check that the encoding of your string is what you expect. I would also possibly check to make sure there aren’t any null / weird bytes between each character causing the data to appear corrupted.

Arthur_Gabhart · June 22, 2018, 7:37pm

Thanks. I do use UTF8.
Every text file is read in in one place, and stripped of a UTF8 BOM, and encoding set to UTF8, and I have a global property Encdng which is set in the App,open

Encdng = Encodings.UTF8

I also have a help tag for the listbox.
This text is correctly formed.
This help tag comes from the class array supporting the listbox.

I am not sure how to access the encoding for a cell on a listbox.
Cellclick reveals only the text inside since it isn’t editable.

I also haven’t added code yet to see what the encoding is. I haven’t had to do the testing of the listbox’s encoding so that is next.
I have had a similar problem also with other file names being malformed.

I hope that was clear because of the .txt it was a file name.

Arthur_Gabhart · June 22, 2018, 11:27pm

Well, it is still clear as mud. I added these 2 lines

Dim txcode as TextEncoding txcode = Encoding(me.Cell(row,column))
The results are for both the string in the listbox and the supporting string:
base: 256
code: 134217984
format: 2
internetname: UTF-8
variant 0

When I look up terms like at Unicode.org, the only thing I am sure of UTF-8 (which is good, but…)

Sam_Rowlands · June 23, 2018, 12:10am

The difference I can see is that the second image is using a different typeface, could it be that the typeface doesn’t support Korean, correctly. It seems like the words are being broken down into individual characters (if you can call them that). I lot like Mandarin, where the words are actual constructs of different elements, to enter these words you can either draw them or type out their base elements.

Arthur_Gabhart · June 23, 2018, 12:54am

Thanks but unfortunately they are both Times New Roman.
I also verified it properly forms where it doesn’t elsewhere.
It has something to with Apple and file names. i can show to someone like Apple that it happens but I can’t predict it.
I just tried something different. Removing the “.txt” from the display and it didn’t work.

As a side note, I have had that problem with some fonts. I have used names I thought were crossplatform, but they weren’t.

Ulrich_Bogun · June 23, 2018, 1:58am

The font in your first table clearly is not Times. It looks rather like Helvetica/Arial.

Are both pictures from the same machine? Or could it be that the second one uses a different version of Times New Roman?

Sam_Rowlands · June 23, 2018, 2:17am

Use “System” for the best compatibility.

Sam_Rowlands · June 23, 2018, 2:25am

If I read directly from my desktop and put the file names into a listbox, file names in Traditional Chinese show up correctly.

Which suggests to me that the data coming from Apple’s APIs to Xojo’s is correct IMHO. My bet is on the typeface as it looks significantly different between both screenshots.

kevin_g · June 23, 2018, 9:11am

Our app (built with Xojo 2014r2 Windows / 2017r3 Mac) which uses strings and UTF8 seems okay with Korean (our app is used in Korea so we have a Korean localisation).
Some ideas:

If you are using the text datatype try using strings instead (we use strings).
Try using the System / Small System font.
If you are letting Xojo draw the column text try drawing it yourself in the paint event.
We have seen our app (possibly the OS) act differently when booted into Asian languages compared to when it is booted in English and the language changed. If you normally boot into Korean try booting in English and then switch to Korean. If this solves your problem it could indicate a bug in Xojo.

Arthur_Gabhart · June 23, 2018, 12:17pm

Thanks Sam. I changed the font from Times New Roman to Arial.
Done and success.
Arial is now the font and it works. No Malformed text.
For me, I need Serifs, but this solves the problem.

Also Thanks Kevin, I now know what my next step is. Change the boot language.
Also Kevin, Confirmations: I never switched to Text. I don’t draw the text myself (it never seemed faster)
Tests: I’ll also try System.

How do report this:
The only clues I have are the font and that it has something to do with a file name. (It also happens with sound and images)

To XOJO I know how to file a bug report but this would be the least straightforward one I filed.

Suggestions on contacting Apple. I have no idea where to start as I’m not in the developer’s program.

Arthur_Gabhart · June 23, 2018, 6:29pm

Well, not sure whether I have enough info. I changed the font back to Times New Roman.
Then I changed the language to Korean and it still happened.
I don’t have localization or lingua set up.
Would that matter?