Non ASCII characters and Windows…

Targets Windows
1

In a current project, I have all kinds of troubles.

I finished to rewrite most of the application…

from using a global
db = SQLiteDatabase

Then
I add db.Close as soon as the RowSet is done…

Then
I removed the global and used locals…

Then, I used:
db = Nil

Then, I realized that a non ASCII character in the name of the data base file (.sqlite) leads to troubles:
The FolderItem is not Nil, but the file does not exists (even if I can see it.)

I started to check the IDE version using 2021… same result.
I fired 2023r1.1: same result.

NB: I had to run the latests IDEs in administrator mode because it either does not run or is not able to load the project (and quit silently) or… whatever.

At last, yesterday, I fired VirtualBox,
quit the Finder (using AppleScript)
run Windows 10 (family was installed by error)
fire Xojo 2023r1.1 in administrator mode,
load the project,
removed occurences of “Français” in the folder(s) and in the .sqlite file.
run…
crash.

How the application work:

  1. It display a splash screen with a welcome and db details, (a)
  2. A click display the view by list, (a)
  3. I click twice in an entry and the view by Record is displayed (a).
    Here comes the crash.

a. the data base is opened and closed.

The application is developed using a MacBook Pro m1 (Ventura / Xojo 2021r2.1).

Yesterday, when I quit the project and shut down VirtualBox, I wanted to KTA…

I started this mess on the end of February 2023 (based on an ongoing work started in 2013 and in use elsewhere on two plants).

2

@Emile_Schwarz
Did you have a question?

1 Like
3

Yes !

Are-you using non ASCII character to name files/folders used in your applications (OS Item Names), specifically on project running Windows ?
If yes, how is that running ? (do you have troubles ?)

4

I know we are living in a Unicode world now, but I’m of the belief that there is no better way of naming files than sticking to the standard ascii range of alpha-numeric only characters.

Even spaces make me nervous, so I don’t think I would be able to sleep at night with a file or path that I was responsible for that had any nonstandard characters in it.

I am sure this is biased thinking coming from someone who is US based and English speaking. But I really believe that sometimes there is no school like the old school, when it comes to certain things like this.

1 Like
5

It may depend on the context, also.
Just take someone, non-english speaker, which has accented characters in his/her name and the path of the user’s account is likely to have them as well. Imagine if Mac OS/Windows would refuse to create the user’s account just because of that. It’s better if the OS manages to handle that by supporting unicode than telling “Sorry, Stéphanie, you can’t name your account with your name”.
In Xojo, for instance, the default encoding (in the IDE) is UTF and that’s great. Beyond the fact that one can use extended variables (Var MyFunnyVariable✎WithASymbol as integer), it’s more open than just supporting basic ASCII.

In Emile’s case, I still wonder why the folderitem isn’t nil but doesn’t exist. Without knowing how it’s built, we can’t tell.

6

The file path created by the OS probably isn’t UTF8. You would need to 1) know the encoding in use by the OS, and 2) convert the path/filename to that encoding.

1 Like
7

@Arnaud_N, I agree with you completely. But in my defense, that was why I mentioned I would be hesitant to have non standard ASCII characters in a file or path that I was responsible for, as opposed to paths created by the system itself.

I would always prefer to store my data under something like /var/lib/my-app/data instead of some system created path possibly containing spaces and other characters that might make me feel uncomfortable :joy:

Seriously, though, my comment was just to show my preference for naming style that should be bullet-proof under most circumstances. Obviously, it is from a mentality that reflects my age (I do remember having to fast forward a tape to a specific counter number, and then press play and wait until dinner was over for a program to load :pensive:).

Obviously the unicode characters should work since all systems support them and they are standard. I’m just stuck in my ways and haven’t had an issue yet when using this method.

I have run into issues in the past trying to do things with Windows based systems running tools like SCP that regardless of arcane escaping techniques, just wouldn’t work with non standard characters or spaces in the path. After renaming the paths/files the right way (my way haha), issues were solved. As a result, it just planted me more firmly in my old school naming conventions…

1 Like
8

The owning folder is created by the project at first run.
The data base file is created “-------------------------------”
The data base file is read at application run,
some data extracted and displayed in a window,
in another window, a list is populated,
when an entry is selected (click | Enter | Return) another window is opened and display the whole Record for the selected entry (using the Unique ID).

All operations excepted the last one goes fine.
The references are closed as soon as useless.

I may have forgotten that… I have a CopyFile script. Sometimes, another folder (Fran|ais instead of Français who already exists) is created and the file copied there… I squashed that bug (destination was wrong, I think).

I removed Français from the OS Item names…

The fun part: I duplicate the .sqlite file, then I removed Français from its name and run. Same behavior.

At last, on my MacBook Pro i5, (running Windows 10 Familly) I still have a lot of troubles.

There is no fun to get a message telling a file is not there while you can see it… (in this case, it was unable to read the file because dbSQLiteFI.Exists = False.)

9

I understand your way of thinking. I guess you have good Unix backgrounds; in my case, I’m more confident with the “Mac classic” roots and would better avoid these Unix folders (hard to remember which is for what) :sweat_smile:

I agree. There are still components/processes that won’t handle unicode well. Still thinking they should update to support it :slightly_smiling_face:

Things were not too much fast back then, as opposed to nowadays :slightly_smiling_face:

I too have hard times going out of my ways; and it’s a characteristic shared among many persons. Not a problem :wink:

Well, it worked better in your experiences, so it makes sense you had to stick with these conventions.
I’m saying you didn’t fail, it’s the tools that did :wink:.

2 Likes
10

That’s the “how it’s built” part I was referring to. The remaining of the app’s functionality is irrelevant :wink:
What calls do you use to find the file?

And have you tried to copy&paste that name from the CopyFile step to your code to ensure the name, where “ç” may have several code points, matches exactly?
In such cases, writing by hand may cause troubles.

Instead of finding out why the bug arises.

So “Français” is also part of the file name?
(please notice all what we have to guess that you didn’t share about the file’s details)

Agreed. But I’m betting it’s your assumption that the file is the one you expect that is wrong.
As an example: if you have a file named “MyFile—Shared” and you code it f=Folder.Child(“MyFile-Shared”), you may think it’s correct, because that’s close, but the file won’t exist (notice the dashes difference). If you write the name by hand, differences that you may not see can lead to the file name being wrong.
And it turns out that, in unicode, many characters can be expressed differently (especially on Mac, where “é” may be a single character or a combination (“´”+“e”); you can’t tell the difference visually, hence the need to be cautious about your expectations).

11

Via
SpecialFolder.Documents.Child("My folder")

And I check that.

But this is irrelevant bacause it works for the first and second window.

Yes.

About visuall deficiencies: I have some. But, this is also irrelevant because… the code works for window 1 and window 2 (nag screen and data list view)…

I found a folder called “Secours Populaire Fran|ais”… (the folder where this one was created was named “Secours Populaire Français”) where I was expected nothing; the CopyFIles1 script put there a support text file in that folder.

This is hard to live; and Xojo 2023r1.1 is not better. Same bad results.

12

So it’d be interesting to know the difference between them. Especially since you said:

You’d agree with me that if a given folderitem is not nil but doesn’t exist, it’s regardless of the enclosing window. In other words, it’s not possible that the folderitem is not nil and doesn’t exist in your window3, yet the same folderitem is either nil or exists in window1 or window2.
How do you check for “folderitem<>nil and folderitem.exists” in your 3 windows, to know that it works in window1 and window2 but not window3?

Sorry for you. I have them too, btw.

Perhaps you could explain what code works for window 1 and window 2? Because, as I said, a given folderitem won’t be nil and not exist in just window 3.

By “the folder where this one was created”, you mean the parent folder? So you had a folder “Secours Populaire Fran|ais” inside another folder named “Secours Populaire Français”?

By “expecting”, I meant the expectation between the real name and how you write it in code.

I still think there’s something solvable by code.

13

One thing I may write… the code works fine on m1…

File exists = False: the debugger told me that.

I will with some more days and go back to the code, compare what open/read in the three windows to be sure there is no divergence (the same code, except the SQLSelect)
One thing is sure, I use the same FolderItem (build in App).

14

Are you looking in the right place as SpecialFolder.Documents… isn’t always in c:\users\username\documents (I’ve yet to put this bug report in), though I might have missed something somewhere due to the mass of text above.

15

Are we talking about windows here still because windows files/folders cant have pipes | in their names.

Would you be able to succinctly explain the issue with examples in as few words as possible? :slight_smile:

16

For a cross-platform app, for filenames one may like to forbid all characters that are not permitted on one platform or another, but that is all. I see no reason otherwise to limit what a user may call a file. Limiting names to ASCII and no spaces is just a big nonsense.

1 Like
17

The problem does not comes from that IMHO. The program is ale to find the files (it read it twice, then the third: is not nil, but do not exist:

image
image911×187 61.3 KB

This is my message. And as you can read on the message left, the data base file have been read and displayed…

You are correct, this is not exactly that character (bad eyes):
image

That is why I do not bothered to use “Secours Populaire Français” as this is the customer’s name… But, people who do use only ASCII…
Look at the other way of writting (left to right, right to left, top to bottom or bottom to top…)
At last, how many years two bytes (and more) takes to comes to Windows ? (for Asian, Jewish, Arabian people, but not only…)

It seems that I now use only one FolderItem Property to read the data base file throughout the project (I removed the last duplicate occurence some hours ago). And apparently, excepted for CopyFile, all seems OK (look at the folder CopyFile creates above). I moved the service file manually and the project apparently works - once more, but how long ? -

BTW: using a single but global property for the files / folder (and TABLE) names was not enough.

For some reason, either Xojo or my eyes, the search process failed.

What also helped was to be able (thanks to a hdmi/vga converter) to connect an external screen and so comparing code / using “master properties” in App throughout the project was far easy than working on a single 13" screen.

Thank you all.

18

How have you created this folder?

It looks like UTF8 char bytes interpreted as Latin1 or something

19

You read the file three times and it only fails on the third try?

There has to be more going on in your app than just reading the file three times for it to suddenly fail as I note you talk about CopyFile later on?

Are you able to replicate the issue with a simple example project?

20

image