Windows Shell command text encoding issue

@Michel Bujardet , this is very odd, the file exist it even go through f <> Nil and f.Exists, it reacts to everything inside the app except the shell method.

@Tim Hare , the command works every time if the full path from top to end is English characters and numeric characters so the environment variables are accepted in Xojo.

I tried German, Hebrew and Russian, only standard English characters work. In Mac no problem at all.

@Tim Parnell , I can assure you ffprobe is not a low level app and you can see its output and handle it appropriately within Xojo Shell, just not when you sent it any character beyond the standard ASCII I suppose.

I wasn’t talking about the ffprobe part, I was talking about the characters getting mangled.
The Xojo engineer I highlighted may know more, I would see what he says.

Normally, shellpath should show something in pure ASCII. It should never show extended character set such as Hebrew.

You have another possible way, it is to build a batch program and run it with f.launch.

To create the file, use TextOutputstream.

BTW: Mac is built on UNIX, Windows is based on MS-Dos. These are extremely different systems, managed in very different ways. Don’t expect respective shells to behave identically.

Hi @Tim Parnell Thanks for the clarification and the pointer to William :slight_smile:

Thanks for the idea @Michel Bujardet but if I understand you idea correctly I wouldn’t be able to interact with the results within the Xojo app so I’ll need to send it to some text file somewhere then read the file stream and if I have more actions to do according to the results create a new batch. Sounds awkward, I might consider it as a last resort.

Have a look at this conversation.

Here is a small program that should test what your Windows does with the different paths :
testshell.xojo_binary_project

Here is what it does on my US Windows 10 :

I just noted, somewhat expectedly, that Windows US-EN CMD (Command prompt) does not support Hebrew.

So indeed, Windows Hebrew is special, but there is no documentation about how and if it is possible to obtain Hebrew support in Roman versions.

This could help create a batch that generates the shortpath (shellpath in Xojo) : https://superuser.com/questions/348079/how-can-i-find-the-short-path-of-a-windows-directory-file

As Julian mentioned, this is fixed in 2018r1. If you are able to grab a pre-release for testing please do.

If you cannot wait for 2018R1, you can use this trick:

dim commandline as string
commandline = "...your command line.."
commandline = commandline.ConvertEncoding(Encodings.WindowsANSI).DefineEncoding(nil)

dim term as new shell
term.Execute commandline

William, so what is the encoding of the shell in 2018R1, now ?

I did test typing a file with a Hebrew name, and it does work perfectly.

[quote=378599:@Christoph De Vocht]If you cannot wait for 2018R1, you can use this trick:
ConvertEncoding(Encodings.WindowsANSI)[/quote]

I am sorry to rain on your parade, Christoph, but WindowsANSI does not support Hebrew :
https://msdn.microsoft.com/en-us/library/aa245259(v=vs.60).aspx

So this cannot help the OP.

First of all, you are all amazing people, I never seen this kind of thoughtful help response.
William, finally good news !!!
To be honest I was on the verge of giving up, not that I don’t know how to workaround the issue I just don’t see the point if an app built on console communication and the code it is built upon doesn’t give you what you need.
of course if it was only Hebrew I could understand but in the last 24 hours I tried numerous languages with no luck and on the other hand I quickly wrote the same code in autohotkey and although it looks like Gibberish when you look at the console’s readout it works perfect in any language I tested so now I’m certain that it is possible to make it work without workarounds just not in Xojo (at the moment)
Plus I did some digging and it seems that every major programming language has a way of dealing with the windows console ‘limitations’ and should Xojo.
I even found documentation in .net that looks like this: Console.OutputEncoding Property supports UTF-16 encoding with the UnicodeEncoding class. UTF-32 encoding with the UTF32Encoding
And in C# public class Test { static void Main() { System.Console.OutputEncoding = System.Text.Encoding.UTF8; System.Console.WriteLine("?????"); } }

[quote]works for me (and yes, even though the Windows console uses UTF-16 you have to set UTF-8; I have no idea why).

C:\>test.exe
???
Note that the console may show ??? in which case you need to set the console font to a TrueType font (e.g. Consolas or Lucida Console), otherwise the console tries converting everything into the OEM character set and characters that cannot be converted become question marks. After that the console shows still no Persian because there is no complex script rendering support, but you can copy/paste it just fine.[/quote]
And here is a test I have made myself, input file:

[quote]F:\TBE\footage_5452\???\??? ??? ???.MOV[/quote] that’s Russian, if you wonder
In console it looks like this: [quote]F:\TBE\footage_5452\?á???ç???Å\?ñ??? ???ç???é???î??? ?Ç???é?ï.MOV’[/quote] (sent via AHK code just like you would do in Xojo)
but unlike Xojo’s gibberish this one actually work, I have no idea how but after a second got the results intact. it even was able to generate through the console a processed file-based on the same Russian path and name with with a postfix naming I created for the generated file to distinguish between the two (and to keep the original intact in the process)
So as you can see other syntex are able to make it work despite the limitations and the fact that it doesn’t look readable during the process but who cares nobody sees it anyway it’s an invisible background process that just need to work right ?

Cheers,
and again, you are all truly amazing people I just hope the fix will come out soon :slight_smile:

[quote=378545:@Michel Bujardet]Here is a small program that should test what your Windows does with the different paths :
testshell.xojo_binary_project

Here is what it does on my US Windows 10 :
[/quote]
Michel this is what came back from your test program:

Looks like current Xojo itself cannot provide the short path in your system. That could be considered a bug, but since I have no access to a Hebrew version of Windows, you are probably the best candidate to file the bug report.

Seems to me you have two choices :

  1. Wait for 2018R1 to come out. I tested that it actually supports input of a path in Hebrew. Chances are it supports just as elegantly Cyrillic. There is no precise release date, but usually there is a version per quarter, and the first quarter ends fairly soon.

  2. Use a batch program to get the short path with FolderItem.Launch so it does not have the limitations of Xojo’s shell, and store it into a file that you can later read from Xojo.

Thank you very much Michel !
You are incredibly meticulous and kind, I believe I can wait for the coming 2018R1 either way I have lots of coding work to do until I actually get to the publish part so just knowing this issue is addressed is all I need at the moment.
About the Hebrew system, I’m not the guy to file the bug report because I’m using US windows installation just like you, the only difference between us is that I can switch keyboard layouts between Hebrew-Russian-English (that is something that anyone can do with just few clicks) other than that I have I have nothing special on my system to make it ‘Hebrew oriented’ the Hebrew example I gave here was just to make a point that non English languages are not passing through the Shell execute method.
This is something I would never need for myself, but want it to be supported for users (all over the world) who hopefully use my software when it’s done.
I’m a bit surprised that it didn’t come up as a big issue from other Xojo users in Europe, Asia or any other non English native country for that matter.
Thanks again and Cheers,

Actually, one of the links that was posted was about Roman accented characters, CP-1252 does support them, so we were able to solve the issue.

The new Unicode support in 2018R1 is amazing, though. William Yu has been pouring a lot of love in the Windows version :slight_smile:

[quote=378674:@Michel Bujardet]Actually, one of the links that was posted was about Roman accented characters, CP-1252 does support them, so we were able to solve the issue.

The new Unicode support in 2018R1 is amazing, though. William Yu has been pouring a lot of love in the Windows version :)[/quote]
this is great Thanks to @William Yu of course :slight_smile:

For the record (and closure) I confirm, The new Unicode support in 2018R1 is amazing. problem solved !
Thanks again for all of you and of course to @William Yu who made it possible :slight_smile: