Lost in Shell encoding hell (Shell, Evernote)

Pascal_B_Kreil · July 12, 2015, 1:29pm

Hi there I’m a bit lost in encoding hell

Here’s the thing

I’m trying to create a new Note in Evernote using the enscript.exe

Problem is with encodings of Umlauts and special characters.

This example

Sub Action()
  dim sh as New Shell
  
  Dim Command As String
  
  // Creating the Shell-Command
  Command = "echo. 2>&1 | " + chr(34) + "c:\\Program Files\\Evernote\\Evernote\\ENScript.exe" + chr(34) + " createNote "
  Command = Command + "/t " + chr(34) + "Römer" + chr(34)
  
  // Changing the encoding to WindowsANSI
  Command = ConvertEncoding(Command, Encodings.WindowsANSI)
  
  System.DebugLog Command
  
  sh.Execute(Command)
  
  System.DebugLog sh.Result
End Sub

should create a new Note, tagged with “Römer” (which translates to “romans” in english and is only an example).

But the Result in Evernote is “RÃ¶mer” and that’s obviously an encodings issue.

Why? Xojo is using UTF8 internally. Using ConvertEncoding to convert to WindowsANSI in every other project works when executing shell commands. I really don’t understand what’s going wrong here.

Entering the Command directly into the Windows Command Prompt works without any issues.

I bet I’m overlooking something but I’m not seeing it (and I’m playing around for some hours now)

Details:

Windows 7 German
Xojo 2015 2.2

Kem_Tekinay · July 12, 2015, 2:15pm

What happens if you skip ConvertEncoding entirely?

Pascal_B_Kreil · July 12, 2015, 2:38pm

Hi!

Sorry, forgot to mention this. It looks exactly the same. That’s why I put the Convert-Line in the code but it seems to do nothing in this case.

Pascal

DaveS · July 12, 2015, 2:43pm

I’m thinking your string did NOT start out UTF-8, because everything I can find shows those characters as part of the UTF-8 encoding.
And that the encoding you are selecting is undoing that.
Try to replace Encodings.WindowsANSI by using UTF-8 explictity

Kem_Tekinay · July 12, 2015, 2:52pm

I’m thinking that it’s sort of an opposite problem. The shell needs WindowsANSI, but Evernote is expecting UTF-8. To test, try this:

 // Creating the Shell-Command
  Command = "echo. 2>&1 | " + chr(34) + "c:\\Program Files\\Evernote\\Evernote\\ENScript.exe" + chr(34) + " createNote "
  Command = Command + "/t "
  dim s as string = """Römer"""
  Command = Command.ConvertEncoding( Encodings.WindowsANSI ) + s.DefineEncoding( Encodings.WindowsANSI )

Kem_Tekinay · July 12, 2015, 2:54pm

BTW, when you check the string after you’ve forced the encoding to WindowsANSI, you’ll find it’s the same bogus characters as in your errant result.

DaveS · July 12, 2015, 2:58pm

I am seeing places that reccomend doing this before you sent the UTF-8 string

chcp 65001

Change Code Page

Pascal_B_Kreil · July 12, 2015, 3:30pm

That creates “RÃÂ¶mer” in Evernote.

Evernote is (to my knowledge) expecting UTF8 for files to be added as new note content. When using cmd.exe and typing in the Command-String

echo. 2>&1 | "c:\\Program Files\\Evernote\\Evernote\\ENScript.exe" createNote /t "Römer"

by hand everything is WindowsANSI and all special characters are correct.

Pascal_B_Kreil · July 12, 2015, 3:33pm

That creates the exact same result “RÃ¶mer”

Kem_Tekinay · July 12, 2015, 3:36pm

[quote=199912:@Dave S]I am seeing places that reccomend doing this before you sent the UTF-8 string

chcp 65001

Change Code Page[/quote]

Pascal, did you try this?

Pascal_B_Kreil · July 12, 2015, 3:42pm

Result when sending UTF8 to the Shell: “RÃ¶mer”
Result when sending WindowsANSI to the Shell: “RÃ¶mer”

Same odd stuff

Kem_Tekinay · July 12, 2015, 3:45pm

I think the Shell is converting the command to UTF-8 first. Try defining the encoding as nil and see what happens.

Kem_Tekinay · July 12, 2015, 3:45pm

Sorry, let me clarify. Convert the entire command to WindowsANSI, then define the encoding as nil.

Michel_Bujardet · July 12, 2015, 3:48pm

That looks somewhat similar to https://forum.xojo.com/13176-shell-output-encoding-problem/0

I just downloaded Evernote and will try to experiment a but further.

Pascal_B_Kreil · July 12, 2015, 3:56pm

You’re my hero of the day

That’s it. Works

Kem_Tekinay · July 12, 2015, 4:49pm

Just to explain why (I think) that works, it seems the Shell class is converting to UTF-8 before sending it to the shell, but the shell expects WindowsANSI. By defining the encoding as nil, the Shell class can’t convert it and the shell is properly interpreting the raw bytes as WindowsANSI.

changwon_lee · July 15, 2016, 3:49pm

Kem Tekinay:

Can you explain what below comment means in code?
“then define the encoding as nil.”