Passing parameters to Python script with Shell.Execute

  1. ‹ Older
  2. 2 weeks ago

    Robert L

    Jun 8 Federal Way, WA (Seattle Area)

    The original problem was providing string parameters to a Python script using Shell.Execute. Problems occur with certain characters. They need to be escaped or dealt with in some way.

    Tim provided a list of characters that he suggested to escape and provided some code to do just that.

    \ ` ~ ! @ # $ % ^ & * ( ) | { } [ ] ' < > ? "

    Markus suggested that simply embedding the parameters in quotes would do the job. I liked that solution, but found that $ required being escaped, despite the string being embedded in quotes. Tim then commented that more than $ would cause problems.

    I tested all the characters in Tim's list and found only one additional character "`". that still had to be escaped. So for my particular situation, it appears that using Markus's solution works if I additionally take care to specifically escape "$" and "`"

    The "double quote" character presents its own challenge that I will not go into. I can use Tim's solution to handle this,

    Tanner pointed to another character that causes problems, the semicolon ";", which is not on Tim's list. That particular character does not cause problem if part of a parameter enclosed in quotes as per Markus.

    I am not sure that between Tim and Tanner, I have a complete list of the potential problem characters. (As per Tim's suggestion, I did buy, but have not finished, the Learning Unix of OS X). But I have not run across anything else.

    For now, I have added these lines to the Markus solution.

    Dim backslashAndDollarSign As String = "\" + "$"
    Dim backslashAndBacktick As String = "\" + "`"
    
    arg = ReplaceAll(arg, "$", backslashAndDollarSign)
    arg = ReplaceAll(arg, "`", backslashAndBacktick)

    It seems to me that the Tim solution should also work although I might wonder if ";" needs to be added to the list.

    P.S. One little trick that perhaps someone does not know. On the Mac, you can get the path of a folder or file by selecting it and keying Option - Command - C which places that path on the clipboard. That path will not be "escaped" however. If you want the escaped version, you can drag the icon of a folder or file onto the Terminal window which will then show the escaped version of the path. You can name folders on the Mac with a startling number of weird characters and then drag that folder onto the Terminal and see if that characters is escaped (preceded by a \ [backslash]). A way to test for characters that Unix deals with "specially" as I would interpret it.

  3. Tim J

    Jun 8 Pre-Release Testers, Xojo Pro Phoenix, AZ USA (desication ce...

    @RobertLivingston It seems to me that the Tim solution should also work although I might wonder if ";" needs to be added to the list.

    And that was a copy/paste error on my part. The semicolon is in my actual functioning code.

    To further explain the characters above and why I wrapped them, look into the way a shell handles the characters. Here's a great BASH script that will show the characters that you should escape when passing them through the command line as arguments:

    for i in {0..127} ;do
        printf -v var \\%o $i
        printf -v var $var
        printf -v res "%q" "$var"
        esc=E
        [ "$var" = "$res" ] && esc=-
        printf "%02X %s %-7s\n" $i $esc "$res"
    done | column

    In an 80 column terminal, the output will look like this:

    00 E ''     	1A E $'\032'	34 - 4      	4E - N      	68 - h      
    01 E $'\001'	1B E $'\E'  	35 - 5      	4F - O      	69 - i      
    02 E $'\002'	1C E $'\034'	36 - 6      	50 - P      	6A - j      
    03 E $'\003'	1D E $'\035'	37 - 7      	51 - Q      	6B - k      
    04 E $'\004'	1E E $'\036'	38 - 8      	52 - R      	6C - l      
    05 E $'\005'	1F E $'\037'	39 - 9      	53 - S      	6D - m      
    06 E $'\006'	20 E \      	3A - :      	54 - T      	6E - n      
    07 E $'\a'  	21 E \!     	3B E \;     	55 - U      	6F - o      
    08 E $'\b'  	22 E \"     	3C E \<     	56 - V      	70 - p      
    09 E $'\t'  	23 E \#     	3D - =      	57 - W      	71 - q      
    0A E $'\n'  	24 E \$     	3E E \>     	58 - X      	72 - r      
    0B E $'\v'  	25 - %      	3F E \?     	59 - Y      	73 - s      
    0C E $'\f'  	26 E \&     	40 - @      	5A - Z      	74 - t      
    0D E $'\r'  	27 E \'     	41 - A      	5B E \[     	75 - u      
    0E E $'\016'	28 E \(     	42 - B      	5C E \\     	76 - v      
    0F E $'\017'	29 E \)     	43 - C      	5D E \]     	77 - w      
    10 E $'\020'	2A E \*     	44 - D      	5E E \^     	78 - x      
    11 E $'\021'	2B - +      	45 - E      	5F - _      	79 - y      
    12 E $'\022'	2C E \,     	46 - F      	60 E \`     	7A - z      
    13 E $'\023'	2D - -      	47 - G      	61 - a      	7B E \{     
    14 E $'\024'	2E - .      	48 - H      	62 - b      	7C E \|     
    15 E $'\025'	2F - /      	49 - I      	63 - c      	7D E \}     
    16 E $'\026'	30 - 0      	4A - J      	64 - d      	7E - ~      
    17 E $'\027'	31 - 1      	4B - K      	65 - e      	7F E $'\177'
    18 E $'\030'	32 - 2      	4C - L      	66 - f      
    19 E $'\031'	33 - 3      	4D - M      	67 - g      
  4. Robert L

    Jun 8 Federal Way, WA (Seattle Area)

    In an 80 column terminal, the output will look like this:

    Perfect

  5. Markus W

    Jun 8 Pre-Release Testers, Xojo Pro #JeSuisHuman Germany, Heidelb...
    Edited 2 weeks ago

    I always find that proper naming and an extra line can make code MUCH more readable. So taking Tim’s code:

    Function Escape( theString As String) As String
    
      Dim CharactersThatMustBeEscaped as String = ("\ ` ~ ! @ ; # $ % ^ & * ( ) | { } [ ] ' < > ? """)
     
      For Each char As String in Split( CharactersThatMustBeEscaped, space )
        theString = theString.ReplaceAll( char, "\" + char)
      Next
      
      theString = theString.ReplaceAll( space, "\" + space )
      
      Return theString
    
    End Function

    or

    Function Escape( theString As String) As String
    
      Dim ArrayOfCharactersThatMustBeEscaped() as String = Split("\ ` ~ ! @ ; # $ % ^ & * ( ) | { } [ ] ' < > ? """, space)
    
      ArrayOfCharactersThatMustBeEscaped.append space
     
      For Each char As String in ArrayOfCharactersThatMustBeEscaped
        theString = theString.ReplaceAll( char, "\" + char)
      Next
      
      Return theString
    
    End Function
  6. Robert L

    Jun 8 Federal Way, WA (Seattle Area)

    And the proper way to deal with characters outside of the ASCII code set?

    For example: cliché

    é causes a problem when I try and pass it to Python using Script.Execute.

    As for "verbose code" with explanatory names and extra lines, I am a big fan. I know at my age that I will be confused when I go back to the code after a few weeks/months. I happily convert the "concise" code that many prefer to my own style. Just the act of doing it, helps me with understanding the contributed code better. And I never look at a gift horse in the mouth.

  7. Markus W

    Jun 9 Pre-Release Testers, Xojo Pro #JeSuisHuman Germany, Heidelb...

    @RobertLivingston And the proper way to deal with characters outside of the ASCII code set?

    For example: cliché

    é causes a problem when I try and pass it to Python using Script.Execute.

    Don’t use them ;)

    But seriously: “pass” as what?

    é in a string is fine, but as a variable name probably not

  8. Robert L

    Jun 9 Federal Way, WA (Seattle Area)

    Don’t use them

    I can accept that. But my original purpose was to pass strings to Python, manipulate them there, and then bring the result back into Xojo.

    With the help of you and others, I succeeded in doing just that with the ASCII characters. The mechanism that I am using is

    pyShell.Execute(PATH_PYTHON3 + SPACE + PYTHON_SCRIPT + SPACE + argument1 + SPACE + argument2)

    But seriously: “pass” as what?

    Pass as what is effectively a "parameter". With the syntax above, I can pass this string data to the Python script for it to work on.

    argument1 and argument2 are strings (and now I have learned to properly escape the problematic characters when they are passed to Python. ) So if I want to come up with a way to do this "in general" I have to be aware that this mechanism will not work for characters outside of ASCII. So I do not know how to use such a mechanism to pass a character like é to a Python script.

    Beatrix suggested early on the possibility of using Base64 encoding and perhaps that would work. I have not yet tried it and was wondering whether there was a more commonly accepted methodology.

    I have also yet to explore an alternate "technique" of writing a text file to disk using Xojo and then having the Python script that I have launched using Xojo (and I could pass its absolute path) read the contents of that file into Python. It seems to me that that would likely address this issue, but again I do not want to go down that route if there is a more straight-forward way of getting the strings from Xojo to a Python script that was called by Xojo using Shell.Execute.

  9. Markus W

    Jun 10 Pre-Release Testers, Xojo Pro #JeSuisHuman Germany, Heidelb...

    Why do you want to manipulate the strings in Python instead of in Xojo? The only reason I could see is for example to use BioPython, but in that case ASCII should do fine …

  10. Robert L

    Jun 10 Federal Way, WA (Seattle Area)

    Why do you want to manipulate the strings in Python instead of in Xojo?

    Short answer. It is more than strings.

    Intermediate length answer :)

    Xojo and Python are very different places (compiled vs interpreted; commercial vs open source; obscure vs hugely popular; small number of third party resources vs huge libraries of third party add-ons; easy creation of graphical user interface vs difficult and immature attempts)

    I am really trying to create the infrastructure that would allow me, when working on projects in Xojo, to drop down into Python and perform certain tasks in that environment and then return the results to Xojo for display or further refinement. My hope is that if I can overcome/understand all the glitches here, and figure out the process, then going forward I will not have to worry about the issue.

    I am an intermediate Xojo programmer and a beginning Python programmer. I like creating apps for my own use and in some cases offer them for free to others. But I am not a commercial developer so paying $ for third party add-ons to Xojo is onerous for me. The extensive library add-ons of Python are free.

    I generally prefer writing my code in Xojo but even as a beginning Python programmer I see certain approaches to problems are easier for me in Python. My biggest problem in Python is not being able to create applications with nice GUI interfaces which I can do in Xojo.

    I am not solely interested in processing strings in Python. It so happens that strings are the only way I know (apart from writing files to disk to access from both environments) to pass data back and forth between Xojo and Python. Changing strings back numbers or arrays or whatever is not hard in either environment. So the string emphasis is just an artifact of trying to come up with a smooth way of pushing data back and forth.

    There are circumstances that in fact I do want to process strings in Python rather than in Xojo. You mention BioPython as an example. I find working with Regex in Python (with re library) is easier for me than the same in Xojo. One thing that launched me in this direction of trying to learn Xojo/Python intercommunication was an occasion that I wanted to use Levenshtein distance as a tool to find name matches in a context with a couple thousand name and some of the names were misspelled. Implementing Levenshtein distance in Xojo code is complicated for me, and I see in Python that I can find a library that will supply this code without my having to understand it and actually implement it myself.

    NumPy & SciPy supply all these tools for scientific computing.

  11. last week

    Markus W

    Jun 10 Pre-Release Testers, Xojo Pro #JeSuisHuman Germany, Heidelb...

    @RobertLivingston I am an intermediate Xojo programmer and a beginning Python programmer.

    Same here.

    My biggest problem in Python is not being able to create applications with nice GUI interfaces which I can do in Xojo.

    Same here.

    If Python got a decent compiler and IDE it would wipe the floor with anything else …

    One thing that launched me in this direction of trying to learn Xojo/Python intercommunication was an occasion that I wanted to use Levenshtein distance as a tool to find name matches in a context with a couple thousand name and some of the names were misspelled..

    How about using an [in-memory?] database and doing a LIKE search instead?

  12. Beatrix W

    Jun 10 Pre-Release Testers Europe (Germany)

    Once upon a time I felt the lure of Python, too. So many many nice codes. Alas, Xojo as language is 1000% better than Python.

    1. whitespace (majorly awful)
    2. case sensitive (are they nuts?)
    3. multiple returns from a function

    The latter one broke me as I never could be sure what a function would return. Commercial code had to make sure that everything was in place - stuff that in Xojo the compiler enforces.

    For a private project: more fun to you.

    Oh, the IDE I used - PyCharm - was pretty nifty. Way too small buttons. But really nice to use. Actually working auto-complete. Debugger values shown inline with the code - that was great.

  13. Markus W

    Jun 10 Pre-Release Testers, Xojo Pro #JeSuisHuman Germany, Heidelb...

    @Markus W wanted to use Levenshtein distance as a tool to find name matches in a context with a couple thousand name and some of the names were misspelled.

    P.S. That would also be VERY slow in my opinion …

  14. Robert L

    Jun 10 Federal Way, WA (Seattle Area)

    I just used Levenshtein distance as an example. I did implement what I needed in yet another language, and it was extremely slow and I had to "translate it" into that language from whatever I could find on the web and debug the code etc. I am happier if I can find a pre-baked solution.

    I have not used the Python implementation so I have no first hand experience. But my impression is that the Python implementation is done as a call to complied C code which was written by professionals to be fast and efficient. Once I get my Xojo/Python linkage project worked out, I will try it.

    As for the slowness, it was very slow but I used various "cheats" like filtering out cases that neither the first letter of the last or first names matched and in this way made it more manageable.

  15. Paul F

    Jun 11 Pre-Release Testers, Xojo Pro Europe, Cambridge

    You may already know of this but thought I'd mention that Joe Strout has a String Utils module on his website that calculates Levenshtein distance along with loads of other functions.

  16. @RobertLivingston My biggest problem in Python is not being able to create applications with nice GUI interfaces which I can do in Xojo.

    wxPython uses the native widgets of each supported platform.

    For an IDE you could look at Visual Studio Code

    I think Einhugur offer some classes to enable interaction between Xojo and Python.

  17. Markus W

    Jun 11 Pre-Release Testers, Xojo Pro #JeSuisHuman Germany, Heidelb...

    http://www.strout.net/info/coding/rb/intro.html

  18. Robert L

    Jun 11 Federal Way, WA (Seattle Area)

    wxPython is a cross-platform GUI toolkit for the Python programming language. It allows Python programmers to create programs with a robust, highly functional graphical user interface, simply and easily.

    While I am very glad that wxPython and the Phoenix variant exist, when compared with using Xojo, these are a long way from easy IMO.

    I have just started working with my daughter on a QGIS project. QGIS Amazing free resource if you are into cartography. But it is one of those things that is married to Python. I would like, in my own life, to get the best of Xojo and Python. So I could do something like working with Xojo linked with Python to do QGIS stuff.

    I will say no more until I have actually done something interesting. Talk/speculation is cheap.

    And I have hijacked my own thread. :(

  19. Robert L

    Jun 11 Federal Way, WA (Seattle Area)

    Summary:
    The problem originally described was one of reliably passing data in the form of string(s) to a Python script using Shell.Execute. There is the related issue of getting data back from that same Python script using Shell.Result.

    Const SPACE = " "
    Const PATH_PYTHON3 = "/Users/owl/anaconda3/bin/python"
    Const PATH_PYTHON_SCRIPT = "/Users/owl/Documents/PythonPractice/PythonLittleApps/Trial.py"
    
    Dim argument1 As String
    argument1 = "abcde"
    Dim argument2 As String
    argument2 = "pqrst"
    
    Dim pyShell As New Shell
    pyShell.Execute(PATH_PYTHON3 + SPACE + PATH_PYTHON_SCRIPT + SPACE + argument1 + SPACE + argument2)
    
    Dim sResult As String
    sResult = pyShell.Result
    MsgBox(sResult)

    One difficulty is passing certain characters that are misinterpreted when they go to the Shell. Characters like $; !; < etc.

    If you need to pass such characters, one can figure out what all these characters are and escape them before sending them to the Shell

    Tim Jones basically contributed:

    Const METHOD_NAME As Text = "FixSpecial"
    Const CHARACTERS_THAT_MUST_BE_ESCAPED As String = "\ ` ~ ! @ # $ % ^ & * ( ) | { } [ ] ' ; < > ? """
    For Each iChar As String In Split(CHARACTERS_THAT_MUST_BE_ESCAPED, " ") // It is split on space.
      sEscaped = sEscaped.ReplaceAll(iChar, "\" + iChar) // each trouble character gets replaced with escaped version
    Next
    
    sEscaped = sEscaped.ReplaceAll(" ", "\ ") // spaces get escaped
    
    Return sEscaped

    Markus Winter noted that if you enclose the arguments in quotes most of the troublesome characters do not need to be escaped although a few still do and based on this:

    Const METHOD_NAME As Text = "EscapedArgument"
    Const SQ = &u27 // ' Single Quote
    Const DQ = &u22 // " Double Quote
    
    Dim backslashAndDollarSign As String = "\" + "$"
    Dim backslashAndBacktick As String = "\" + "`"
    Dim backslashAndSingleQuote As String = "\" + SQ
    Dim backslashAndDoubleQuote As String = "\" + DQ
    
    arg = ReplaceAll(arg, "`", backslashAndBacktick)
    arg = ReplaceAll(arg, "$", backslashAndDollarSign)
    arg = ReplaceAll(arg, SQ, backslashAndSingleQuote)
    arg = ReplaceAll(arg, DQ, backslashAndDoubleQuote)
    
    arg = DQ + arg + DQ
    
    Return arg

    Either of these formulation work equally well as best I can tell. Just run the arguments through one or the other of these functions before passing them to pyShell.Execute

    pyShell.Execute(PATH_PYTHON3 + SPACE + PATH_PYTHON_SCRIPT + SPACE + FixSpecial(argument1) + SPACE + EscapedArgument(argument2))

    The Trial.py script might be:

    import sys
    def main():
    
        passedArg1=sys.argv[1]
        passedArg2=sys.argv[2]
        sReverseArg1 = PassedArg1[::-1]
        print(sReverseArg1)
        print(passedArg2)
    
    main()

    This suffices for when you want to accept and send back simple ASCII text.

    The next problem was sending and receiving back non-Ascii characters should you need to do that. Characters like é. Basically, the above does not work.

    It turns out that easiest solution really lies on the Python side of things.

    If you want to deal with Non-Ascii characters either when you receive them or send them back from Python when called from the Shell.Execute, try adding the codec "stuff" to the PYTHON_SCRIPT code

    import sys
    def main():
    
        import codecs
        sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())
    
        passedArg1=sys.argv[1]
        passedArg2=sys.argv[2]
        sReverseArg1 = PassedArg1[::-1]
        print(sReverseArg1)
        print(passedArg2)
    
    main()

    And then it works.

    **********

    P.S. Beatrix suggested the possibility of using BASE64 encoding to traverse the gap from Xojo to Python and then Python back to Xojo.

    BASE64 encoding represents the text in characters that are all part of the ASCII set and which contain none of the troublesome characters like $ and <. This methodology can also deal with non-ASCII characters because they get coded as BASE64 characters that are all ASCII.

    I submit this here just in case some situation arises that this approach is preferable. I did get it to work although not very elegant.

    Const SPACE = " "
    Const PATH_PYTHON3 = "/Users/owl/anaconda3/bin/python"
    Const PATH_PYTHON_SCRIPT = "/Users/owl/Documents/PythonPractice/PythonLittleApps/TrialB64.py"
    Const NO_LINE_BREAK = 0
    
    Dim argument1 As String
    argument1 = "abcde"
    Dim argument2 As String
    argument2 = "pqrst"
    
    Dim pyShell As New Shell
    pyShell.Execute(PATH_PYTHON3 + SPACE + PATH_PYTHON_SCRIPT + SPACE + EncodeBase64(argument1, NO_LINE_BREAK) + SPACE + EncodeBase64(argument1, NO_LINE_BREAK))
    
    Dim sResult As String
    sResult = DecodeBase64(pyShell.Result, Encodings.UTF8)
    MsgBox(sResult)

    On the Python side of things, you do not have to use codec "stuff" because relying on BASE64 encoding and decoding.

    import sys
    def main():
        import os
        import base64
        # decode to get the string that was passed
        incomingBase64Arg1 = sys.argv[1] # gets the Base64 from Xojo
        bDecodedArg1 = base64.b64decode(incomingBase64Arg1) # get the binary string
        utf8DecodedArg1 = bDecodedArg1.decode('utf-8') # convert to a utf8 string
    
        # do something in Python (reverse for example)
        sReverseArg1 = utf8DecodedArg1[::-1]#+"<CR>"#+"\n"
    
        # code in preparation for sending it out from Python to Xojo
        outgoingBase64Arg1=base64.b64encode(bytes(sReverseArg1,'utf-8')).decode('utf-8')
    
        # Put in output buffer
        print(outgoingBase64Arg1)
    main()

    I apologize for any typo's or stupid things that may exist in the code samples.

  20. Tim J

    Jun 11 Pre-Release Testers, Xojo Pro Phoenix, AZ USA (desication ce...

    The Tkinter occam that ties the TCL/TK UI components into Python is your best option for truly cross platform support - every Unix, Mac OS, and Windows supports it. You could always look into PAGE for building a Tkinter GUI for Python use. While not as directly tied in like JavaFX Scene Builder or Xojo, it's still pretty good at giving you what you need (and enforcing a MVC model design).

    PAGE Editor on Sourceforge

  21. Robert L

    Jun 11 Federal Way, WA (Seattle Area)

    Thanks. But my intent is to use Xojo for lots of the coding work and not only the GUI. So I can use Python if I need it to crack part of a problem and Xojo to crack other parts. And make the GUI in Xojo.
    Fortunately, since this is all for myself, I do not have to worry about dual platform.
    But dual platform is certainly interesting in that Xojo CAN do it but injects a little more complexity with edge stuff.
    Meanwhile, I will look at some of these Python GUI builders. But I do not see one that everybody says this is great and easy.

or Sign Up to reply!