Passing parameters to Python script with Shell.Execute

  1. 2 weeks ago

    Robert L

    Jun 7 Federal Way, WA (Seattle Area)

    Mac OS 10.13.5
    Xojo 2018 r1.1

    My goal is to process some data in Python using the Shell.Execute command. I am passing that data as "parameters" of the Shell.Execute command. As I understand it, you can consider the string that is passed in the Shell.Execute command as a bunch of space-delimited "parameters".

    If you want to fire up a Python script, then the first "parameter" I am passing is an absolute path to Python on my machine. The second "parameter" is the absolute path to the Python script that I am intending to run. After this, additional "parameters" that are provided are passed to the Python script as arguments.

    For this example, consider trying to pass two strings as arguments to the Python script.

    Const SPACE = " "
    Const PATH_PYTHON3 = "/Users/owl/anaconda3/bin/python"
    Const PYTHON_SCRIPT = "/Users/owl/Documents/PythonPractice/PythonLittleApps/Trial.py"
    
    Dim argument1 As String
    argument1 = "abcde"
    Dim argument2 As String
    argument2 = "pqrst"
    
    Dim pyShell As New Shell
    pyShell.Execute(PATH_PYTHON3 + SPACE + PYTHON_SCRIPT + SPACE + argument1 + SPACE + argument2)

    This works fine. In the Python script I am successful in accessing the argument1 and argument2.

    However, there are many characters that cause problems for me when included in argument1 or argument2 strings.

    Certainly, you cannot have a space. That makes sense since spaces are used as delimiters. However, in my brief testing I have run into other problems. < and | and & are all characters that are "unacceptable" and causes errors

    For example if argument1 is "abc&de" is passed I get an error.

    bash: de: command not found

    I deduce that "&" is a special character somehow specifying a bash command. "<" and "|" also cause trouble. I presume there are others.

    Well, suffice to say, I know nothing much about bash. I do not know the language.

    What I want to be able to do is to pass any string in argument1 and argument2 for processing by Python. But I am getting tripped up.

    Is there some "finite" list of unacceptable characters that I can find and try to filter out? Is there some methodology that prevents the characters being passed from having some special meaning (like initiating some bash command) so they will be successfully passed to Python?

  2. Beatrix W

    Jun 7 Pre-Release Testers Europe (Germany)

    I've done the following when sending data to Python that might have problems with encoding: In Xojo I base64-encoded the string and let Python do the base64-decoding.

  3. Tanner L

    is not verified Jun 7 Pre-Release Testers Toronto, Canada

    @RobertLivingston Is there some "finite" list of unacceptable characters that I can find and try to filter out? Is there some methodology that prevents the characters being passed from having some special meaning (like initiating some bash command) so they will be successfully passed to Python?

    Here's an old reference to special characters and escape sequences on OSX. Hope it helps.

    https://www.macobserver.com/tips/macosxcl101/2002/20020531.shtml

  4. I'm not somewhere I can test at the moment but what happens if instead of this:

    pyShell.Execute(PATH_PYTHON3 + SPACE + PYTHON_SCRIPT + SPACE + argument1 + SPACE + argument2)

    you do this:

    pyShell.Execute(PATH_PYTHON3, PYTHON_SCRIPT + SPACE + argument1 + SPACE + argument2)
  5. Robert L

    Jun 7 Federal Way, WA (Seattle Area)

    I'm not somewhere I can test at the moment but what happens if instead of this:

    It makes no difference

  6. Markus W

    Jun 7 Pre-Release Testers, Xojo Pro #JeSuisHuman Germany, Heidelb...

    Try putting quotes around your arguments

    Const quote = chr(34)
    Param1 = quote + “abc&def” + quote
  7. Robert L

    Jun 7 Federal Way, WA (Seattle Area)

    In reply to Tanner,

    Here's an old reference to special characters and escape sequences on OSX. Hope it helps.

    Characters Use
    * ? [ ] ^ { } ~• • • • Filename pattern matching and expansion
    $ • • • • • • • • • • • Used to denote variables
    ! ^ • • • • • • • • • • • History reference and substitution
    & • • • • • • • • • • • Used to send commands to the background
    | • • • • • • • • • • • Pipe
    > < • • • • • • • • • •Redirects output/input
    ; • • • • • • • • • • • Command seperator
    space • • • • • • • Argument seperator
    tab • • • • • • • • • Filename completion (in tcsh)
    esc • • • • • • • • • •Filename completion (in csh, may cause problems in tcsh)
    ( ) • • • • • • • • • • •Subshell execution
    ` • • • • • • • • • • •Quote used for command substitution
    \ ' " • • • • • • • • • •Quote characters

    It does help. Apart from the "Filename pattern matching and expansion" group these characters are mostly "bad actors" as far as my use is concerned.

    As I was testing these various characters, I noticed that "$" and "\" did not actually cause an error, they just did not appear in the parameter as seen by Python. That got me thinking about the possibility of "\" being an escape characters as is so frequently the case in programming situations. (I do not know the purpose of $ just disappearing)

    That works. The bad actors can be escaped: $ -->\$ or ; --> \; or | --> \|

    If they are escaped, they get to Python just fine.

    So if I know all the bad actors, I can write a routine to substitute the escaped version before submitting the string as a parameter. I just need a "complete" list of bad actors and Tanner has pointed me to at least some of them.

    I submit this because my experience might be useful in some contexts for myself and others. But if the below holds up, it is better.

    Try putting quotes around your arguments

    In my limited testing, Markus's suggestion works and has the huge advantage of being "simple". If I run into glitches down the road, I will repost. For now this would seem to be the solution to my problem.

    I thought that Beatrice's approach was very clever. A trick to keep in the back of my mind. But it is nice to avoid the overhead if Markus's idea holds up.

  8. Markus W

    Jun 7 Pre-Release Testers, Xojo Pro #JeSuisHuman Germany, Heidelb...
    Edited 2 weeks ago

    If your arguments contain single or double quotes then you might want to escape them with the backlash. But the others should be covered by passing it in quotes.

    So I would make a method EscapedArgument with

    Function EscapedArgument( arg as string ) as String
    
    arg = ReplaceAll( arg, SingleQuote, BackslashAndSingleQuote )
    arg = ReplaceAll( arg, DoubleQuote, BackslashAndDoubleQuote )
    
    arg = quote + arg + quote
    
    Return arg
    
    End Function

    and use it like

    pyShell.Execute(PATH_PYTHON3 + SPACE + PYTHON_SCRIPT + SPACE + EscapedArgument( argument1 ) + SPACE + EscapedArgument( argument2 ) )

    You could also make it an Extension method and call it InEscapedForm

    Function InEscapedForm( extends arg as string ) as String

    and use it like

    pyShell.Execute(PATH_PYTHON3 + SPACE + PYTHON_SCRIPT + SPACE + argument1.InEscapedForm + SPACE + argument2.InEscapedForm )

    Note: not on my Mac, so typing this on my iPad

  9. Tim J

    Jun 7 Pre-Release Testers, Xojo Pro Phoenix, AZ USA (desication ce...

    I have a method named "FixSpecialWithSpaces":

    Function FixSpecialWithSpaces(s As String) As String
      For Each t As String in Split("\ ` ~ ! @ # $ % ^ & * ( ) | { } [ ] ' < > ? """, " ")
        s = s.ReplaceAll(t, "\" + t)
      Next
      
      s = s.ReplaceAll(" ", "\ ")
      
      Return s
    End Function
  10. Robert L

    Jun 7 Federal Way, WA (Seattle Area)

    The $ character is particularly pernicious in the setting of passing strings. I said above that it simply "disappeared" when it was not escaped, but it actually disappears and takes all the characters that are positioned after it with it. They all disappear as well.

    This bad mojo is not defeated by simply enclosing the string in Double Quote as per Markus.

    To deal with $, in my experience, requires that you specifically take care to escape it.

    So I would add two lines to the code(EscapedArgument) that Markus outlined.

    Dim backslashAndDollarSign As String = "\" + "$"
    arg = ReplaceAll(arg, "$", backslashAndDollarSign)
  11. Tim J

    Jun 7 Pre-Release Testers, Xojo Pro Phoenix, AZ USA (desication ce...
    Edited 2 weeks ago

    It's more than just the $ character. That's why my function above handles them all.

    The reason that the characters following the $ are lost is because the shell sees $TheStuff as a shell variable named TheStuff and does inline replacement (with "" in this case). You are correct that quoting the string does not fix it, it must be escaped.

    I strongly recommend Dave Taylor's "Learning Unix for OS X: Going Deep With the Terminal and Shell " as a guide to getting familiar with the Unix Terminal intricacies. He knows that you're not coming into the book as a Unix expert, but he also doesn't talk down to you as if you were a total computer "newb" (as my support lead would say).

  12. Robert L

    Jun 8 Federal Way, WA (Seattle Area)

    The original problem was providing string parameters to a Python script using Shell.Execute. Problems occur with certain characters. They need to be escaped or dealt with in some way.

    Tim provided a list of characters that he suggested to escape and provided some code to do just that.

    \ ` ~ ! @ # $ % ^ & * ( ) | { } [ ] ' < > ? "

    Markus suggested that simply embedding the parameters in quotes would do the job. I liked that solution, but found that $ required being escaped, despite the string being embedded in quotes. Tim then commented that more than $ would cause problems.

    I tested all the characters in Tim's list and found only one additional character "`". that still had to be escaped. So for my particular situation, it appears that using Markus's solution works if I additionally take care to specifically escape "$" and "`"

    The "double quote" character presents its own challenge that I will not go into. I can use Tim's solution to handle this,

    Tanner pointed to another character that causes problems, the semicolon ";", which is not on Tim's list. That particular character does not cause problem if part of a parameter enclosed in quotes as per Markus.

    I am not sure that between Tim and Tanner, I have a complete list of the potential problem characters. (As per Tim's suggestion, I did buy, but have not finished, the Learning Unix of OS X). But I have not run across anything else.

    For now, I have added these lines to the Markus solution.

    Dim backslashAndDollarSign As String = "\" + "$"
    Dim backslashAndBacktick As String = "\" + "`"
    
    arg = ReplaceAll(arg, "$", backslashAndDollarSign)
    arg = ReplaceAll(arg, "`", backslashAndBacktick)

    It seems to me that the Tim solution should also work although I might wonder if ";" needs to be added to the list.

    P.S. One little trick that perhaps someone does not know. On the Mac, you can get the path of a folder or file by selecting it and keying Option - Command - C which places that path on the clipboard. That path will not be "escaped" however. If you want the escaped version, you can drag the icon of a folder or file onto the Terminal window which will then show the escaped version of the path. You can name folders on the Mac with a startling number of weird characters and then drag that folder onto the Terminal and see if that characters is escaped (preceded by a \ [backslash]). A way to test for characters that Unix deals with "specially" as I would interpret it.

  13. Tim J

    Jun 8 Pre-Release Testers, Xojo Pro Phoenix, AZ USA (desication ce...

    @RobertLivingston It seems to me that the Tim solution should also work although I might wonder if ";" needs to be added to the list.

    And that was a copy/paste error on my part. The semicolon is in my actual functioning code.

    To further explain the characters above and why I wrapped them, look into the way a shell handles the characters. Here's a great BASH script that will show the characters that you should escape when passing them through the command line as arguments:

    for i in {0..127} ;do
        printf -v var \\%o $i
        printf -v var $var
        printf -v res "%q" "$var"
        esc=E
        [ "$var" = "$res" ] && esc=-
        printf "%02X %s %-7s\n" $i $esc "$res"
    done | column

    In an 80 column terminal, the output will look like this:

    00 E ''     	1A E $'\032'	34 - 4      	4E - N      	68 - h      
    01 E $'\001'	1B E $'\E'  	35 - 5      	4F - O      	69 - i      
    02 E $'\002'	1C E $'\034'	36 - 6      	50 - P      	6A - j      
    03 E $'\003'	1D E $'\035'	37 - 7      	51 - Q      	6B - k      
    04 E $'\004'	1E E $'\036'	38 - 8      	52 - R      	6C - l      
    05 E $'\005'	1F E $'\037'	39 - 9      	53 - S      	6D - m      
    06 E $'\006'	20 E \      	3A - :      	54 - T      	6E - n      
    07 E $'\a'  	21 E \!     	3B E \;     	55 - U      	6F - o      
    08 E $'\b'  	22 E \"     	3C E \<     	56 - V      	70 - p      
    09 E $'\t'  	23 E \#     	3D - =      	57 - W      	71 - q      
    0A E $'\n'  	24 E \$     	3E E \>     	58 - X      	72 - r      
    0B E $'\v'  	25 - %      	3F E \?     	59 - Y      	73 - s      
    0C E $'\f'  	26 E \&     	40 - @      	5A - Z      	74 - t      
    0D E $'\r'  	27 E \'     	41 - A      	5B E \[     	75 - u      
    0E E $'\016'	28 E \(     	42 - B      	5C E \\     	76 - v      
    0F E $'\017'	29 E \)     	43 - C      	5D E \]     	77 - w      
    10 E $'\020'	2A E \*     	44 - D      	5E E \^     	78 - x      
    11 E $'\021'	2B - +      	45 - E      	5F - _      	79 - y      
    12 E $'\022'	2C E \,     	46 - F      	60 E \`     	7A - z      
    13 E $'\023'	2D - -      	47 - G      	61 - a      	7B E \{     
    14 E $'\024'	2E - .      	48 - H      	62 - b      	7C E \|     
    15 E $'\025'	2F - /      	49 - I      	63 - c      	7D E \}     
    16 E $'\026'	30 - 0      	4A - J      	64 - d      	7E - ~      
    17 E $'\027'	31 - 1      	4B - K      	65 - e      	7F E $'\177'
    18 E $'\030'	32 - 2      	4C - L      	66 - f      
    19 E $'\031'	33 - 3      	4D - M      	67 - g      
  14. Robert L

    Jun 8 Federal Way, WA (Seattle Area)

    In an 80 column terminal, the output will look like this:

    Perfect

  15. Markus W

    Jun 8 Pre-Release Testers, Xojo Pro #JeSuisHuman Germany, Heidelb...
    Edited 2 weeks ago

    I always find that proper naming and an extra line can make code MUCH more readable. So taking Tim’s code:

    Function Escape( theString As String) As String
    
      Dim CharactersThatMustBeEscaped as String = ("\ ` ~ ! @ ; # $ % ^ & * ( ) | { } [ ] ' < > ? """)
     
      For Each char As String in Split( CharactersThatMustBeEscaped, space )
        theString = theString.ReplaceAll( char, "\" + char)
      Next
      
      theString = theString.ReplaceAll( space, "\" + space )
      
      Return theString
    
    End Function

    or

    Function Escape( theString As String) As String
    
      Dim ArrayOfCharactersThatMustBeEscaped() as String = Split("\ ` ~ ! @ ; # $ % ^ & * ( ) | { } [ ] ' < > ? """, space)
    
      ArrayOfCharactersThatMustBeEscaped.append space
     
      For Each char As String in ArrayOfCharactersThatMustBeEscaped
        theString = theString.ReplaceAll( char, "\" + char)
      Next
      
      Return theString
    
    End Function
  16. Robert L

    Jun 8 Federal Way, WA (Seattle Area)

    And the proper way to deal with characters outside of the ASCII code set?

    For example: cliché

    é causes a problem when I try and pass it to Python using Script.Execute.

    As for "verbose code" with explanatory names and extra lines, I am a big fan. I know at my age that I will be confused when I go back to the code after a few weeks/months. I happily convert the "concise" code that many prefer to my own style. Just the act of doing it, helps me with understanding the contributed code better. And I never look at a gift horse in the mouth.

  17. Markus W

    Jun 9 Pre-Release Testers, Xojo Pro #JeSuisHuman Germany, Heidelb...

    @RobertLivingston And the proper way to deal with characters outside of the ASCII code set?

    For example: cliché

    é causes a problem when I try and pass it to Python using Script.Execute.

    Don’t use them ;)

    But seriously: “pass” as what?

    é in a string is fine, but as a variable name probably not

  18. Robert L

    Jun 9 Federal Way, WA (Seattle Area)

    Don’t use them

    I can accept that. But my original purpose was to pass strings to Python, manipulate them there, and then bring the result back into Xojo.

    With the help of you and others, I succeeded in doing just that with the ASCII characters. The mechanism that I am using is

    pyShell.Execute(PATH_PYTHON3 + SPACE + PYTHON_SCRIPT + SPACE + argument1 + SPACE + argument2)

    But seriously: “pass” as what?

    Pass as what is effectively a "parameter". With the syntax above, I can pass this string data to the Python script for it to work on.

    argument1 and argument2 are strings (and now I have learned to properly escape the problematic characters when they are passed to Python. ) So if I want to come up with a way to do this "in general" I have to be aware that this mechanism will not work for characters outside of ASCII. So I do not know how to use such a mechanism to pass a character like é to a Python script.

    Beatrix suggested early on the possibility of using Base64 encoding and perhaps that would work. I have not yet tried it and was wondering whether there was a more commonly accepted methodology.

    I have also yet to explore an alternate "technique" of writing a text file to disk using Xojo and then having the Python script that I have launched using Xojo (and I could pass its absolute path) read the contents of that file into Python. It seems to me that that would likely address this issue, but again I do not want to go down that route if there is a more straight-forward way of getting the strings from Xojo to a Python script that was called by Xojo using Shell.Execute.

  19. Markus W

    Jun 10 Pre-Release Testers, Xojo Pro #JeSuisHuman Germany, Heidelb...

    Why do you want to manipulate the strings in Python instead of in Xojo? The only reason I could see is for example to use BioPython, but in that case ASCII should do fine …

  20. Robert L

    Jun 10 Federal Way, WA (Seattle Area)

    Why do you want to manipulate the strings in Python instead of in Xojo?

    Short answer. It is more than strings.

    Intermediate length answer :)

    Xojo and Python are very different places (compiled vs interpreted; commercial vs open source; obscure vs hugely popular; small number of third party resources vs huge libraries of third party add-ons; easy creation of graphical user interface vs difficult and immature attempts)

    I am really trying to create the infrastructure that would allow me, when working on projects in Xojo, to drop down into Python and perform certain tasks in that environment and then return the results to Xojo for display or further refinement. My hope is that if I can overcome/understand all the glitches here, and figure out the process, then going forward I will not have to worry about the issue.

    I am an intermediate Xojo programmer and a beginning Python programmer. I like creating apps for my own use and in some cases offer them for free to others. But I am not a commercial developer so paying $ for third party add-ons to Xojo is onerous for me. The extensive library add-ons of Python are free.

    I generally prefer writing my code in Xojo but even as a beginning Python programmer I see certain approaches to problems are easier for me in Python. My biggest problem in Python is not being able to create applications with nice GUI interfaces which I can do in Xojo.

    I am not solely interested in processing strings in Python. It so happens that strings are the only way I know (apart from writing files to disk to access from both environments) to pass data back and forth between Xojo and Python. Changing strings back numbers or arrays or whatever is not hard in either environment. So the string emphasis is just an artifact of trying to come up with a smooth way of pushing data back and forth.

    There are circumstances that in fact I do want to process strings in Python rather than in Xojo. You mention BioPython as an example. I find working with Regex in Python (with re library) is easier for me than the same in Xojo. One thing that launched me in this direction of trying to learn Xojo/Python intercommunication was an occasion that I wanted to use Levenshtein distance as a tool to find name matches in a context with a couple thousand name and some of the names were misspelled. Implementing Levenshtein distance in Xojo code is complicated for me, and I see in Python that I can find a library that will supply this code without my having to understand it and actually implement it myself.

    NumPy & SciPy supply all these tools for scientific computing.

  21. Newer ›

or Sign Up to reply!