HTTPSocket and Apache 2.4.42

I have a Xojo web app that makes a simple call to a web service that authenticates the user. After upgrading the server from ancient Apache 2.2.15 to 2.4.41 my Xojo app could no longer receive/process the result coming back from the server. I know that I’m making an HTTP/1.0 Post operation and thus am expecting a similarly versioned response. I’ve used PAW and/or Curl to test the web service call with no issues under either version of Apache. I assumed the updated service was sending back the wrong format so I added the directive to Apache to always send back HTTP/1.0 and confirmed the response has the HTTP/1.0 headers now…and yet my Xojo program just doesn’t seem to acknowledge a response. It “times out”. Even though my Server logs indicate that the POST was received…processed…and a response was returned. If I shut down Apache 2.4.41 and restart Apache 2.2.15 – the Xojo app works fine again. Ive been comparing the raw response from Curl and see no difference in the response. Since the problem only plagues my server running Apache 2.4.x I have to assume it’s fixable by a configuration change. I don’t really want to change the Xojo program to use the HTTP 1.1 newer POST format if I don’t have to. The argument values I’m transferring are in Form Data format. Any ideas appreciated. I really need to run the more modern (security patched) Apache version.

you could try to play around with SetEnv and force-response-1.0, downgrade-1.0 or force-no-vary
Source: https://httpd.apache.org/docs/2.4/env.html

Is Apache keeping the connection alive (indicated by the response header Connection: keep-alive)? If so then try to get it to close the connection after each request (Connection: close).

I had a similar (but likely different) issue when an Apache upgrade stopped accepting duplicate Host headers - https://forum.xojo.com/55250-httpsocket-apache-and-duplicate-host-headers/0

There are good ideas in that thread (even though none of them worked for me.)

Yes, I had tried the SetEnv with force-no-vary with no success. I did use Curl to verify that the HTTP headers being returned are now HTTP/1.0 – and yet still the Xojo program was unhappy. I don’t believe the connection is being kept alive. In fact, I can shut down the new apache server and fire up the old version and the app starts working again…then I shut-down the old version and switch back to the new and it’s failing again. I’ve tried stopping and restarting the Xojo App – uncertain why that would be necessary since other apps like PAW have no problem hitting either server. The only real change I see in the HTTP response is the Apache Version. It’s very strange. The authentication web service ultimately runs a system script – which I can trace and see the request was received and processed by the server…but the Xojo app “times out” waiting on a response. (I placed a break point as soon as the result comes back) When it times out I have a nil result. It’s just crazy. Maybe I can set some sort of debug level so I can see the exact outgoing response from Apache. Figured somebody may have run into this before. I was able to reproduce the issue on Xojo versions 2017.x - 2019.x – of course all other web services on this box are working just fine. Thank you everyone for your thoughtful replies. They are all worthwhile.

There’s a similarity in our stories: my Xojo app failed while other tests (using Curl, or just a web browser) would work.

To debug it, I ended up doing a low level packet capture on macOS as described here https://support.apple.com/en-us/HT202013 Seeing the actual raw data being transfered was key to figuring out the problem.

Re-reading your description again, I have an idea: could this be Chunked transfer encoding? https://en.wikipedia.org/wiki/Chunked_transfer_encoding

here’s an example where Chunked is being sent to HTTP1.0 clients and causing trouble: https://github.com/linkerd/linkerd/issues/1847

I’m getting closer… I used the tip about trapping network packets on my Mac. The “send” operation looks fine…but I really don’t see anything returned from Apache. I built a very small test app aside from a button and a textbox on a window…this is all the code.

Dim myhttp As New HTTPSocket
Dim authserviceUrl as String
Dim runresult as String

authserviceUrl = “http://123.456.789.123/cgi-bin/wcps.account.list.cgi
runresult = myhttp.Post(authserviceUrl,10)
TextArea1.Text = runresult + " ERRORCODE:" + str(myhttp.ErrorCode)

It works perfectly with my old Apache 2.21 running on the server. Not so much on Apache 2.41 – I’m sure it’s one of the myriad of options for handling CGI. I don’t think it’s chunking because the my Curl test call with current options doesn’t return with chunking enabled. It’s something about CGI handling. I wanted to test updating my sample code to use HTTP1.1 via Xojo.Net.HttpSocket. I know that a page return is done via events (runs asynchronously). So I’m trying to modify the above code to work

myhttp is a generic object dragged from the controls and modified with a superclass of Xojo.Net.HttpSocket

Dim authserviceUrl as String
authserviceUrl = “http://139.126.108.157/cgi-bin/wcps.account.list.cgi
myhttp.Send(“POST”, authserviceUrl)

I updated the PageReceived handler to simply display the error code

TextArea1.Text = " ERRORCODE:" + str(HTTPStatus)

I know I’d have to do a little more to get a STRING out of the Xojo.Core.memoryblock. I’m missing something as it won’t compile. I do really like the asynchronous design, but have to admit that in this instance…there’s nothing wrong with waiting on a synchronous event. The timeout is a nice feature.

Okay I thought you are communicating with a stand alone webserver behind an Apache just acting as Reverse Proxy. If you are using CGIs then this is somehting different. If your attempts fail, then I would suggest to use a stand alone webserver rather than a .CGI

There were a lot of changes to cgi and authentication between 2.2 and 2.4

What I have posted below is from a working server and should be enough for a simple functional test. You will need to substitute /home/${VHOST}/cgi-bin with the path to your own cgi script folder.

Beyond that, https://httpd.apache.org/docs/trunk/upgrading.html

Sorry, I don’t have any examples that use POST.

# Apache 2.4.6 httpd.conf extract

        #set cgi folder, params fakename "real path"
        ScriptAlias /cgi-bin/ "/home/${VHOST}/cgi-bin/"

        <Directory "/home/${VHOST}/cgi-bin">
                DirectoryIndex index.cgi
                AllowOverride None
                Options None
                Require all granted
        </Directory>
#!/usr/bin/perl
# index.cgi
print "content-type: text/html\
\
";
print "<html><body>\
";
print "<p1>cgi-bin is configured</p>\
";
print "Time at server: ";
system "date\
";
print "</body></html>\
";
// XOJO 2019r2.1 Web Edition
Dim myHttp as new HTTPSocket
Dim URL As string = "http://server.invalid/cgi-bin/index.cgi"
Dim result As String

result = myHttp.Get(URL, 10)
TextArea1.Text = result

I thought I’d let everyone know I did isolate the problem…I still don’t know why it’s a problem…but here it is…
For testing I built a very simple script that doesn’t even take any parameters. Something like this.

#!/usr/bin/perl
print "content-type: text/html

";
$reply = pwd;
print "The working directory is $reply ";

I found that the above FAILS – however…IF I perform the command before I print the html header… it works. like this (below)

#!/usr/bin/perl
$reply = pwd;
print "content-type: text/html

";
print "The working directory is $reply ";

This wasn’t an issue under my older Apache instance… but my new Apache instance doesn’t like it at all.
Perhaps because it doesn’t have the expected html and body elements. This the “working” example doesn’t have them either…and it works just fine under my new Apache instance.

Any effort to “shell-out a command” causes a failure IF I’ve already started printing the page content back. I’ve not seen any difference with chunking settings etc. I’ve tried forcing an html 1.0 response and verified the setting works…but made no difference.

A sincere thank you to everyone who contributed their thoughts and ideas. It put me on the path to a solution.

thank you for updating this thread. after reading your .cgi perl experiments I strongly advice not to use these .CGIs anymore and better switch to xojo webapps as stand alone web services behind an apache reverse proxy for security reasons. Otherwise any tampered parameter will be interpreted and run against your system e.g. http://sensitive/something?dir=%3Bcat%20/etc/passwd

Sounds like good advice Jacob, not sure I understand completely. The Xojo app itself is running as standalone, not as CGI. I’ve been passing parameters as Form Parameters or as JSON (not as URL parameters) – still since they are not being sent via HTTPS – I suppose they can be intercepted and altered. Someone could provide malformed parameters such that shelling out could cause some mayhem. Typically I encode the parameters in the script before shelling out with them…and then decode them in the shelled out command. Also, this Xojo program and associated CGI script is only used internally. The server etc is not available via the internet etc. The authentication is more about convenience for the user having their jobs stamped with their username…and alerts going to their email address (retrieved via their user credentials).

Robert, that is truly bizarre. Are you sure the two cgi files are otherwise identical? (e.g. both have 755 permissions, no invisible unicode characters in the files, etc.)

I wonder if it could be https://httpd.apache.org/docs/trunk/mod/mod_cgid.html#cgidscripttimeout that is misconfigured?

  • in the first case, the content-type header is output, and Apache starts the timer. Then the cgi script has to fork to output the pwd command, and since this takes a non-zero amount of time, mod_cgi assumes the process has died.
  • in the second case, the pwd command runs first, and so the output lines are all generated at once, so mod_cgi doesn’t think your process is dead?

Edit to add: and perhaps there’s an interaction with Connection: keep-alive that is not being sent by Xojo but is being sent by other test apps?

Of course is your Apache talking https with valid certs (e.g. lets encrypt) to your visitors
Only the reverse side of the Apache and your Xojo Webservices are internally talking unencrypted via http.

Apache really doesn’t care about HTML tags.

Apache output is heavily bufferred and similar to PHP, the recommended approach is to assemble the response and write to the buffer in one shot. Apache can be fussy about end of line delimiters too. The last two characters in the response buffer should be an LF/CR pair.

# try changing this
print "The working directory is $reply ";

# to this
print "The working directory is $reply\
"';

My actual test is just a single script with the same $reply=pwd; occurring both before and after the printing of the “Content-type…” line. I just toggle off/on the line I want. I checked for hidden characters. None found. I actually was using a newline
terminator (I just failed to copy that in my example here) – For fun I tried removing it…and it makes no real difference. I just get an extra linefeed at the end of my returned page when I add that. I also checked permission and ownership of the file. No problem there. I added the CGIDScriptTimeout option to the mod_cgid configurations. No difference there. It is very strange. I tried not even outputting the captured reply and it doesn’t work either. It seems that IF I perform a external call out …it has to be prior to the output of Page. Even something as “instant” as pwd or and echo command. I wonder if my Apache used mod_cgi and my new one uses mod_cgid — maybe there is a difference in there. I’ve figured out how to work around the problem, but still don’t know why I have a problem. It is curious, but may have to be filed under my “unknown mysteries of the universe”.

@Robert Bednar I think I may have isolated your problem but lack the time to research a resolution.

Below are a couple of dumps from Wireshark. Apache sends the response in both cases but when pwd is called within the response Apache drops the content-length header. There may be a .htaccess tweak to force the header but I don’t have time to dig into the Apache documentation at the moment.

The relevant section of the protocol spec is here
https://www.w3.org/Protocols/HTTP/1.0/spec.html#Content-Length

Use of the word ‘should’ in a protocol spec usally means ‘optional’ in the wild.

# pwd called within response - no output from Xojo
00000000  47 45 54 20 2f 63 67 69  2d 62 69 6e 2f 74 65 73   GET /cgi -bin/tes
00000010  74 31 2e 63 67 69 20 48  54 54 50 2f 31 2e 30 0d   t1.cgi H TTP/1.0.
00000020  0a 41 63 63 65 70 74 3a  20 2a 2f 2a 0d 0a 41 63   .Accept:  */*..Ac
00000030  63 65 70 74 2d 4c 61 6e  67 75 61 67 65 3a 20 65   cept-Lan guage: e
00000040  6e 0d 0a 48 6f 73 74 3a  20 xx xx xx xx xx xx xx    n..Host:  xxxxxxx
00000050  xx xx xx xx xx xx xx xx  xx xx xx 0d 0a 0d 0a      xxxxxxxx xxx....
    00000000  48 54 54 50 2f 31 2e 31  20 32 30 30 20 4f 4b 0d   HTTP/1.1  200 OK.
    00000010  0a 44 61 74 65 3a 20 57  65 64 2c 20 31 35 20 4a   .Date: W ed, 15 J
    00000020  61 6e 20 32 30 32 30 20  30 38 3a 34 38 3a 30 32   an 2020  08:48:02
    00000030  20 47 4d 54 0d 0a 53 65  72 76 65 72 3a 20 41 70    GMT..Se rver: Ap
    00000040  61 63 68 65 2f 32 2e 34  2e 36 0d 0a 43 6f 6e 6e   ache/2.4 .6..Conn
    00000050  65 63 74 69 6f 6e 3a 20  63 6c 6f 73 65 0d 0a 43   ection:  close..C
    00000060  6f 6e 74 65 6e 74 2d 54  79 70 65 3a 20 74 65 78   ontent-T ype: tex
    00000070  74 2f 68 74 6d 6c 3b 20  63 68 61 72 73 65 74 3d   t/html;  charset=
    00000080  55 54 46 2d 38 0d 0a 0d  0a                        UTF-8... .
    00000089  54 68 65 20 77 6f 72 6b  69 6e 67 20 64 69 72 65   The work ing dire
    00000099  63 74 6f 72 79 20 69 73  20 xx xx xx xx xx xx xx   ctory is  xxxxxxx
    000000A9  xx xx xx xx xx xx 2f 63  67 69 2d 62 69 6e 0a 20   xxxxxx/c gi-bin. 


# pwd called before response - Xojo processes response
00000000  47 45 54 20 2f 63 67 69  2d 62 69 6e 2f 74 65 73   GET /cgi -bin/tes
00000010  74 31 2e 63 67 69 20 48  54 54 50 2f 31 2e 30 0d   t1.cgi H TTP/1.0.
00000020  0a 41 63 63 65 70 74 3a  20 2a 2f 2a 0d 0a 41 63   .Accept:  */*..Ac
00000030  63 65 70 74 2d 4c 61 6e  67 75 61 67 65 3a 20 65   cept-Lan guage: e
00000040  6e 0d 0a 48 6f 73 74 3a  20 xx xx xx xx xx xx xx  n..Host:  xxxxxxx
00000050  xx xx xx xx xx xx xx xx  xx xx xx 0d 0a 0d 0a      xxxxxxxx xxx....
    00000000  48 54 54 50 2f 31 2e 31  20 32 30 30 20 4f 4b 0d   HTTP/1.1  200 OK.
    00000010  0a 44 61 74 65 3a 20 57  65 64 2c 20 31 35 20 4a   .Date: W ed, 15 J
    00000020  61 6e 20 32 30 32 30 20  31 37 3a 31 34 3a 30 36   an 2020  17:14:06
    00000030  20 47 4d 54 0d 0a 53 65  72 76 65 72 3a 20 41 70    GMT..Se rver: Ap
    00000040  61 63 68 65 2f 32 2e 34  2e 36 0d 0a 43 6f 6e 74   ache/2.4 .6..[b]Cont
    00000050  65 6e 74 2d 4c 65 6e 67  74 68 3a 20 34 38 0d 0a   ent-Leng th: 48..[/b]
    00000060  43 6f 6e 6e 65 63 74 69  6f 6e 3a 20 63 6c 6f 73   Connecti on: clos
    00000070  65 0d 0a 43 6f 6e 74 65  6e 74 2d 54 79 70 65 3a   e..Conte nt-Type:
    00000080  20 74 65 78 74 2f 68 74  6d 6c 3b 20 63 68 61 72    text/ht ml; char
    00000090  73 65 74 3d 55 54 46 2d  38 0d 0a 0d 0a 54 68 65   set=UTF- 8....The
    000000A0  20 77 6f 72 6b 69 6e 67  20 64 69 72 65 63 74 6f    working  directo
    000000B0  72 79 20 69 73 20 2f xx  xx xx xx xx xx xx xx xx   ry is /x xxxxxxxx
    000000C0  xx xx xx 2f 63 67 69 2d  62 69 6e 0a 20            xxx/cgi- bin. 

Good Job isolating the problem Mathew. This makes sense and explains why Xojo is having such a tough go in HTTP1.0 If the Content-Len isn’t being returned – that explains why I was getting the failure. I probably should switch over all my service calls to HTTP 1.1 — I had good luck with that in a test application. Wiresharking the packets to identify this issue is above and beyond any effort I expected from the community. I think you have exactly pinpointed the root cause of my trouble…and thankfully I have a simple enough workaround. Thanks again to the rest of the Xojo community. Sharing your ideas and helping troubleshoot is so helpful.