Eddie’s example has a WebTimer with a 2 second delay. If Oha (which I haven’t used before) is waiting for connections to become stable, then sure, the results will be affected.
I just wanted to clarify we’ve been (and we currently are) working on performance. The numbers are there and the archived versions of Xojo can be used to compare.
Sorry folks for the delayed response as it’s been a super busy weekend over here on my side and I haven’t been at the computer much.
I can fill in some of the details here… I just added a reply to the Testers thread and indeed @Ricardo is absolutely spot on in that there are some huge improvements on the wrk side (nearly 2X!!!) in reqs/sec.
But now for some bad news in that I scrolled back to my very first post on this thread which refreshed my memory that I had started with Eddie’s Electronics (but in actuality I did this in parallel with my own app as that’s what I really care about). But since then I forgot this detail so my follow up replies are actually from my own app (e.g. NOT Eddie’s). Doh! 
So needless to say, things are not going to match up at all.
I can also speak to why I’m only seeing negligible improvements from version to version within my own app. Although the landing page is relatively simple, there’s one SQL query involved. I haven’t profiled my code while load testing, but this could very well be my choke point.
Over the next day or so, I’ll do a revised test with Eddie’s that matches the very first post of this thread. I’ll test the following…
Across both wrk and oha with the following setup…
wrk
wrk -d 60s -c 100 --latency “https://mysever.com”
oha
oha -z 60s -c 100 --latency-correction --disable-keepalive “https://myserver.com”
Also in looking at oha’s switches, it appears there’s one to enable http/2 so if I have time I’ll try some runs with that as well.
Here’s the latest refreshed data. To see the latest beta data, see the other thread within the Testers category.
SERVER SETUP
DigitalOcean Droplet
Debian 12 x86
Premium AMD w/ NVMe SSD
2 GB / 1 AMD CPU
50 GB NVMe SSDs
Premium tier ($14/mo. as of this writing).
LIFEBOAT SETUP
nginx backend
No load balancing
Instances: 1
Max Req. Body: 1M
Manage --UploadPath: Off
Enable gzip compression: On
Enable HTTP/2: On
XOJO SETUP
For each Xojo version I opened the example project…
Sample Apps » EddiesElectronics » Web
Xojo Cloud Target: Turned Off
Linux Cloud Target: Turned On
LOAD TESTER SETUP
Each load tester was run twice with data captured from the second run.
The intent was to capture a “warm” run rather than “cold” one in case there’s any kind of caching at play.
PERFORMANCE METRICS
oha
wrk
ORIGINAL DATA
2019r3.2
oha
wrk
2023r4
oha
2024r1
oha
wrk
2024r1.1
oha
wrk
NOTABLE ITEMS
The first thing to call out is that under 2019r3.2, Eddie’s Electronics was only stable when running on debug mode on my Mac (M1, Sonoma 14.5). As soon as the binary was placed on the server, Eddie’s Electronics would run but as soon as the web page was fully loaded, it would immediately crash. Not likely a big deal as we just care about loading the page and assuming each load tester was able to do so for each connection, the data should be solid, but take this data with a grain of salt. But mind you as this is Web 1.0, this data was mostly captured to try and extrapolate the performance gains that Web 2.0 brings to the table.
The other notable item is that although I did attempt to capture data with the --http2 flag for oha to force HTTP/2 connections, this switch didn’t appear compatible with the other options so I didn’t pursue this further
The final thing to note is that just for kicks, I attempted to load Eddie’s Electronics in a browser while each load tester was running (not during the data captures above but separate runs). This was very interesting in that during the oha runs, the website was still very responsive with very quick page loads…
Firefox
Web Developer Tools » Network (Disable Cache Enabled)
wrk on the other hand effectively makes the server inaccessible during the runs and I was unable to get Firefox or even Safari to load up Eddie’s Electronics at all until wrk finished. But mind you I was doing all of these actions from the same Mac and from a data capture perspective to isolate things better, these tests should have been done on a different computer and even a different network.
So just to validate things a bit further, I tried loading up the page on Safari on my iPhone via cellular. Alas wrk was still such a heavy load on the server, the page once again refused to load until the test completed.
ANALYSIS & THOUGHTS
So although some of my analysis on the prior/incorrect data was incorrectly placed, my overall thoughts on small but incremental improvements still stand. Things are getting better and the choking point in this analysis might not be with the Xojo Web 2.0 server itself, but maybe something else such as SQL or maybe disk I/O. Regardless, things are still headed in the right direction.
To Matthew’s point about load times of 5 secs being meh, yes, I would tend to agree. But I’m also not convinced that it’s 5 secs to load based upon my cursory load in a web browser while oha was running with <1.5 secs load time. Maybe oha is having problems finding the final page load time due to requests intermittently being sent thereafter.
I would also agree with you Matthew regarding throwing money at performance bottlenecks. But alas, Xojo Web 2.0’s performance is what it is. If there is a bottleneck in the ServerSocket class, there’s not much we can do about it over here on our side other than scale out server resources to compensate. If there are other ways of mitigating this that I’m not thinking of, I’d love to hear from others. I know there’s many out there that have specialized in Xojo Web Development apps, so what are your best known methods here? What is missing in our discussion?
He also raised a solid point in that if wrk is showing 6224 requests/60 secs, then that means that the page only gets loaded 57 times in a minute (108 requests to load Eddie’s Electronics), which definitely adds credence to his analysis of “meh.”
But this also circles back to the discussion we’ve had about using the right tool for the right job. As I mentioned, Xojo Web 2.0 isn’t going to be the next Twitter, serving hundreds of millions of users. But for small to medium sized projects it should be a solid option depending upon architecture and the other positives that Xojo brings to the table (e.g. fast development times, shared code with other projects, etc.)
@Ricardo_Cruz It would be great to get your thoughts on all of this. On one hand maybe Eddie’s Electronics isn’t the best project to performance test against, but on the other hand, this is exactly the type of app that many Xojo users will be using Web 2.0 for. Also not to go too much into the weeds, this might be a valuable exercise in of itself to capture real-world best known methods to improve performance for Xojo Web Developers.
Well, just answering ‘meh’ is not much help is it 
Meh can actually be pretty darn good and more than enough to solve many things, so let me try to elaborate what ‘meh’ actually can mean. It can for instance mean the ability to deliver a 93kb image 9862 times, or a 1.3kb html file 89666 times in 20 seconds. To me that is not a bad thing at all, and more than enough for many things.
I don’t think Eddie’s Electronics example was prepared for this kind of testing. Falling Xojo Web into the SPA (Single Page Application) category of web apps, I think it would be more fair to test against a login web page, which is what users would normally see.
Like this one:
Examples > Platforms > Web > Managing User Login
For measuring complete user journeys (doing a login and then performing some actions), then tools like Puppeteer can help. Cucumber could be also interesting, as you could cover your app with BDD tests and then compare how long it takes to complete, between different versions of Xojo.
For benchmarking Xojo Web, I prefer to run the tests locally, as I don’t really want to be testing how fast is my connection. It can vary from day to day. Chrome (and Puppeteer) can be configured to emulate slow connections if needed.
The web framework also caches static assets in the browser (CSS, JS and fonts), so even if your first visit takes ~1s to load your website over the internet, when you press on refresh, most of the content will be cached in your browser and it won’t need to download them again.
I don’t think there is a bottleneck there (or in TCPSocket). At least at the moment, the slowest points in Xojo Web are coming from the web framework itself, which is what we’re improving.
Yes, you’re doing a much better job explaining this perspective than I was. This is precisely why I was originally placing things in the good to maybe suburb rating. For the right kind of app, with a user base not in the millions, Xojo Web 2.0 is more than enough. Especially now that we’ve worked through some oha issues and that actual req/sec is 8X the previous value, this further solidifies this stance.
But I also completely see where Matt is coming from, especially when comparing to industry leading large scale app deployments where there are ten to hundreds of millions of users. Xojo Web 2.0 won’t be able to compete, but it also wasn’t engineered for this kind of scale. Also to be fair, Matthew was originally seeing the skewed oha data of ~5 sec average load times and today’s numbers minus the other switches, is now showing closer to 0.5 sec.
Yes, this is a great idea and I’ll switch over for future tests. It doesn’t match what I’m doing with Xojo Web, but I agree it’s a pretty common use case.
Some great ideas here and definitely something to think about and ponder.
At least for me, I’m also super interested in determining thresholds for if/when I should start looking at upgrading or expanding out my infrastructure with Web 2.0. Although I’m not using Xojo Cloud (Rackspace backend IIRC), the same scenario holds there as well in determining at what point one should move up from a Small to Medium instance, maybe provide scaling into different worldwide regions, etc. But perhaps this is something that @Jason_Parsley might have some insight on.
This is fabulous news and although I haven’t commented on this, my own app indeed uses some REST APIs behind the scenes so I’m thrilled with all the latest r2.62932 improvements as well as the other work you and the team are doing here.
Hi Patrick - This varies greatly depending on what the app is doing. I’m a huge fan of real world (as close as possible) testing. I will contact you with more details.
