I’m migrating a bunch of webapps from a DigitalOcean Droplet (1 CPU 512MB ram) to an AWS t2.small (1 CPU 2GB ram) instance. The apps are running ok and responding, but the AWS CPU usage per app is hovering at 15-20% per app when idle. I have tested with a test app that has just one window and no code, and it too is idling at 15-20%. On the DigitalOcean Droplet, the idle load was 1-3% per app.
Has anyone any experience with AWS who might be able to shed light on why AWS CPU loading is so high?
The apps were compiled with Xojo 2015r2.4 (32-bit). They run standalone with Nginx reverse proxy.
The servers are configured with the same config files, the only differences between the two are:
DigitalOcean Droplet - 32 bit CentOS 6
AWS - 64 bit CentOS 7 with the necesssary 32-bit libraries installed (the apps run without error)
The AWS was upgraded from a t2.micro instance (10% of hardware CPU) to a t2.small (20% of hardware CPU) , but that hasn’t made any difference.
AWS uses Xen and started with S3 block storage by default. They optimize for maximum tenancy on compute nodes while storage is handled at the object storage layer. This means 1 vCPU most likely represents a smaller fraction of compute resources on AWS then it does on DigitalOcean.
DigitalOcean uses KVM and most likely has newer hardware. Some of AWS regions are quite old at this point. DigitalOcean always had local storage and so they could provide more CPU time per customer because local storage and memory is your biggest constraint.
My guess is a combination of different CPU scheduling, processor speeds, and tenants on hardware node.
If possible, try rebuilding using 2015r3 (or later) as this bug fix might be relevant:
370 Framework » Linux Reduced thread CPU usage on Linux for mostly idle threads. This difference will most likely be visible on single core processors, but this fix also benefits multi-core and other platforms.
Thanks Phiilip, very useful info. I’ve been happy with DigitalOcean, but I need to make this move for a third-party. As it happens, they just upgraded the AWS instance to 2 CPUs and 2GB and now the load per webapp is <1%. I’m surprised that there has been such a big change.
Be aware that AWS t2 instances are burstable in their performance. They will be punished for running too long at high CPU and then be throttled. See additional information here: https://aws.amazon.com/ec2/instance-types/
A fixed performance instance may be better suited.
[quote](from AWZ Instances type web-page)
Burstable Performance Instances
Amazon EC2 allows you to choose between Fixed Performance Instances (e.g. M3, C3, and R3) and Burstable Performance Instances (e.g. T2). Burstable Performance Instances provide a baseline level of CPU performance with the ability to burst above the baseline. T2 instances are for workloads that dont use the full CPU often or consistently, but occasionally need to burst.
T2 instances baseline performance and ability to burst are governed by CPU Credits. Each T2 instance receives CPU Credits continuously, the rate of which depends on the instance size. T2 instances accrue CPU Credits when they are idle, and use CPU credits when they are active. A CPU Credit provides the performance of a full CPU core for one minute.
For example, a t2.small instance receives credits continuously at a rate of 12 CPU Credits per hour. This capability provides baseline performance equivalent to 20% of a CPU core. If at any moment the instance does not need the credits it receives, it stores them in its CPU Credit balance for up to 24 hours. If and when your t2.small needs to burst to more than 20% of a core, it draws from its CPU Credit balance to handle this surge seamlessly. Over time, if you find your workload needs more CPU Credits than you have, or your instance does not maintain a positive CPU Credit balance, we recommend either a larger T2 size, such as the t2.medium, or a Fixed Performance Instance type.[/quote]
Thanks Matthias. It seems the AWS approach and plethora of options is better suited for organisations that have the technical resources to manage the servers. When the new release of Xojo comes out (soon I believe), I’ll upgrade and see if compiling with the latest version using 64-bit improves the situation and allows me to scale back the AWS instance. Up till now, I have found that 1 CPU core and 1GB ram have been fine on other VPSs for my web apps.
So, thanks everyone for your inputs. I’m always impressed by the knowledge and helpfulness of people on this forum.
Just wondering… I have now been doing some tests with a x86-64 bit xojo webapp (compiled in 2016r4.1). Simple app and may not have any relevance in your case. The app is a simple page with a Timer (firing every 2 secs) putting SQLDateTime in a Label and a database connection to a MySQL database doing a COUNT(*) every 5 seconds and putting the count in a ListBox.
The count is done from a Module but the timer is from a session.
I have compiled the app in StandAlone mode on port 8080 and using top to monitor CPU usage when running on a t2.small instance in AWS. I am daemonizing the webapp. CPU usage is 0.7-0.9% even with multiple (10 browser tabs) open.
Is your app doing something that could cause amazon to tax your EC2 instance more severely? File-handling for example?
In all cases, the apps wait for a user connection then load up a form and wait for user input. No timers are running. I did run a test project that simply contained a blank window, and that too had high CPU load. At the moment the only big difference between out setups is that you are using a later version of Xojo. Still, I wondor how it can be that on the DigitalOcen’s most basic server the apps run fine (<3% CPU) but on AWS they spike to 20%. I’ll keep digging and report back if I find anything. Thanks for your help.