Web app very slow

We have a complex web app that runs “OK” in AWS. Not blazingly fast, but not too shabby. But on the client’s internal network, it is slowing to a crawl and we are getting timeout errors. Their network is very fast (ping time from our web server to the endpoints it’s talking to is 1-2 msec) so I don’t think that’s the problem. Similar ping times from client desktops to the server the web app runs on.

Example: we fill a listbox from a JSON endpoint; payload from the JSON server is about 2-3 KB. Sometimes that listbox (which defaults to only 10 items) doesn’t even fill. We also put an image into an imagewell with a URL. When we change that URL, sometimes the imagewell doesn’t change.

Web App is standalone, port 8080, running on Red Hat Enterprise Linux. Currently built with 2017r1, but I’ve also tried 2017r2.

Any suggestions would be appreciated, as the client is getting angry.

Run it in chrome, record the session see what is timing out or taking a long time, you might have a name resolution issue after moving it internal.

I would concentrate on the differences between the servers. If it runs ok on AWS but not on the client server that’s where I would look.

32-bit or 64-bit? Number of processors? RAM? Those sorts of things.

@JulianS: thanks for the Chrome tip. Doubt that there’s a name issue because they’re using IP addresses. But I will try and decipher the Chrome network dev panel.

@Bob Keeney: I’ve never been able to run a web app in Linux on 32 bit; could never get the missing libraries resolved. So both the AWS and customer servers are 2 core, 64 bit 8 GB RAM. I don’t have any visibility into the servers, but their IT staff says that neither processors nor memory are stressed.

So I’m at a loss here.

I do know that this project takes almost a minute to load on my 2.3Ghz i7 MacBook, and crashes Xojo about twice a week.

Sounds like it could be an unexpected routing or DNS problem.

I know that if we access our local server from an external IP address, forcing the communications to go out our router and back in, it gets very slow sometimes. Accessing the server by its internal IP address is much faster…

@Steve Upton: thanks, but everything is internal to the client. They’re on the same internal network. Ping time from client desktop to server is 1 msec.

On what network server is it hosted? What’s the os version?
Disk/RAM processor, does it run alot of processes in the background?

  • Try clearing the browser cache

@derk: Red Hat Enterprise Linux 7.x, this is the only non-system process on the machine. It’s in a VM, but I believe the disk is 60GB SSD and the RAM is 8 GB.

can you post the response of the command:

ldd /path/to/your/webapp

That’s the LDD command, it will tell on what libraries your app depends.

@Derk Jochems: Here you go:

linux-vdso.so.1 =>  (0x00007ffd32567000)
XojoConsoleFramework64.so => /opt/appoportal/Portal Libs/XojoConsoleFramework64.so (0x00007f097c64c000)
libc.so.6 => /lib64/libc.so.6 (0x00007f097c285000)
libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007f097c035000)
libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f097bcfe000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f097bae1000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f097b8dd000)
libm.so.6 => /lib64/libm.so.6 (0x00007f097b5db000)
libc++.so.1 => /opt/appoportal/Portal Libs/libc++.so.1 (0x00007f097b229000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f097b013000)
/lib64/ld-linux-x86-64.so.2 (0x00007f097eb5b000)
libffi.so.6 => /lib64/libffi.so.6 (0x00007f097ae0b000)
librt.so.1 => /lib64/librt.so.1 (0x00007f097ac02000)

I still suspect routing, perhaps on the server itself…

Are you accessing by raw IP address or a host name?

What happens if you run it on a “normal” machine on the network and access it from the the same client machine?

Are you able to access it from a browser on the host machine?

[quote=348347:@Charles Weger]@Derk Jochems: Here you go:

linux-vdso.so.1 =>  (0x00007ffd32567000)
XojoConsoleFramework64.so => /opt/appoportal/Portal Libs/XojoConsoleFramework64.so (0x00007f097c64c000)
libc.so.6 => /lib64/libc.so.6 (0x00007f097c285000)
libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007f097c035000)
libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f097bcfe000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f097bae1000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f097b8dd000)
libm.so.6 => /lib64/libm.so.6 (0x00007f097b5db000)
libc++.so.1 => /opt/appoportal/Portal Libs/libc++.so.1 (0x00007f097b229000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f097b013000)
/lib64/ld-linux-x86-64.so.2 (0x00007f097eb5b000)
libffi.so.6 => /lib64/libffi.so.6 (0x00007f097ae0b000)
librt.so.1 => /lib64/librt.so.1 (0x00007f097ac02000)[/quote]

Thanks, all versions seem to be ok.

Perhaps it could be something else. Not sure though as “local” seems to be slower then “aws” it sounds like the machine setup is other (or perhaps network as suggested by @Steve Upton )

@Steve Upton: can access both by IP and hostname, no apparent difference.
This is in a huge datacenter; there are no “normal” machines.
And the host machine is a virtualized Linux server, so no browser is available.

Something that hasn’t been mentioned here is that you may have a “noisy neighbor”. That is virtualized hardware like this is often installed on physical hardware which is configured with 32, 64 or 96GB of RAM and 8 or 16 processor cores and then they cram as many VMs onto the hardware as they can. Depending on what you’re paying, you may only get getting a slice of a virtualized core. If one of the VMs is using a lot of CPU or I/O it can detrimentally affect the other VMs on that machine, effectively making them run really slow… and yes, looking just at your VM, everything looks fine, because yours isn’t doing anything. I’d have the service provider make sure that this isn’t happening.

Another thing you could do is try running “top” using a Shell object within your web app. it’d be essentially the same as doing it from the terminal except that it’s a snapshot of a specific time.

top -n 1 -b

That’s one sample, in batch mode (so you get to see everything, not just one screen’s worth).

@Greg O: interesting. I would hope that their IT people (this is a BIG company, so there are lots of them) would have tools to monitor this kind of “noisy neighbor” problem, but maybe they aren’t looking. I’ll ask.

Running top would only let me see what’s going on in my own VM, though, so that wouldn’t help me detect noisy neighbors. And I’m fairly certain we’re the only process on this VM. It doesn’t even have Apache or MySQL. Just the Xojo standalone web app.

It seems like the problem is happening on container controls that are parsing 4-5 KB of JSON to fill up list boxes (via .addrow) and various other controls. Is there maybe something I should be doing to prevent over-the-wire refreshes on a listbox while it’s being filled?

The only time my WebApp ran slowly was when it was running on a VM with only one virtual CPU. Moved to 2 or 4 virtual CPUs to fix it.

Yes, but it will let you confirm that the VM isn’t overloaded in RAM or CPU. [quote=348376:@Charles Weger]It seems like the problem is happening on container controls that are parsing 4-5 KB of JSON to fill up list boxes (via .addrow) and various other controls. Is there maybe something I should be doing to prevent over-the-wire refreshes on a listbox while it’s being filled?[/quote]
That could be, but then the issue is probably one of pure bandwidth. How many rows/columns of data are we talking about here?

How is the machine virtualized? (Full? (KVM, VMware, VirtualBOX) or Container). How many cores assigned (check /proc/cpuinfo and /proc/meminfo)

Test the network speed with something like iperf (from workstations to server and viceversa).

Also try traceroute in both ways to detect any routing problem as Steve said.

I have some Web apps under CentOS 7, virtualized (KVM, 8Gb, 4 core) and works fine even over Internet, they are CGI’s, and their load on the server is not very high. The physical server is not over loaded.