High availability web applications, and how to optimise them

There comes a time in a serious web developer’s life, where he/she will have to deal with a high availability web application. For many, the sudden realisation that their code will no longer cope on the server it’s running on is normally met with, “what next?” I asked myself this question three years ago, when a group of websites I develop and manage outgrew a couple of dedicated servers they were hosted on. They were two separate servers with two separate providers, with round robin DNS as a means of load balancing. Pretty poor config.

You may have heard the terms “horizontal scaling” and “vertical scaling”. Vertical scaling is the practice of adding more resources to an existing server – more RAM for example. The reasons for this should seem obvious. Horizontal scaling means adding more servers rather than resource as such; the idea being that requests and work can be shared out among multiple servers, reducing response times (assuming a well thought out structure and a good load balancing solution is in place).

Currently, the group of websites I mentioned earlier runs on three load-balanced web servers – none of them host any databases – instead, this is done on a completely dedicated database server, and it works really well. Of course, there’s more to just setting up a bunch of servers and a load balancer, and hoping they work. For example, how are you going to control file updates within the server cluster? I use rsync on a scheduled script basis and only upload to one server but it’s worth making sure you only rsync relevant directories. For example, should you set it to copy log files? In my honest opinion, the answer here should be no – you would get mismatched data and it would make your life very difficult if you needed to diagnose issues with a specific server in your cluster.

It’s not just about servers.
Okay, so hardware is one hurdle. Of course, horizontal scaling means your code has to be scale-aware as well. In a PHP application, by default if you use sessions ($_SESSION), session data is stored in flat text files in a temporary directory on the server. Unless you’re rsyncing this directory (don’t, it will get messy), you will need to look at a session handling method that all of your servers can use. Two solutions here spring to mind – set up the temporary directory to be a shared directory available to all servers. The other (my favourite, and the one I have deployed) is to use a database session handler. Be aware though that the session database tables used in a HA application will be heavily used, and so will need appropriate configuration and indexing to ensure it doesn’t become a bottleneck.

Once you’ve tackled scaling of the code, you should then start looking at optimising your code and databases. Could your database tables benefit from more indexes? Yes indexing increases disk space, but this is a relatively cheap commodity compared to an extra second or two for your application to load a feature, because you didn’t index your database properly. What about database queries – are you pulling out more data than you need? Are you running more queries than you have to? One thing I learned last year was Foreign Keys in MySQL. Previously, if I was deleting records in a relational database I would be running a query to delete data for each table. Foreign Keys allow you to set up relationships and cascade those deletes, so one delete query to the parent table will crank up the database server and automatically process the other deletes for you.

Code optimisation.
Next comes image optimisation, minification of CSS and Javascript, and optimisation of your server side code. Consider:

$x = array(1, 2, 3, 4, 5);
for ($i=0; $i<@count($x); $i++) {
 echo $x[$i]."\n";
}

This is perfectly valid code, and will produce the following output:

1
2
3
4
5

However, if you look closely at it there are two flaws. Firstly, it is using inline error suppression (the @ sign before a function suppresses any errors PHP might generate for said function) – this slows down your script a little which would be fine in a little used application, but many of them in one script will cause slowdown. The other one is the use of the count() function in the for() line. This function is counting the number of elements in the array $x every time the for() loop iterates. This is a waste of server resource. A better way to write this would be:

$x = array(1, 2, 3, 4, 5);
$xCount = count($x);

for ($i=0; $i<$xCount; $i++) {
 echo $x[$i]."\n";
}

Never stop optimising.
Tuning your application is like a car enthusiast tuning a car. There’s always some tweak you could make that might get a little extra performance. For example, today I calculated that the bandwidth that I could save on a HA application I built, simply by using a Google CDN hosted version of jQuery, was close to 20 GB per month! Naturally, I have now switched it and am awaiting bandwidth reports to see how it has affected it. I’ve also enabled more aggressive image caching across websites by using .htaccess modifications.

One of the next forms of optimisation I will be making is using gzip compression to reduce the bandwidth usage further – this compresses data on the server before sending it out so impacts the CPU, but reduces bandwidth consumption. It’s a trade off, but when your server cluster is pushing upwards of 2TB a month on just three servers, it may be a compromise worth making!

Monitoring changes and progress.
How do you know whether your changes are working? Keep an eye on CPU utilisation and memory usage on the server, hard drive wait times and review bandwidth graphs. Use free services such as Google Analytics, and download the PageSpeed Chrome so you can check whether your page load times have improved.

It’s a lot of work and something that should be done on an ongoing basis. Your ultimate aim is to keep those applications online at all times. Finally, you should change one thing then review – never make multiple changes at the same time, as you don’t know for certain what provided the biggest benefit.

Leave a Reply

Your email address will not be published. Required fields are marked *