Last Friday, I took my first Catalyst application live. It was a particularly tense time – this was a live, public facing release of a completely new server stack. The short version: Catalyst rocks!
Previously, pretty much all of my serious stuff has been hand-rolled stuff in Apache/mod_perl, so starting a new project using nginx/FastCGI/Catalyst was a bit of a leap of faith. There was a lot of technology risk in this project, and as a public facing one failure was not an option.
The project itself was a redevelopment of part of an existing site which was suffering performance issues – this new setup needed to be better than the existing offering, not just match it. The redevelopment gave me the opportunity to clean out some of the bit-rot that had crept into the original codebase, so there was some advantage there.
Initial testing on a development server was fine – the site behaved itself, and performance was good. From a functionality point of view, we had a green light.
The code was installed at its new home (a pair of VPSs) and tested again to make sure it was okay (even though it was .deb packaged, the double-check couldn’t hurt). Initial testing on the new hosts were also good – no errors, and the debug logs gave pretty good timings, but it was the concurrency issue that really concerned me.
The site was running using the Catalyst FastCGI wrapper script, with CATALYST_DEBUG turned on and errors going to console rather than back to the web server. Each server was started with 5 children. nginx was set up to only check the local FastCGI server (note to self: go back and get it to do the fallback to the other server…)
Everything was set, so DNS was updated. Almost immediately, the requests started coming in, and a RR TTL of 10 minutes meant that it was going to get busy really quickly.
Watching the debug logs fly past, I was convinced there was a problem – there was no way I could be getting that kind of performance out of these servers. I went back and forwards across the site, looking to find the breakage. I kept refreshing my Twitter search, looking for people complaining (I’ve found it to be one of the fastest ways of getting error reports 🙂 ). I bugged everyone on my IM list to keep testing and let me know as soon as they found a failure. I had the people handing the support email addresses on red alert for complaints.
And several hours later…nothing. It was not just holding up (which was the best I had hoped for, given that there was no way of doing any semblance of advanced capacity planning on this entirely new stack) but was absolutely screaming along. We were now at ‘peak’ time, when I was sure that we were probably going to need to put some more beef behind the site, but the initial setup still seemed to be getting the job done.
The new software stack is awesome – there’s no other word for it. The combination of nginx/FastCGI/Catalyst has delivered the kind of results that I’d have to have had experienced to believe – even if I went back in time and told my earlier self about it, I would still have been sceptical.
Just checking the Google Analytics reports, the site has had over 1.5 million pageviews in the last week, settling on a figure of around 330,000 pageviews/day. This is a totally database driven site, running on a pair of most basic VPSs (256Mb of RAM each) for the website with a dedicated server running the MySQL database. The VPSs are reporting an average of 25% CPU usage, a little bit of free RAM (‘free -m’ reports 33MB) with a data transfer approaching 1.5Mbit/sec at peak times. These little things are really not sweating it.
I’ve already got another part of the system redeveloped in Catalyst, undergoing final testing on a staging area, and I’ve got really high hopes for it. I now feel totally confident about moving all of the remaining site components into the Catalyst framework, which has now become the new organisational standard for Perl website development.