Devs' blog: Our shiny new platform

By Nick Loman Published 4 May, 2009

Eurogamer moves house.

The last few months have seen broken nights and broken relationships for the back-room boys responsible for keeping Eurogamer's sites running. Last year, we realised that changes were needed to our infrastructure. To continue growing at our current rate, and to launch new services such as Eurogamer TV HD - coming soon! - we would need more of everything. More bandwidth, faster network connections, new servers and more physical space to house them in.

A feeling of impending doom came over me, as I realised how much work this would mean. We have endured site moves before - five of them over the past 10 years - moving hardware around the East End of London like Alan Sugar selling car aerials.

But we hadn't moved in over four years and we knew that this would by far the biggest and trickiest move yet. 10 major websites depend on this infrastructure, which attract over four million visitors a month. If the site goes down, or components get broken, it means working without stopping - sometimes right through the night - until they are back up. The wife wasn't going to be happy.

The boss had told us that any down-time would be quite unacceptable. When your plan involves keeping your wife and your boss happy, you know it's going to be hard.

We knew we would have to make a complete replica of the sites on our new infrastructure, before we were able to move any physical servers.

'Devs' blog: Our shiny new platform' Screenshot avocado

Mark finds nourishment where he can at DX Networks

The fun part? Getting our hands on a load of new kit. We ordered 16 new servers to go in our new dedicated rack space in Telehouse North, one of the original London datacentres. This place creaks with network cables, which run 100-deep along every surface. Geek heaven. It also has its own cafe - a total luxury - dramatically improving our days spent cabling up and plugging in.

Plus we signed a deal with up-and-coming network provider DX Networks to provide us with a diverse multi-homed network and multiple Gigabit internet connections.

Servers having arrived, we pondered on the important job of naming them. With most of the Eurogamer team being huge fans of The Wire, there was no contest - so we started installing our Linux distribution of choice, Gentoo on mcnulty, kima, bubs, lester, avon, bunk, stringer, prez, marlo, herc, carver, snoop, bodie, carcetti, weebey, rawls and of course, omar.

Naming rituals over, then began the slow process of installing and configuring the sites on the new hardware. Much effort was spent trying to automate the processes of installing servers. We devised a method to build a web-server replica (Eurogamer needs eight web-servers) that handled the Linux installation and software configuration in a single automated step.

Easter was nearing, and the WAGs made it clear that working over Easter would not be tolerated. So, we stepped up a gear, pulled several very late nights, and at 01:57 on Thursday, 8th April, we began redirecting site traffic to our new hosting.

Most of you didn't notice anything had changed.

A few observant forumites helped us debug the odd problem - thanks to you.

We broke a few things, and it took another two weeks for all our systems to come back to normal. We made up to our girlfriends and tried to get back to normal life. We stopped checking Eurogamer quite as obsessively, "just in case there's a problem".

Finally, just as things started to calm down, disaster struck. One of our new servers - carver (unreliable as his TV namesake) - decided to develop hard drive read errors, causing database corruption. So, fresh from the move, and arrogant that our server squad wouldn't let us down, we had committed the cardinal sin of not having set up all our disaster recovery procedures back in place, meaning we had to recover the MySQL database from binary logfiles. This meant several hours of downtime. Lesson learnt: you have to move your database replicas and your backup procedures at the same time as your critical services.

But this event I hope did not take away from a story which I'm sure you agree is full of heroics by some very talented and attractive backend developers. Most importantly, we are now well placed to ensure that Eurogamer can grow to meet the demands of the next few years.

Picture of Nick.

About Nick Loman

Read more articles from Nick by visiting the Nick Loman archive page.

You may also like...

Comments (31) Latest comment 2 years ago

Comments threads automatically close after 30 days, but please feel free to continue chatting on the forum!