This month’s Game Developer magazine has a fantastic piece by Rod Humble VP of Development at SOEs Studio 1. The feature covers the practical side of keeping a 24x7 operation like EQ going year after year. Or to put it another way: even if you have managed to create a great game this is all the boring dev / ops stuff that you have to get right if you are going to provide a viable service.
I think that the piece is print only, so for those of you that don’t have a subscription here are a few highlights:
Separate Server Operations from development. EQ has the two teams as peers with separate reporting lines and a well defined division of responsibility. Ops don’t code and coders don’t load stuff onto live servers.
Remember that 27x7 really means 24x7. Examples are that while it might seem like a good idea to down servers to patch a particular feature, its really not a good idea to stop everyone playing for something that might only impact a small number, so patch releases should be scheduled and schedules should be adhered to.
The game must move on. So there is a constant tension between getting things 100% fixed and the drive to add new content and features (the piece covers content mix but I want to stay away from game design in this summary – so go buy the zeen for that). EQ has completely changed the graphics engine twice – which Humble likens to changing the engine in a F1 car while it is still racing round the track.
Jilt the Jolt. Coding 24x7 to complete something may seem a good idea at the time but SOE believe that 8 hour days and rest are better for long term productivity. Also while coding till 4AM then staring again at mid day is probably something that everyone who has written more than an Excel macro has experienced – there is more long term value in having a team work as a team, so SOE insist that people start at 10AM at the latest.
Anyone interested in these issues should also check out the new IGDA white paper: Quality of Life in the Game Industry: Challenges and Best Practices.
Being big and old helps. Studio 1 has about 160 staff, 50 of which are on EQ, the ability to pull in coders and artist – especially ones that originally coded stuff is valuable. Size also de-risks some technical elements like having enough servers for launch when you don’t know how popular a game is going to be.
It’s not a licence to print money. Humble offers the statistics that for one game he developed back in ’97 the number ratio of legit to pirate copes was 1:7, but he warns that creating an online game where you have more control of pirate copes does not mean that you can simply up lift profit by a factor of 7. An MMO is a 24x7 service, players hate bad service and good service costs a lot of money. Content, GMs, hardware – it all stacks up.
International means also not in English. Which adds even more complexity to everything from coding NPC trigger phrases to translating manuals and customer service.
Backup your data. Yup a point so boring and so vital that it’s not made often enough.
This just touches the surface of feature, there is so much more good stuff in the piece – including a short section on getting Do’s and Don’ts for creating and MMO, that I recommend you grab a copy of Game Developer if you can.