The ‘boring’ secrets of EQ

This month’s Game Developer magazine has a fantastic piece by Rod Humble VP of Development at SOEs Studio 1. The feature covers the practical side of keeping a 24x7 operation like EQ going year after year. Or to put it another way: even if you have managed to create a great game this is all the boring dev / ops stuff that you have to get right if you are going to provide a viable service.

I think that the piece is print only, so for those of you that don’t have a subscription here are a few highlights:

Separate Server Operations from development. EQ has the two teams as peers with separate reporting lines and a well defined division of responsibility. Ops don’t code and coders don’t load stuff onto live servers.

Remember that 27x7 really means 24x7. Examples are that while it might seem like a good idea to down servers to patch a particular feature, its really not a good idea to stop everyone playing for something that might only impact a small number, so patch releases should be scheduled and schedules should be adhered to.

The game must move on. So there is a constant tension between getting things 100% fixed and the drive to add new content and features (the piece covers content mix but I want to stay away from game design in this summary – so go buy the zeen for that). EQ has completely changed the graphics engine twice – which Humble likens to changing the engine in a F1 car while it is still racing round the track.

Jilt the Jolt. Coding 24x7 to complete something may seem a good idea at the time but SOE believe that 8 hour days and rest are better for long term productivity. Also while coding till 4AM then staring again at mid day is probably something that everyone who has written more than an Excel macro has experienced – there is more long term value in having a team work as a team, so SOE insist that people start at 10AM at the latest.
Anyone interested in these issues should also check out the new IGDA white paper: Quality of Life in the Game Industry: Challenges and Best Practices.

Being big and old helps. Studio 1 has about 160 staff, 50 of which are on EQ, the ability to pull in coders and artist – especially ones that originally coded stuff is valuable. Size also de-risks some technical elements like having enough servers for launch when you don’t know how popular a game is going to be.

It’s not a licence to print money. Humble offers the statistics that for one game he developed back in ’97 the number ratio of legit to pirate copes was 1:7, but he warns that creating an online game where you have more control of pirate copes does not mean that you can simply up lift profit by a factor of 7. An MMO is a 24x7 service, players hate bad service and good service costs a lot of money. Content, GMs, hardware – it all stacks up.

International means also not in English. Which adds even more complexity to everything from coding NPC trigger phrases to translating manuals and customer service.

Backup your data. Yup a point so boring and so vital that it’s not made often enough.

This just touches the surface of feature, there is so much more good stuff in the piece – including a short section on getting Do’s and Don’ts for creating and MMO, that I recommend you grab a copy of Game Developer if you can.

Comments

[Brian Whitener]

1.

I liked the comment in the article on how they always had to have a programmer on "Call" like the lead doctors at a hospital. The programmers take turns for whose week it is (to prevent burnout). The week that the programmer is “on call” they have to keep their cell phone/beeper by them all night so if anything happens they can come in for an emergency (such as IGE figuring out the latest dupe and making millions of plat :).

They also have this HUGE plasma screen that shows the status of all the EverQuest servers.

The article talked about how they determine when to patch or not. A lot of times a thousand or so players will complain about a minor bug that is specifically hindering them (such as a quest turn in), and they can not understand why SOE won’t just patch it. Although it is easy to patch, doing so would cause all 420,000 subscribers to download that patch, and could kick everyone off the servers (between 50,000-100,000 playing at a time). That would anger a lot more players then the 1,000 who the small bug is hindering.

[Brian Whitener]

2.

By the way, if you would like to sign up for Game Developer magazine(for free) go to:

http://submag.com/sub/gd?wp=free

For other free magazines check out:

http://www.slickdeals.org/forums/showthread.php?t=20652

[DivineShadow]

3.

Fortunately for this nascent industry there is *plenty* of experience in running 24x7 mission-critical applications out there in other industries; I suppose its only a matter of looking (and/or hiring) outside the 'box'.

[Richard Bartle]

4.

Brian Whitener>By the way, if you would like to sign up for Game Developer magazine(for free) go to:

This doesn't work for non-US subscribers (although they still get the very annoying "are you sure" style pop-up when they try to navigate off the page or close the window).

Richard

[Ren]

5.

Brian Whitener >By the way, if you would like to sign up for Game Developer magazine(for free)…

Even better, if you become of a member of the IGDA (Independent Game Developers Association) http://www.igda.org/ you get Game Developer Magazine for free where ever you are. IGDA is a not for profit organisation that does tireless work both for developers in general and in areas such as bringing academia industry together in addition they have a lot of SIGs (Special Interest Groups) such as IP and Women in Games that they are always looking help with – here endeth the plug.

[Ren]

6.

DivineShadow > Fortunately for this nascent industry there is *plenty* of experience in running 24x7 mission-critical applications out there in other industries

Certainly is. Why of course I don’t think that these are ‘boring’ facts is that I have a history in the telco industry, where ‘dial tone’ is all. And sure at the level that the piece was written there were no surprises for anyone that has worked in 24x7 ops – though I always like to see the dev part of world, er, educated in some of the hard facts about keeping a system running the day after you turn it on.

But I wonder if there are any specifics of running an MMO that differ significantly from running any other ecommerce Op – say, for example, an e-commerce based site like Amazon. Certainly in terms of the kind of user numbers one deals with in telco land both the user numbers and indeed cost / revenues of MMOs are pretty small. However, the criticality of customer retaliations are probably higher i.e. ‘players’ care more than ‘users’, hence the cost of upsetting them is higher, thus CS is probably proportionally higher (ad to this the complexity of the ‘application’) – which leads me to think that this is where the key difference lie rather than in the paperwork.

Of course I’ll know all this for sure pretty soon when I hear Richard, Ted, Julian and friend-of-the-show Jessica Mulligan speaking at the Community Work: Managing Multiplayer Culture seminar at ITUC Copenhagen Friday 21 May 2004 (there might still be places - see site for details). Opps, another ad, but it is a 3/4 TN line up .