Engineering Everquest

IEEE Spectrum (Online) presents us with a  reminder of the engineering feat that is Everquest behind the scenes.  A reminder of the very long trunk and tail  that is the MMO unseen.

Comments

[ren]

1.

Mmmm hosting – drool

David Kushner>
"Internet service providers were in the habit of oversupplying bandwidth," recalls Michael Bowen, executive director of technical operations for Sony Online Entertainment. "They didn't believe if you buy a pipeline this size, you fill it. But we filled it so much that other people in San Diego lost Internet access because pipes were so full. They scrambled to increase capacity because of us." Sony's ISP—the company declines to name it —raced to boost the bandwidth, ripping up Interstate 5 in the process. [my emphasis]

Pfft

---8<----

traceroute: Warning: Multiple interfaces found; using 212.74.64.46 @ hme0
traceroute to everquest.com (64.37.156.16), 30 hops max, 40 byte packets
1 fe5-0-svc2.LON.router.COLT.NET (212.74.64.33) 0.680 ms 0.831 ms 0.435 ms
2 fe6-2-cr1.LON.router.COLT.NET (212.74.64.214) 0.433 ms 0.789 ms 0.869 ms
3 pos1-1-cr1.NYC.router.COLT.NET (212.74.74.1) 69.250 ms 68.606 ms 68.522 ms
4 ge-6-2.car2.NewYork1.Level3.net (63.214.53.253) 68.582 ms 68.180 ms 68.480 ms
5 ae-1-51.bbr1.NewYork1.Level3.net (4.68.97.1) 68.437 ms 68.215 ms 68.196 ms
6 as-0-0.mp2.SanDiego1.Level3.net (4.68.128.149) 134.090 ms so-3-0-0.mp1.SanDiego1.Level3.net (64.159.1.137) 135.016 ms 177.104 ms
7 so-10-0.hsa1.SanDiego1.Level3.net (4.68.113.38) 134.941 ms so-8-0.hsa1.SanDiego1.Level3.net (4.68.112.130) 134.350 ms so-10-0.hsa1.SanDiego1.Level3.net (4.68.113.38) 134.834 ms
8 vl861.sdtermswitch-1.sonyonline.net (63.215.251.130) 134.362 ms 135.165 ms 134.755 ms
9 vl80.sdtermsw-1.sonyonline.net (64.37.144.153) 135.049 ms 134.172 ms 135.689 ms
10 vl180.sdt-slb1.sonyonline.net (64.37.144.18) 134.619 ms

---8<----

So it looks like Level3 then...Oh, yeh, Sony decline to state, but Level3 are happy to press release: http://www.level3.com/press/3405.html.

So Sony appear to be are alleging that Level3 are selling contended bandwidth eh, hmmm as an ex-ISP dude that’s an interesting comment from a customer.

Sorry I appear to have geeked out for a moment there.

[Chris]

2.

Very good chance they have changed service providers since 1999.

[ren reynolds]

3.

Chris > Very good chance they have changed service providers since 1999.

phew, that's ok then.

[Terence Park]

4.

I recall during a server downtime back in 2000 that the EQ message stated UUNET as their ISP. Kinda scary if they were able to fill up the bandwidth provided by a major internet backbone provider back then. Anyone know if Wolrdcom had a colocation center in San Diego at that time?

[Will Leverett]

5.

That's a pretty fair explanation of what goes on deep in the dungeons. It's been my experience that most companies are also set up with similar operation centers.

One thing that the article does not mention is that NOCs also monitor the quality and levels of internal network operations. This is a major priority for NOCs, especially with development occurring across the globe. With work on different projects being done simultaneously at locations such as U.S., Korea, Japan, China, India, and Europe, the management of traffic is like a heart pumping blood to the limbs.

Also, the reference of patches being 25-30MB is now a little dated, at least for some of your games with larger subscriber bases. Today's major content patches in some of these games are now reaching several hundred megabytes (Chronicle 3 for Lineage II weighed in about 800MB).

[Factory]

6.

"Custom-built software then automatically loads all the game software and the programs the new machine uses to communicate with other servers,"
Aparantly a shell script is now to be called 'Custom built software'.

[Dr, Cat]

7.

I think it's interesting that they have 20-30 computers for every 10,000 or so players, giving a ratio of one computer per 400 subscribers, or one computer per 100 currently logged on users. Certainly that's readily affordable and profitable with the subscription rates they charge. But's its worlds different from what we have in Furcadia, with still only ONE server computer handling peak usage levels around 3800 users logged on, and a total user population of around 50,000. Comes from a different game design philosophy, efficiency being a central pillar of my work. Of course with a smaller development team, it takes more time to put in enough features to really tax the machines, too. :)

We are probably going to double the size of our game server farm to two machines within the next year, I suspect.