My guildmates and I in WoW are an unstoppable force, able to mow through furlongs full of Furbolgs crying "Strength and Honor!" all the way. Well, that's not quite true...While we have every reason to believe that we can pwn Sunken Temple (aka The Temple of Atal'Hakkar), we have been thwarted time and time again.
Are we too low level? Nah. Do we have poor leadership? Certainly not (as long as I'm not leading). Are we not geared properly? Please -- this is a guild where members know the WoW Auctioneers on a first-name basis.
No, the problem is a Blizzard of technical obstacles.
Yes, we've heard about authentication and post-patch problems. Those have become the natural state of affairs. But this week we were foiled in our assault on ST for two different technical reasons. The first time, one of the six mini-bosses essential to the Shaman L50 quest, was just...missing. Not there. And of course he was the sixth one we went after.
But it was our second obstacle which got me thinking about WoW and its prospects. Last night, we were foiled because of the billing problems that Daniel Terdiman over at cnet covers in his blog here . Monthly credit card-billed customers with bill dates on April 20th and 21st -- like me -- were unable to log in because the automatic billing hadn't charged their cards. Blizzard's work-around (calling the account department, or re-entering your billing info via the account management page) were quickly inundated with requests and pushed out of commission themselves.
So while I futilely sought to access Blizzard's Account Management page, my guildies and I chatted about all of this over Teamspeak last night. Apart from blowing off steam about these continual problems (all the while imagining Blizzard's devs blissfully unaware as they take their Maseratis out for a spin), we wondered whether this is simply growing pains of the largest MMO ever, or whether there is reason to believe that the problems will be more sustained.
Could it be that WoW's demands, due to its sheer unprecedented size, not only on Blizzard's own game servers, but also on all the (I assume) third-party vendors/providers that handle things like authentication and, now, billing, are going to continue to push the limit for the forseeable future? (This recalls, by the way, Second Life's own recent challenges with scale and the increased threat of attacks, one of which knocked the whole world offline this past week.) Do we think that the presumable bump that comes with the expansion pack will exacerbate this problem?
Assuming Blizzard is internally, and properly, panicked about these problems, and are therefore doing everything they can to solve them, then is these problems' persistence a sign of a structural problem -- that WoW's scale and the size of its user-base make demands that the third-party support vendors have never had to cope with before? We didn't happen to have one of our serious tech-heads on TS with us for this conversation, so I'm raising it here. How much of a threat are the problems of scale for the virtual worlds that reach such size first?
I think it would be a nice gesture to Blizzard's current customers if they could stop selling copies of their game while they iron out all the kinks in their system.
There's an article on C-Net about wait queues and AT&T bandwidth issues. If Blizzard cannot handle the influx of new users, they should stop allowing new users until they can.
Posted by: yap | Apr 25, 2006 at 12:30
I assume the answer to your question is – yes. I get the feeling, and this is only a feeling as I have no inside track on Bliz, that MMO developers come from a games background and on the whole games do not have to deal with scale issues.
Where I’m coming from here is a long background in Telco land - and there 5.5 million customers is not a lot. Anyone that has created a service on a telco scale or a decent .COM like Amazon will tell you that when things get big you engineer a service differently as some principles just do not sale whereas other are just not work thinking about till you get to a certain size.
The problem is that online services are a symbiosis of technology and service elements and it’s often difficult to re-engineer these while both keeping all elements in sync and keeping services live, so I assume bliz are experiencing some horrible growing pains I can imagine the systems architecture plan on a white board with a phased replacement of just about every element and the service architect in a corner sobbing.
The answer – call in some telco-heads (I’m sure they have my number :>)
Posted by: ren reynolds | Apr 25, 2006 at 18:44
I concur with Ren. Six million users only sounds like an impressive number. However, a zillion Internet portals, e-commerce companies, telecommunications carriers, cable companies, etc. all run Internet infrastructure on a much, much larger scale. Tons of companies have billing infrastructure on a similar, if not larger, scale.
Now, scalability may require a skillset that the existing MMOG server engineers just don't have (and if the industry persists in pretending that you can just have developers of single-player games turn into competent server developers overnight, the skillset will continue to not exist in the MMOG community).
The problems at Blizzard likely exist on multiple levels, as well -- software architecture, software implementation, hardware architecture, storage architure, database architecture and implementation, network architecture, specific issues with the products and vendors they're using, etc. etc. etc. All of these need to come together harmoniously in order to deliver a stable service.
I have been watching Blizzard's job posting for an Oracle DBA with growing amusement over the months. It's evolved, bit by bit, from "You need to be a hardcore gamer", to, essentially, "It'd be nice if you liked games, but please, we're not crazy, once you have the job you need to understand a little bit about our game in order to troubleshoot our database."
Presumably, eventually that'll hold true across the board, and they'll get desperate enough to hire people who know what they're doing with scalable systems, first and foremost, and stop worrying about whether or not they adore games. (Not very many people start out in most companies with a deep and abiding passion for whatever it is that company does, but practically everyone successful ends up with vertical expertise and a deep professional interest in that field.) Then there'll be a lengthy period of time that'll be needed to "permanently" fix, rather than band-aid, the technology issues.
Posted by: Lydia Leong (Amberyl) | Apr 25, 2006 at 22:59
Um. Just, you know, saying... what makes you think code monkeys at Blizzard are driving Maseratis?
Vivendi corporate management in Paris may be giving thanks that they can afford the payments on their Alfa Romeos... but we are talking about a large, diversified entertainment (and water) conglomerate. Whose asses have just been bailed out by a piffling little studio in an industry they don't understand that they acquired with some other assets when Cendant discovered that CUC, which it had taken over, had been run by fraudsters for years and had to bail on the piece of that acquisition which made no sense to their corporate strategy.
Paul Sams can probably afford to drive a Maserati, but my take is that he's a fairly sensible guy, and probably drives a BMW at most. As for the rest of Blizzard--we're talking Honda Civic territory.
This isn't fucking Hollywood. This is the game industry. God help us, to be sure.
Posted by: Greg | Apr 25, 2006 at 23:06
Um, Greg, I was kidding about the Maseratis...
Thanks, Ren, and Amberyl, for the helpful comments. It's interesting to consider the possibility that a multi-million (billion) dollar product may be hamstrung by a cultural bias (thou shalt be a gamer) which they are slowly getting away from.
Posted by: Thomas Malaby | Apr 26, 2006 at 09:26
Concur with Ren and Amberyl - what little I know of the Blizzard architecture doesn't sound very encouraging.
And it's not a "cultural bias". It's the freaking goddamn arrogance of the industry. It's a pervasive mindset that game coders are gods gift to programming, and the thought that we can do a better job than anybody else. We end up paying the price time and again.
Posted by: Robert 'Groby' Blum | Apr 26, 2006 at 10:01
I consider myself a very good programmer, and I have even had some success in being the ultimate cat-herding authority for a nontrivial programming project, and I am humbled by the WoW team's success in making the game itself work as well as it does. Sometimes, admittedly, I do run across an individual WoW bug which makes me suspect that under all the maintenance fixes must lurk an unreasonable amount of weird unfixable architectural brokenness. (E.g.: not being able to register a guild unless the charter is in your main pack? That that bug could occur spontaneously whispers disturbing things about their design, and that it is evidently hard to fix that bug might give me nightmares if I thought about it too much...) However, overall I am very impressed with the work of the game team. If I had been given a comparable budget and schedule, and a magic wand to wave to solve the problem of associating myself with a first-rate game designer and a first-rate art director, then while I would choose to do various things differently, and I would likely even be able to do some things better, I very much doubt I could do as well overall.
However, once you get slightly outside the team developing WoW itself, it seems as though there must be some entrenched technical mediocrity in the Blizzard organization. In particular, the technology of scaling ordinary web servers has been fairly well-understood (and fairly cheap) for years now, but still Blizzard's ordinary web sites (WoW-related content, sure, but just stock web servers, not custom WoW servers) have often been flaky or downright unusable. The unusability is concentrated at times of peak load, yes, but it's entirely predictable peak load, like the hits on the server status page when something bad happens to blow away most of the servers. That seems embarrassingly bad, and it seems doubly bad that Blizzard doesn't seem to be embarrassed about it.
Posted by: Bill Newman | Apr 26, 2006 at 10:49
I thought Amberyl's observation was a good one.
Personally, I suspect that "You need to be a hardcore gamer" is actually code for "We will work you like a dog for hours guaranteed to destroy your marriage, not to mention pay you below the prevailing salary in this area for your professional skills, for the privilege of being able to call yourself a 'game developer.'"
With industry expansion (due to investor interest in VU/Blizzard's success), demand for game developer jobs seems likely to increase over the current demand. That increased need for competent generic techies (not every "game developer" has to be a game designer) will probably force development studios to offer more competitive salaries and fewer of the 20-hour-day marathon coding binges.
In other words, game development will become just another tech job. It'll be sort of sad to see the mystique evaporate, but so it goes.
I still harbor occasional fantasies of returning to the days when programmers were lab-coat-wearing acolytes of an arcane cult that sacrificed trees to the gleaming Big Blue Box. "You may leave your programming request with me, unbeliever. If the High Programmer deems you worthy of the cycles, we may code and run your job in, say, a month or two."
One day this whole "personal computer" fad will kick over, and programmers like myself will once again stride the earth as living gods....
Meanwhile, we and game developers will probably just have to endure becoming mere mortals, punching the 9-to-5 clock to pay for a mortgage and a Honda Civic like everyone else. Bit of a comedown for a living god, but every Golden Age turns eventually to base iron.
--Bart
Posted by: Bart Stewart | Apr 26, 2006 at 11:19
Groby wrote:
it's not a "cultural bias". It's the freaking goddamn arrogance of the industry. It's a pervasive mindset... (emphasis added)
You say tomato...;-)
Posted by: Thomas Malaby | Apr 26, 2006 at 11:53
Bill: That seems embarrassingly bad, and it seems doubly bad that Blizzard doesn't seem to be embarrassed about it.
Remember the post-lauch waiting lists that got press 2 months into the game? At the time, I was certain that this would be the brake on WoW's meteoric growth. How long could you expect your player base to endure a queue to get into the game? How much will your customer wait?
Obviously alot longer than I thought.
What amazed me then, and now, is Blizzard spins this so well into positive press. Issues that should be driving people away in swarms are instead seen as symptoms proof of its popularity- and if it's THAT popular, you should stick around.
- WoW's login queues: well, they're growing.
- WoW's instability: well, that's because of growing pains, everybody loves them, you should too.
- WoW's PvP servers have imbalanced factions: well, when you have to distribute the players across so many servers... which they have to do because they're so popular... you're bound to have some clustering...
- Substandard customer support? Growing pains...
When I talk to friends that play WoW, they mention 2 reasons for staying, despite their unhappiness:
1) It's popular.
2) There's no decent alternative.
When you try to dispute #2, They either cite the lower subscription numbers as proof of the game's inferiority or refer to the developer's reputation (SOE in particular).
Posted by: Chas | Apr 26, 2006 at 11:56
Speaking as a confirmed WoW addict, but playing on a European server, some of the problems being discussed here seem to be limited to the United States, and a little parochial. Payment problems? I haven't experience any, or heard of any over here. (That's not terribly surprising, as Blizzard will almost certainly be using different third parties to handle payments in different parts of the world.) Queues to log in to Azeroth in the first place? Not over here... Unbalanced servers? Maybe, but not by much. (And on my server, the balance is skewed more by the fact that the horde players really get stuck into their roles and enjoy making people's lives a misery.) Azeroth going offline for crashes or "emergency maintenance"? Maybe four times in twelve months of relatively intensive play. I've seen other services crash more often than that. The website going down when the game servers crash, snowed under by players trying to find out what's going on? Predictable, and annoying, but not the worst problem in the world.
So, if the question is "why are 6 million people world-wide putting up with this?", maybe it's the wrong question... There are lots of us out here getting better service than you seem to have, happily bashing furbolgs, clearing the temple, and working on MC.
Posted by: David Roy | Apr 26, 2006 at 12:46
One thing I'm always curious of is why they allowed the scale. Some of the finest restaurants in the world have a very limited seating. Some of the best consulting agencies will only take a certain number of clientel. Blizzard could have raked in the cash, had a very limited user base, perfected that world system and pushed it out on a schedule that wasn't breakneck. Instead all these services are trying to capture everyone and everything, busting at the sides, testing poorly, implementing poorly, CRM is bad, employee retention is bad. It never ever has to be this way, but when you're greedy it always seems to occur.
Posted by: C9 | Apr 26, 2006 at 16:51
Blizzard is trying to be the McDonalds of the online gaming world, not the Côte d'Or.
Posted by: Carnildo | Apr 26, 2006 at 20:20
I, for one, would settle for the In-N-Out Burger of the online gaming world. :-D
Posted by: Thomas Malaby | Apr 26, 2006 at 23:52
IT does appear that WOW US (Australia/NZ) has some quite severe issues, that have been ongoing (I've been playing since Jan 2005). I suspect that they made some infrastructure decisions way before launch that have made scaling up the US WOW harder than it should be, but not so hard as to make it worth while to justify a timely revamp. We hear now of new hardware being rolled out to some existing/new US servers, but this is a slow process and many realms have not been given timeframes as to when this will happen.
Posted by: Juan Incognito | Apr 27, 2006 at 00:36
I follow the hosting business, professionally, and it continually astounds me that very, very few game companies take advantage of the managed services capabilities of hosters -- they generally choose to do colo instead. Running stuff like webservers is certainly not rocket science, and hosters have that kind of stuff down to a science, even for infrastructure on a massive scale. It's certainly a cost-effective approach, as well, doubly so when you don't already have techies on your staff that are experienced in large-scale operations.
Blizzard's WoW website tends to go offline or respond sluggishly enough to be virtually unusable, whenever the game servers are experiencing an adverse event. (And while network problems sometimes affect both, it usually appears to be the servers themselves that are having issues.) That kind of collapse-under-load behavior is unacceptable in any kind of mission-critical website, and for the kind of functionality that Blizzard's site offers, it's not hard to fix. Blizzard's inability (or unwillingness to spend the money) to fix that kind of basic issue is extremely telling.
Broadly, good operations people don't come cheap, and you don't find them at the entry levels that game companies tend to like to hire at. Moreover, only certain of them will willingly walk into crisis situations, since those crisis situations almost invariably involve not getting sleep (and your partner not getting any sleep, either, since your pager will be doing a little dance on your nightstand every couple of hours).
Posted by: Lydia Leong (Amberyl) | Apr 27, 2006 at 04:06
With reference to Chas' post, I get similar reponses when I ask friends who play WoW. I too have problems in trying to point out that there are better (or at the very least, other) alternatives out there. Considering that WoW is an anomaly in the MMOG industry, as the 1st to break out into a truly mass-market. They've been breaking records with the sheer no. of subcribers since the game's launch (Perhaps their aggressive marketing campaign may have something to do with it and the fact that it is a relatively easy game to pick up and get into). It seems to me that the mindset which exists with other pop culture products has translated over to MMORPGs (or more specifically, WoW) where "most popular = the best". Which isn't surprising since there is a large portion of the WoW player population who could be classified as "non-gamers" (meaning people who don't play alot of games or just the one).
Developer reputation aside, surely there exists another scale on which a game's "inferiority/superiority" can be determined other then just the number of subcribers/players it has? But then again, that too has its own set of problems.
Posted by: Gabey Goh | Apr 27, 2006 at 04:56
The queues never bothered me. It seems like a good way to encourage people to play on different servers. Those with a strong interest in a particular server will wait it out. Those who aren't too attached to the server will move to a new one if the queue is annoying enough. Is there a better way to keep server populations spread out? I think it would be more effective with open character transfers.
There are some very strange bugs in WoW, but I don't think they are any worse than other MMOs. Other than DAoC, I can't think of any MMO that had fewer serious bugs at launch.
And I'll be the first to admit that my own programming skills are mediocre at best. I couldn't make the backend better than what Blizzard has done.
"Assuming Blizzard is internally, and properly, panicked about these problems..."
I had only a few complaints about WoW and this is one of them. I'm not convinced that Blizzard is serious about fixing bugs. There have been numerous class/skill bugs, some of them serious, that were found on the test server, had plenty of documented evidence, and went live anyway. I understand that sometimes you are going to want to fix a more serious bug, even if a patch introduces new ones, but doing this every single patch seems excessive.
And when a company representative says that something is a bug one patch, promises a fix, and then later says that it's "working as intended," it gives the impression (whether accurate or not) that the "bugs" are actually a cowardly nerfing method. That's more of a PR issue than a code issue, but it mattered more to me as a player. I can put up with bugs as long as I have confidence that they will be acknowledge and fixed. I lost confidence in Blizzard.
Summary: It's not the total quantity of bugs that bothers me, but the way they seem to be increasing every patch (and ignored or re-defined) instead of being fixed.
Posted by: AFFA | Apr 27, 2006 at 12:53
A few assorted comments.
Thomas wrote:
It's interesting to consider the possibility that a multi-million (billion) dollar product may be hamstrung by a cultural bias (thou shalt be a gamer) which they are slowly getting away from.
Interesting that you use the term "culture" here, because that's essentially what it is: trying to maintain the company culture. It's essentially like bringing in someone that can't speak English into a company that works in a predominantly English-speaking area. Sure, the person can do the work, but it can potentially result in problems in the work environment. If the DBA can't relate to the designers geeking out about the latest must-have game, he or she is going to be viewed with at least a certain amount of distrust. This can cause long-term rifts in your team that can do serious harm; the designers not consulting with the DBA when making designs can lead to crying down the road on both sides. This isn't to say that the company absolutely can't hire non-gamers, but there's a good reason why such a preference was stated in version 1 of the job description.
One of the big problems is that online games don't follow normal usage patterns of other online services. Why do the server status pages have trouble when the servers have trouble? Because the several thousand people unable to log on go to the website and hit "reload" on the web browser repeatedly to see if their server is up or not. Consider a non-gaming application that is similar: hitting refresh on eBay near the end of an auction. Anyone that has done this can tell you that eBay has had trouble with this, and you often miss last-second bids because the system barfs at the wrong time.
Or, consider that a web server that is unavailable one minute out of every hour is not usually a business-ruining problem; a game server that goes offline one minute out of every hour is going to cause mass screaming, usually at your CSRs. For a game sever, it's better to concentrate that downtime into 3 hours once a week. Anyway, let's at least accept that there are some challenges in each industry that aren't easy.
On the other hand, this isn't to say that any company gets a free pass because they've grown so much. People said the same thing about every game that grew a bit bigger than the previous games out there. Yet, people keep making the same mistakes over and over again. Nobody wants to learn from the previous generation; everyone thinks they're so much smarter than the idiots that ran those old, insignificant games in the past.
Or, you could be cynical and say that the WoW developers have learned from history: players may say they care about things like stability, customer service, and bug fixing, but has this really affected the bottom line? Many people, myself included, thought that many of these problems would hinder a game like WoW. Yet, we have been yet to be proven right. As long as people keep paying, why upset the current way of doing things?
Back when I worked on Meridian 59 at 3DO, I was told repeatedly to help design new content instead of focusing on fixing bugs like I wanted. I knew that we needed to focus on fixing bugs in order to prevent problems down the line, but the managers said that fixing bugs wasn't sexy enough to keep people interested. If you did manage to fix all the bug, people would then demand new content (which will likely introduce new bugs).
I think WoW has shown this strategy to be wiser than I thought it was at the time: You can frustrate your players, prevent them from playing at inconvenient times, string them along with empty promises, whatever. Just don't bore them with a lack of new content. If you look closely at the history of EQ, I think you'll see some parallels with complaints and what really made the company money.
Interesting thoughts,
Posted by: Brian 'Psychochild' Green | Apr 29, 2006 at 07:12