One Careful Owner

Unlike most of the eBay auctions we talk about, this one is for some very real, very big iron. Following the cancellation of Wish, the hardware that was going to support 10,000 players in a single world is now up for auction. So, if you want to, for example, build a virtual world which supports 10,000 players, this could be just what you need.

Comments

[Mike Rozak]

1.

88 nodes x 2 CPUs x 2.8 ghz = 492 Ghz of CPU-power.

493 Ghz / 10000 players = 49 mHz per player allocated.

49 mHz/player / (4 fully-active NPCs per player + 1 player character) = 10 mHz per character

Not to mention losses in efficiency having 166 processors talk to one another.

This is more CPU than I guestimated contemporary MMORPGs were using, but obviously Wish is/was doing more AI. Or are my calculations wrong?

[Jessica Mulligan]

2.

This gives whole new meaning to "Hey, if you quit, can I have your stuff?"

[Dave Rickey]

3.

Your calculations aren't wrong. However, in addition to the inefficiencies of different boxes talking to each other, ICE (the distributed computing platform Wish was running on) is built almost entirely in Java.

And it's a lot more than most games are using. Calculated your way, original EQ had somewhere around 20 MHz per player (and a player/NPC ratio of about 40 to 1, so each NPC got about 500 KHz worth of processor). DAoC gave each NPC about the same amount, but Moore's law provided much faster CPU's.

So figure that not allowing for ineffiencies, you could run about 1 million NPC's on that much hardware, "always on" (assuming they were no brighter than the current NPC's).

--Dave

[MrFrans]

4.

Maybe Blizzard can get some cheap expantion of there server park. :p

[Edward Castronova]

5.

this is the most fascinating set of calcs i've seen in awhile. soemthing totally new to me, i had no idea that tech people reasoned through cpu load this way.

please expand and help us out a little...

quick question would be: what's the substantive difference in what an npc does when it has 10 vs 20 vs 2 vs 200 MHz to work with. dave, what do you mean when you say an MPC is "on"? or, how many cycles do current NPCs need to seem brighter than they are now?

[Young Freud]

6.

While I could barely guess what the lower Hz cycles would provide, but 200 MHz/player sounds almost like the system requirements for a modestly detailed single-player computer game i.e. something like Fallout, Jagged Alliance, or original Half-Life (which wouldn't be too shabby for an AI).

[Bruce Woodcock]

7.

The only thing that struck me is that they actually took the time to have their corporate logo painted in the front of each card. While I'd expect such things from big companies like SOE and EA who probably like to show off their server farms, a startup really should be spending its time and money on more important things. It's not a big deal in and of itself, but it does make you wonder what their other priorities were.

Bruce

[Mike Rozak]

8.

Edward Castranova wrote - quick question would be: what's the substantive difference in what an npc does when it has 10 vs 20 vs 2 vs 200 MHz to work with. dave, what do you mean when you say an MPC is "on"? or, how many cycles do current NPCs need to seem brighter than they are now?

I'll explain what I know as a non-insider:

NPCs (at least the way I'd design them) would have about three states: suspended, slightly active, and active. NPCs that are very far (> 100M) from any PC would just be stopped and doing nothing. NPCs between 20m and 100m would be moving about, but they wouldn't be fighting (unless they had ranged attacks and new about the PC), and might take shortcuts with pathfinding. Completely active PCs would do all the necessary calculations to appear real.

The more memory and CPU you give to a NPC, the more intelligent it can become. First comes better path finding. Then planning and organization among the NPCs. Then natural language processing. Then actual NPC learning (which Black & White does).

Furthermore, the more memory and CPU you devote to the system in general, the more realistic it can be, with weather effects, newtonian physics, better NPCs, more trees (since trees make pathfinding more difficult), more data logging/analysis, better data compression of client-server communications, etc.

Using more CPU also lets you put more code into a scripting language, which is ultimately more flexible and (I'd claim) easier to track down bugs. Fewer bugs results in lower production costs.

It appears that MMORPGs are running 20-40 mHz per player... A typical text MUD uses 1 Mhz per player, and (for the most part) only has monsters active when they're in the same or neighboring rooms to players.

Young Freud wrote - While I could barely guess what the lower Hz cycles would provide, but 200 MHz/player sounds almost like the system requirements for a modestly detailed single-player computer game i.e. something like Fallout, Jagged Alliance, or original Half-Life (which wouldn't be too shabby for an AI).

Yes, but of that 200 mHz, most is devoted to graphics and whatnot. The part that would be offloaded onto a MMORPG server (if it were offloaded) is about 20-40 mHz.

[exbetatester]

9.

Their priorities wrere in hypeing their game. Numbers was what they wanted to show en when numbers were just bot useing miners who were suppose to be beta testing. Example i needed some one to help me to test a bug i found so asked first person i seen his response "sorry i got this running on my little comp letting my mining macro run while i play EQ" I reported this to the devs their response "so?" Very little beta testing was ever done. And not much stress testing ither.

If they had gotten rid of the free loading macroing players and gotten only real beta testers they would have done alot better.

[Dave Rickey]

10.

Ted>>quick question would be: what's the substantive difference in what an npc does when it has 10 vs 20 vs 2 vs 200 MHz to work with. dave, what do you mean when you say an MPC is "on"? or, how many cycles do current NPCs need to seem brighter than they are now?

Mike basically covered it, in some games you have multiple levels of activation, at a minimum NPC's a long way away from the players are generally "off", or suspended, doing nothing since there's no-one there to see it. This means that some forms of emergent AI/procedural content generation aren't feasible, since they won't happen when nobody is there.

More CPU cycles can mean either smarter AI or more of them active at a time. "Smarter" means more adaptive (learning to some degree from the players), more complicated state machines (more ways of reacting to what the players do), fuzzy state machines (displaying "intent" and something that can look like emotional states), and so on.

The current architectures and CPU budgets are the absolute bare minimum, most of those CPU cycles are burned on the "aggro loop", constantly checking to see what is near the NPC and how it wants to react to it (with generally only three choices: Run away, do nothing, or attack).

More CPU cycles and more ram means more complicated behaviours (ambushing, evasion, tactical maneuvering, attack selection to match weaknesses), more "intelligent" cooperation between NPC's (base building, cooperative deployment, strategic maneuvering), and so on. More agents means a whole range of emergent behaviours currently not in use become feasible.

Yes, I had some plans for all that raw horsepower.... There are a lot of things the current generation of games does not even attempt to do, because the architectures and hardware cannot support it. This made sense in 97, or even 99, when affordable hardware was barely adeqate to provide the minimum AI, but Moore's Law marches on (Wish's server wasn't much more expensive than UO or AC1 were originally spending).

--Dave

[magicback]

11.

Question:

Current discussion is on MHZ per NPC, what is the typical MHZ allocated for group AI (e.g. running multiple NPCs as a unit with tactical performance that would challenge a similarly outfitted PC unit)?

[Dave Rickey]

12.

0. None, nit, nada. All NPC's in all current games are basically on their own, CoH represents the high water mark, with healer AI that will stand back and heal allies (and, for that matter, enemies, as long as they are NPC's), and use Teleport to avoid being counterattacked.

Some specially scripted encounters will call in allies from farther than normal range, but there's no real group awareness involved (simply a boss mob that has been set to call in reinforcements from certain spawns once it takes a certain amount of damage).

--Dave

[Mike Rozak]

13.

Magicback wrote - Current discussion is on MHZ per NPC, what is the typical MHZ allocated for group AI (e.g. running multiple NPCs as a unit with tactical performance that would challenge a similarly outfitted PC unit)?

I think I read that Dungeon Siege 2 was going to have some tactical AI, although I could easily be wrong. (It's not a MMORPG)

Tactical AI doesn't take that much CPU because a) you only need one AI per 10+ units, and b) the AI only makes decisions at 10+ second intervals, so it's not run as often.

MMORPGs might not want tactical AI (or any real AI for that matter) because it suddenly makes the enemy a lot more difficult to fight. If a party attacks a dungeon, the alarm will be rung, and 100's of the dungeon's inhabitants will rush out and slaughter the party. For that matter, even a handful of enemies can put up a pretty good fight with a half-decent AI.

You might also want to create a "director" AI whose job it is to see what players do and then react. This would involve creating/destroying quests, creating armies of orcs to attack a village, etc. Much of the stuff that Wish was going to do with human GMs could eventually be replaced by a director AI, or at least augmented by it. For some reason, thoughts of 2001's HAL come to mind: "Alert! Alert! The Sword of Denethon quest will fail in 47 hours. You should visit it and replace the boss monster as soon as possible..." Sometime later all the monsters are stumbling around the world singing, "Daisy... Daisy... A bicycle built for two..." in a low-pitched voice.

[Abalieno]

14.

> All NPC's in all current games are basically on their own, CoH represents the high water mark, with healer AI that will stand back and heal allies (and, for that matter, enemies, as long as they are NPC's), and use Teleport to avoid being counterattacked.

Both DAoC and WoW do this (DAoC in some rare occasion but at least since Shrouded Isles).

This is also about something I wrote in different occasions, the AI isn't needed to do a good mmorpg, nor it can be helpful, from my point of view. Instead you have a nearly infinite stack of possibilities by tinkering with scripting.

I'm not waiting to see complex AI algorythms or mini-sandboxes to entertain the academics. Scripting alone is a powerful tool that may completely change the aspect of the genre. Imho.

[magicback]

15.

Dave> 0. None, nit, nada. All NPC's in all current games are basically on their own, CoH represents the high water mark, with healer AI that will stand back and heal allies (and, for that matter, enemies, as long as they are NPC's), and use Teleport to avoid being counterattacked.

That's a shame as I would have thought that there is some level of power allocated to this. How about as an utility function to debug NPC scripts such as teleporting a stuck NPC back to the spawn group?

Mike> MMORPGs might not want tactical AI (or any real AI for that matter) because it suddenly makes the enemy a lot more difficult to fight.

I think allocating CPU power to group AI adds additional possibilities to the game and allow the genre to move beyond current standard strategies and terms (e.g. aggro, pull, spawn, etc).

HAL didn't come to my mind, but more football playbooks. Which leads to scripting out tactical strategies instead of CPU-eating adaptive AI.

[luca girardo]

16.

Does anyone know if any commercial or open source library-product exists for AI simulation in mmogs (at least as stand-alone product) that could be integrated in a flexible architecture?

Another question I was wondering about: we can define a rate of cpu cycles per player and at the same time if e include the whole mmog server side infrastructure (game simulation, data storage, NPC simulation, etc) to define a MHz rate per player, where can we place the current limit today based on the operative costs and profitability?

[Jim Purbrick]

17.

There's also some ex-Wish space available at www.napoftheamericas.net.

Thanks again to James Vallord-Costa for the information.

[Andres Ferraro]

18.

The only thing that struck me is that they actually took the time to have their corporate logo painted in the front of each card.

Sometimes those are nice touches the vendor will do 'free', sometimes you do small things to help along the sense of belonging to a team, or having a mission.

[Damion "Ubiq" Schubert]

19.

AI in most standard MMOs can be divided up into three parts:

1) Pathfinding. Unmentioned in this thread so far, pathfinding takes the lion's share of all AI decision making cycles.

2) Targetting and using their powers effectively (a.k.a. decision making). This is deciding who to target, and how to use the powers they are granted effectively. Most game's NPCs will use powers like Heals and Stuns effectively, and if it's hard for us as designers to tell an NPC how to use a power effectively, chances are we won't give him that power).

3) Aggroing and Bringing a Friend. When to attack someone, when to call for help, and how many people to bring. This is the part that's primarily responsible for the 'pull and kill' model that's fairly effective and popular, but everyone here seems to hate.

The trick with AI is that it's easy to make AI that's more realistic and/or effective than what you have now. It's also easy to make a Quake bot that always hits - the hard part is dumbing it down so humans can actually defeat the bot. The same problem is evident for us MMO developers.

For example, anyone who has ever played any MMO knows that, most of the time, you have your entire party focus their firepower on one enemy, preferably the one that has the fewest hit points and does the greatest damage, or the one that can heal the others. Reducing the number of damage dealers as quickly as possible is the most assured way of winning. It's actually fairly trivial for us to code our monsters to do the same thing. Unfortunately, monsters that use teamwork too well actually present a challenge that's usually too hard for the users to overcome. It also creates an experience that's a world of suck for your mages. By the time you dumb that model down to something you can fight against, you're usually left with 'each guy is fairly autonomous, but uses his own powers appropriately'.

[Dave Rickey]

20.

Okay, first and foremost, let's agree that the purpose of AI is to act as a foil for the players. To fight like the minions opposing any hero: Poorly, and with little chance of success. I don't want the AI to win, I want it to be more *fun*, which is a much harder and more subtle problem than just making it harder to beat.

Which is trivial, as any "Epic" encounter from EQ or Camelot easily demonstrates. Pump up the hit points and damage, throw in a couple of "You're Dead" special attacks, some fancy light shows and a little messing with the player's control of the situation (spell immunities, knock-backs, status effects on their character), and presto: Instant harder opponent, no brains neccessary. Slightly scripted finite state machines can do it easily, and with a comparatively tiny amount of processor.

Yes, most of the AI CPU cycles are burned on pathfinding. Which makes it a shame that it counts as "AI", because if you take it out of the equation it points out how tiny the influence of AI in the game is. Pathfinding is a neccessary function, and without decent pathfinding the "game" attributes of combat are trivially manipulated. But it's not *useful*, in a design sense.

So why are we focused purely on what AI does in combat? Why does AI have no place in how we build our worlds (beyond juggling proximity and aggro ranges when we're placing NPC's)? Why does any discussion of AI inevitably turn into a discussion of how "tougher" AI is both easy and counterproductive?

Answer: Because it turned out to be easier, more predictable, to ignore it and solve the problem by hand, through the investment of thousands of "world builder" hours. Making AI that populates a world and makes it interesting through its interactions is conceptually hard and managerially unpredictable. Doing it by hand is labour intensive, but predictable.

Only one little problem: Hand-building doesn't scale well. Everquest originally had about 6 people dedicated to populating the world and making quests. Camelot had about 14. EQ2 and WoW had 40+ (and they worked at it for longer as well, nearly twice as long). Reportedly, the next batch will have more than that (on both number and time). This is why our budgets are inflating so fast (from less than 5 million for both EQ and Camelot, to around 15 for SWG, to reportedly more than 30 for EQ2 and WoW).

It worked up to now. Sure, the budgets ballooned, the content generation process became more a matter of management than inspiration, the quest formulas became so overdone that they almost write their own endings, but most of our players are new so they don't know that.

Fine, what happens next? When the next jump in production values requires a budget of over $100M? When the market isn't growing by 60% a year? When the players are *all* as experienced and jaded as the denizens of F13 and CorpNews? When "Content" that is nothing more than 2-dimensional scene dressing just isn't enough any more, no matter how well done?

Because it's not as far off as some would like to think. 3 to 7 years, depending on your assumptions. A couple more development cycles at most, and then hand-building hits the wall, *hard*, and has nowhere to go. AI is going to be critical to getting us out of that trap, but not AI as we've come to think of it.

--Dave

[Mike Rozak]

21.

Dave Rickey - Only one little problem: Hand-building doesn't scale well. Everquest originally had about 6 people dedicated to populating the world and making quests. Camelot had about 14. EQ2 and WoW had 40+ (and they worked at it for longer as well, nearly twice as long). Reportedly, the next batch will have more than that (on both number and time). This is why our budgets are inflating so fast (from less than 5 million for both EQ and Camelot, to around 15 for SWG, to reportedly more than 30 for EQ2 and WoW).

While I'm all for AI, some counterarguments:

1) I wonder if the mass-market wants intelligent AI, as Damion Schubert and I posted. Mass-market TV/movies certainly don't want intelligent villains, or even intelligent protagonists in Sitcoms.

2) If I were a large company, I would try to out-content my competitors because content costs lots of money, and (as a large company) I would have lots of money. Hollywood produces huge special-effects blockbusters that Europe does not because Hollywood has the money (and skill) to produce these. It's all about using your strengths, in this case capitalization.

3) I suspect that throwing 40 content developers at a problem is easier than throwing 40 AI programmers at the problem. AI has a tendency to interact in unpredictable ways (such as Oblivion's guards killing deer because they're hungry, getting in trouble with other guards for doing illegal stuff, killing one another, and letting bandits take over the town). It's hard enough for 1 developer to manage AI unpredictability. 40 developers is much more than 40x harder to manage, and will probably produce an incredibly complex AI system that goes crazy. While content developers will also step on each others' toes and create unintendend consequences, I don't think they'll be as unmanagable as an army of AI developers.

A counter to my counterargument: Theoretically you wouldn't need as many AI programmers, perhaps 10-20, to produce the same "amount of entertainment" as 40 content developers. Hearding 10 AI developers is doable. But, if this is is true, large companies will be wary of promoting AI because they lose their advantage of capitalization. Small companies, on the other hand, would embrace AI.

[Mike Sellers]

22.

Making AI that populates a world and makes it interesting through its interactions is conceptually hard and managerially unpredictable.

Well said, Dave (the whole post). And, like developing 3D engines a few years ago, I suspect AI construction is an area we'll see go from being unknown to a black art to a high-end commodity quickly -- certainly within the 3-7 year cycle you mentioned. And that as Mike Rozak implies, this won't come from the large companies that can afford the brute-force approach of throwing more low-level designers at the problem.

I do find it telling that the scope of MMOG AI in discussions like this one has narrowed progressively to primarily pathfinding and combat tactics. This as much as anything else highlights to me the narrowing scope of MMOG gameplay.

We may or may not want "intelligent" villains, but how about NPCs that show at least minimal awareness of what's going on around them?

[Dave Rickey]

23.

The main problem to date as been the downside of emergent effects: One man's emergent gameplay is another man's debugging nightmare. What we have lacked is a design approach and control scheme that would let us do this effectively. It's the classic "verb problem" we were arguing about a week or two ago: The interactions of nouns (discretely placed NPC's) rises linearly, the interaction of verbs (behaviours built into the AI) rises exponentially (if not geometrically).

If we can find ways to create "well behaved" AI that preserve the strength of emergent systems (a lot more structure out than was put in), then the fundamental assumptions change. It's a feedback and control problem (in other words, *software* design, not just game design).

--Dave

[Mike Sellers]

24.

I would say that the minute you open your doors to thousands of human users, the number of un-debuggable emergent effects rises with a large exponent. We take in stride, more or less, dupe exploits, wildly creative forms of griefing, hyper-inflation, and other pathological emergent effects -- and yet we're concerned about AI? AI ain't the problem at that scale. IMO it's (part of) the solution. Control rods for the reactor is one way to look at it, in an abstract sense.

As for creating "well behaved" AI, I'm reminded of Hofstadter's illustration of Godel's theorem: in effect any AI system that is robust enough to be useful likely is not provably well-behaved; and any demonstrably well-behaved system is likely too weak or brittle (not sufficiently expressive) to be useful in a complex game space.

[Mike Rozak]

25.

Mike Sellers wrote - I do find it telling that the scope of MMOG AI in discussions like this one has narrowed progressively to primarily pathfinding and combat tactics. This as much as anything else highlights to me the narrowing scope of MMOG gameplay.

True. I'm working on NLP and conversations right now, but if I talked about that instead of combat, someone would ask why such AI was needed at all in a MMORPG. The idea of talking with a NPC is completely foreign to a MMORPG or CRPG. Combat is well known and people can imagine how more intelligent adversaries might improve the game. People have difficulty imagining a conversive NPC, because they think of all NPCs as being quest-dispensers, shopkeepers, or monsters.