Datamining the Forums

When I'm thinking about a particular virtual world, studying it, playing in it, I find it useful to read both official forums and unofficial fan site forums. It's one source of information about what players are thinking, doing and wanting in a given virtual world, one of the visible tips of the larger below-surface iceberg of the sociality and culture of a given virtual world.

I don't find it that hard to use forums for gathering information, nor does it seem to me to be that difficult to recognize the evidentiary limits of the data available on them.

Why, therefore, is the question of what forums are good for such a perennial issue? Why, in particular, do many developers seem to exhibit varying degrees of hostility to forums as a source of information, and cast them instead as a tool for "community management"?

I'm thinking about this question again because of an outbreak of cross-blog conversation between some of the usual suspects in response to SWG developer Chris Cao's irritated response to the SWG forums.

It's conventional wisdom not just among developers but among savvy gamers to bash most forums as misanthropic noise, or to talk as Cao does about people who "play the boards" rather than "play the game". I'm not saying that's incorrect. It's true: a lot of what is said in a game forum is informationally-poor. A lot of the posters do seem to be the kind of people who have scary obsessions and a total lack of proportionality. A decent number of forum habitues spend more time in the forums than the virtual worlds. A lot of them are creeps, assholes, weirdos. A lot of them are cruel or stupid or unhelpful.

This goes as much for the sycophants who praise the game and the developers as it goes for the critics. In both cases, a lot of what is going on is attention-seeking behavior, and one of the elementary mistakes that some developers make (SWG has a long history of this) is praising and rewarding the sycophants. But that falls under the heading of community relations. Or, in the way I tend to see it, under the heading of sovereignty and political power. The sovereign who surrounds himself with fawning yes-men tends to end up going out naked in public to the jeers of the crowd.

Let's talk instead about how you use signal-rich but informationally-poor environments to your advantage. I'm a historian by training, and I think this is one of the basic methodological arts of history. We're data miners who know how to rapidly assess the probable worth of an archive, how to rapidly work through huge bodies of documentation to find what we need, how to read through, in and around sprawling mounds of information.

Reading a given forum for an half-hour or hour every day when I'm focusing on or thinking about a given virtual world, I feel pretty confident about my ability to find nformation that is useful to me and ought to be useful to a live management team. I'll add one qualifier: I'm much better at understanding and sifting through a forum if I'm actively playing in that virtual world. Let me give some examples of the kinds of posts that I think are useful and relatively easy to identify rapidly.

1. Posts that describe major schisms or fractions among the players invested in a virtual world, and the specific issues that are defining or shaping those schisms at the moment. This takes a while to become clear, and actual play helps to spell it out some. But you almost don't need to really read the messages in detail after a while: you can track the back-and-forth motion of these debates just from headers and volume of posting activity. So, for example, looking today at the World of Warcraft General forum, I can see that the long-running divide between casual and raiding players is focusing somewhat around planned changes to PvP. Looking at the SWG forums today, I can see that planned changes to crafting are an important focal issue.

This is information. It doesn't help the live management team decide which faction is correct in its claims. You can't really tell from forums just how many players are speaking for any given faction, and most forum posts aren't going to be articulate enough to actually persuade a developer to think one way or the other. You can't even be sure just how intensely a given faction feels about a specific issue. But it helps to understand where the points of tension are, and why they figure as they do in these kinds of conflicts.

2. Post that give specific and useful information about exploits, bugs, technical problems, major flaws in the game design or structure of play. I have no idea why live management teams from time to time protest that exploits, bugs  and technical issues are best uncovered by other means, save the concern that by posting an exploit, a player encourages its use. No, scratch that: I have a cynical thought about why a lot of developers and community managers take this position. It's because they think these discussions create bad publicity, and because in many cases players are identifying technical issues that the live management team knows about and has no intention of trying to fix. The alternative is worse: that whatever the other information sources about technical issues might be, they're even more informationally impoverished than the forums are. In its early history, SWG's forums contained detailed, informationally-rich information about technical and design problems that developers either ignored or actively claimed did not exist. When live management teams then protest that they can't learn anything useful from their own forums, in this context at least that leaves only two alternatives: that they don't want to be confronted with knowledge about technical issues in a public space (and don't want players to share knowledge between themselves in a public space) or that they're totally clueless. Either way, bad for Zathras.

3. Idiosyncratic posts that are informationally and analytically rich. When I'm scanning a very dense, highly populated index of sources, posts, or information of any kind, I have some heuristics that help me decide where to invest labor in reading or retrieving a given listing. Some of those are personal and non-transferrable, a result of experience or specialized knowledge. But a lot of them are built from basic skills that I think most people can learn. Titles tell you a lot: you watch for unusual phrasing, imaginative use of words, and yes, evidence of literacy. D00dspeak IS a warning signal: it says, don't bother if what you're looking for is a usefully distinctive response to the state of a virtual world, unless for some reason you have a particular interest in d00dish constituencies. Look for posts that are talking about issues that an aficionado of the game recognizes as important, but that don't seem to just repeat common sentiments. The number of responses to a post might be a key: distinctive posts, if they aren't lost in the noise of ordinary posting, often tend to generate a lot of replies. The reputation capital of a given poster is important. Any regular reader of a forum learns to pay attention to certain names and to seek out what they have to say.

In particular, what a developer should be watching for here is the equivalent of Martin Luther's complaints stuck on the door of a church: a manifesto which may galvanize or persuade players to action, which may mobilize players or transform the way they see the virtual world. It's not just that the developer-as-sovereign should be concerned about the political impact of such posts, but also that they may contain valuable insights about what the game needs which the developers themselves cannot arrive at independently.

---------

This last point informs a lot of my private suspicion about why developers often like to rubbish their own forums as sources of information, or see them as a backwater tool for "managing" (e.g., in many cases, placating or misdirecting) their community.

Why do sovereigns in the real world sometimes misread or misunderstand the political communities around them, or rely preferentially on tainted or flattering sources of information? Because, in many cases, they cannot afford to come to know that which they do not want to know. If the developer-sovereign doesn't want to decide how to "fork" the changing state of their virtual world in terms of the antagonisms between factions of players, or if different factions within the development team are themselves competing to influence the world's design in one direction or the other, it may seem best to close one's ears and hum real loud when confronted with knowledge about these debates. If the developer-sovereign has no intention to devote the necessary resources to fix a long-standing technical issue, or cannot find a way to fix a path-dependent design flaw, then it may be best to plead ignorance--which requires claiming that one does not read or cannot usefully engage information found in a public forum. If a powerful manager within a live team has an idee fixe about how a virtual world has to be, and tyrannizes his colleagues on that issue, it may be best for him to pretend that he's never seen any dissenting views from players on the forums.

If a development team clouds the processes by which it considers and dispenses with problems, it can also avoid making concretized promises for which the players will (often unfairly) hold them accountable. Political sovereigns in the real world also mystify the processes through which their specific interventions into the world take shape. That is not just the pragmatics of power: it is also an expectation that we have of authority, and that authority has about itself, that decisions which are utterly transparent, robbed of their mystery, are somehow made threadbare, shabby in their naked instrumentality.

So I don't want to give the wrong impression. This is not a plea for live management teams to subordinate themselves to the democratic will of their customers, to simplistically "listen better", to do community management more adroitly, or even to fully demystify the processes by which they come to decisions. More modestly, I'd only suggest two things. First is to stop rubbishing forums as a source of useful information. If developers really mean that they can't get any information from their forums, then I think they're pretty lousy at gathering information. If they don't really mean it, if this is just a cover for other, deeper issues, then I'd just as soon stop hearing the same old song. Second is that maybe, just maybe, there is a middle ground between the occasional desperate eruptions of information from behind the development veil and complete transparent disclosure of every tiny detail of the development process. That there is a way to communicate through and from forums that goes beyond "management" as a way of soothing customers and into something the relation between the rulers and the ruled. I am not adroitly "managed" if my political rulers tell me that mass media, the public sphere, citizen forums, blogs, and the like are just a load of goddamn noise, or too difficult to parse or listen to, or that there are citizens who "live society" and then there are those annoying bastards who keep insisting on "discussing how to live in society". And yes, yes indeed, there is a bad signal-to-noise ratio in American public culture, particularly in online spaces, but that is not an all-purpose excuse for ignoring what gets said, or listening only to the people who echo that which the ruler already thinks he knows. That doesn't serve citizens well; it does not serve authorities well. It is in no one's best interests. The same goes for virtual worlds. If those who manage them don't know how to listen to their forums, they ought to learn. If they don't want to listen to their forums, however unrepresentative or fractional participation in them might be, they're asking for all kinds of trouble. Demonstrably in the still-young history of virtual worlds, some of them have gotten exactly the kind of trouble they were asking for.

Comments

[monkeysan]

1.

I think one of the problems that occurs is when the same forum system is used for more than one primary purpose.

For example, it's simply a bad idea, imo, for developers to use boards as a means of communication with players, yet simultaneously decry those same boards as a disorderly festival of non-representative attention seekers. While the idea of class reps a la Anarchy Online, SWG, et al. may be have reasonable PR value, it has the side effect of legitimizing the forum system as an official source of information that may outweigh any PR benefit.

I see that as a bad thing. The forums should either be left to the community with no expectation of interaction with 'blues' or they should be *heavily* moderated to maintain reasonable signal-to-noise ratios.

I don't see the latter as ever being feasible, so I gravitate toward the former.

Let people 'play the boards game.' I can't say for sure, but if I were a developer, I'd be more than content to have customers who are essentially content to pay $15 a month to use the cheapest feature of my services.

The problem is when you provide a two-sided message to customers about the function of those boards--namely, that (1) They're not official and customers should view them as a portal for interaction among players and (2) promptly use those boards to engage customers in conversation with devs and post official information.

[Ghosty]

2.

The forums at UO Stratics may be an exception to monkeysan's comment. The general ('U-Hall') board is inhabited by Developers, CS, Stratics Staff and players alike, with a degree of controlled chaos - there's usually only four to six topics of major interest at any one time, with a lot of bandwagon-jumping goes on. It's pretty easy to see why the devlopment team uses the forums as a guage of player discontent or approval on their current projects.

[Sara Jensen]

3.

D00dspeak IS a warning signal: it says, don't bother if what you're looking for is a usefully distinctive response to the state of a virtual world, unless for some reason you have a particular interest in d00dish constituencies.

I agreed with you before I worked on a live team. After years of it, I learned that literacy skills aren't an indictator of actual game experience -- I'm interested in the feedback of people who know what they're talking about, whether or not they know how to say it.

Also, "data mining" is a technical term about processing real, hard data. The ability to read boards at a glance is a valuable skill, but it's subjective, and it's not "data mining."

[Timothy Burke]

4.

I understand what you're saying about the meaning of "data mining", Sara, but I think it's a good conceptual analogy to the kinds of mental and methodological processes involved in extracting useful information from very large bodies of information (in the fashion that historians do, for example).

[magicback (Frank)]

5.

Amazon does a good job of "informational mining" in their concordance, text stats, and particularly SIPs. Autonomy I think is the company or a competitor company that does these text analysis. Very cool and effective.

However, these text analyzers do not fully address the context issue, so the ability to glance at boards for key information is still in the domain of the human mind.

---

Now as for forums, see it as what it is: a mostly open forums for people wanting to speak their mind to do so. It's really not the venue for developers to solicit feedback (use another channel) or to provide company information (use another channel like an official bulletin board, which is one-way only).

It's great that developers can come by the forum and casually chat with the players, but it has to be clearly communicated that everything said does not represent the company and in no way hold any water officially. The official stuff goes through the official channels. Forum stays unofficial and casual whether managed by the company or by a third-party.

Frank

[Michael Chui]

6.

Whenever I start a new profession that I haven't played for a while (I've done them all.. =P), the first thing I do is spend a few weeks reading through every message and copying into a text file posts that are useful and relevant to what I have in mind for playing the profession. I have a number of players flagged, mentally, as "when they say something, pay attention". They might be wrong or inaccurate, but they're more likely to not be, and more often than not, they are wrong in a detail that isn't really under consideration. In other words, they know what they're talking about.

Granted, in Dragonrealms, there are staff (volunteer) positions called Liaisons whose job it is to play messenger boy between the players and the Gurus, who are supposed to take the burden of coding. They do this both in the game and in the forums, and seem to give equal weight when it's desired. (I've noticed that in-game usage is dramatically less; people seem to prefer using forums or even email to make contacts with devs; I certainly prefer it, but that's a personality thing.) They also make a point, from time to time, to explicitly inform the players that they're taking notes on what is posted on the forums, and there are definitive cases where the devs lift material suggested by players for inclusion into the game (with their own generous tweaks, of course). I've personally submitted two major systems with full release of IP rights to them, and hope to see them in development within the decade. ^_^

[Michael Chui]

7.

Bah. Frank posted at the same time.

It's really not the venue for developers to solicit feedback (use another channel) or to provide company information (use another channel like an official bulletin board, which is one-way only).

Naturally, I am compelled to point at Dragonrealms' forums, which include several levels of each. Most of the folders have an Announcements folder, which may not be posted in by players (I'm not sure if this is an "etiquette" rule enfoced by moderators, or a hardcoded fact.), a Reply to Announcements folder, and a General Discussions folder. Besides those, there are perhaps a dozen more specialized folders for particular topics, most interestingly a Bugs folder and a Conflicts (for player-player scuffles) folder and the board moderators enforce this by moving posts. Some folders are blatantly "Make a suggestion in me!", like the folder "New Spell Suggestions". If such a folder does not exist, people liberally throw their suggestions in whatever folder seems most pertinent.

Incidentally, a blog with comments disabled is an effective way to make announcements, but a way to let players discuss them (with or without developer participation) is not a bad idea. See blog support for television series. There's no reason to make it one-way: just don't make the reciprocation "email the author". =)

[Richard Bartle]

8.

One of my students last year did a final-year project which was to build a database of meta-information for forum postings. He concentrated on bug reports, but it was extensible. Basically, he spent 2 hours a night reading through postings in various fora related to FFXI, categorising what he read and tracking developments (and ultimately fixes). Using his tool, it was possible to get a quick snapshot of which bugs were of most concern to the players, and find possible solutions that they were suggesting.

This kind of thing is not true data mining, in that it has to be done manually, and it takes a special kind of individual who likes the work (fortunately, my student was just such an individual), but it can be done by part-time workers and it provides the kind of management summary that could really make a difference.

Timothy Burke>If developers really mean that they can't get any information from their forums, then I think they're pretty lousy at gathering information.

I agree: there's wheat among that forum chaff.

Richard

[magicback (Frank)]

9.

Thus, there are technologies and skills that when used properly invalidate the argument that "they can't get any information from their forums".

Thus, it's down to intentions and actions.

Blogs and forums to me implies informality, whereas posts on the company's main website are viewed as formal. An example of this is what goes on the WoW website and what goes on the WoW forums.

Blizzard has been pretty good at making the two channels separate and distinct.

---

Richard, was your student able to create a macro or agent to help process the information? Doing it all manually sounds painful.

Also, did any of your students create a database that compare game features item by items. I thinking that this may be an interesting projects where forum topics can be indexed to specific function of features in the database.

Frank

[Mike Rozak]

10.

Random comments... Makes me think of some forum tools:

- The ability to easily create a private ranking of users, as in: UserXYZ always posts random and meaningless posts (like this one) => ignore his posts, or UserABC has quality posts => read them.

- Some sort of summarizing tool to whiddle down overly verbose posts to just a few sentences.

- Spell checker to identify doodspeak.

- Use bi-grams or tri-grams to identify doodspeak. (Probably overkill.)

- Color-code messages by user to help identify times when a few posters try to drown out everyone else by making lots of posts.

- Design the forum software to encourage users to enter text in a manner that's machine parsable. For example: Have a per-message subject that's only one line, or provide checkboxes for "bug", "rant", "feature request", etc. along with each message. Or allow users to rate something as their #1 issue, but only allow them one #1 issue.

- Said forum tools could keep track of what percent of players actually view the message. Use this to get a wider audience.

- Quick voting per comment (one per player) of "I agree", "I disagree", etc. (Perhaps keep the tally hidden from players.)

- The value of a post is inversely proportional to the size of the signature.

PS - I was going to write a huge "Mike Rozak" signature, the kind where a big "M" is make up of lots of small M's, but Terranova uses a proportional font, so it won't work. I suppoose I could find the lyrics to my favorite song and post them in my signature every time.

[Endie]

11.

I was kinda intrigued by this, and ended up writing in general terms about techniques I would use to filter for a useful subset of postings to allow for situations where the consumers of the archive aren't professional historians!

One tool would be a variation on bayesian filtering, but the nature of forums, and the data available on people's posting practices, mean that there are plenty of other possibilities.

Of course, trackback is kinda strange on TN at the moment, so no doubt my trackback will appear to some future, irrelevant posting, but I did it at my site

Endie

[Andy Havens]

12.

Yes and amen.

Let's see... I spent 10 years in retail where we had to pay customers US$25-40/hour to sit in a 2-way-mirrored room to talk about our products at any length. The only real "natural" conversations that occurred were almost entirely negative and were related to either billing or functional product issues; ie, service calls. We spent tens and hundreds of millions and millions of dollars training customer service reps in how to take care of customers who had issues that often led to service termination. Those conversations were measured and monitored and analyzed very carefully in order to help maximize the training of our reps and the responses to customers, so that we could more adroitly keep pissed-off people from leaving.

Now... we have a situation where customers -- without being paid one red cent -- are taking additional time away from their lives not just to complain, but to discuss, philosophize, disect, admire, suggest, etc. all kinds of aspects of a product. They feel so deeply about a game that they feel compelled to talk to the company, with one another, to post ideas, questions, comments, rants, agreements, etc. etc.

I don't care if it's "players who post" or "posters who play" or "planters who post a peck of pickled peanuts." Any game development team that isn't spending a decent amount of time and money going through these forums to find out what to do next is idiotic. Pure and simple. This is customer data; they are giving you FREE MONEY VALUE. Apply directly to forehead.

Now... if I were king of the forest, what I'd do is make the forum system as specific as possible to both the needs of the users, and to the job of retrieving useful data. The one problem I've had with game forums is that all kinds of posts are mixed way the hell up in too few topic ladders and the search/tagging functions are woefully 1996. A PHP based BBS? Surely we can do better than that... I'd love to see a major game come out with something more like a forum wiki or a MySpace for their game that makes it very clear, "Here is where you go to talk about character optimization," and "here is where you go to bitch to the devs," and "here is where you go to ask noob questions," etc.

That way, it's better for the gamers, and easier for the devs to troll for info.

When News Corp bought MySpace, someone asked Murdoch, "Why did you do it?" I love his answer... "To listen."

[Sara Jensen]

13.

Mike said "- The ability to easily create a private ranking of users, as in: UserXYZ always posts random and meaningless posts (like this one) => ignore his posts, or UserABC has quality posts => read them."

Even normally less-than-astute posters will occasionally write a gem. I rarely even looked at posters' names when I was scanning the boards. I was a lot more interested in -- again -- what they had to say, whether or not they'd ever passed any test of worth.

More importantly, you can't read posts in a vacuum. Their context and all the accompanying dialogue in the thread is just as important. This guy might have written something you really agree with, but if none of the other posters do, you're probably both wrong.

A system that encourages the developer to only read a handful of posts, out of context, is one that'll mislead her.

Now, pulling some arbitrary "best and brightest" posters into a private forum is a old tradition of achieving the same goal -- when the devs are in a hurry, they only have to read that one board. It would be unfortunate to rely on their feedback alone, but it is a timesaver, and it doesn't rob the reader of context.

[Thabor]

14.

Unfortunately I have nonthing substantial to contribute. to this discussion.

Just wanted to say nice job to Timothy, and I'd have to totally agree with Andy. Its totally alien for me to see a software company identify an information processing problem without taking action to improve the process.

[Matt Mihaly]

15.

First thing I look at when I'm doing competitive analysis on another game is their forums (before the game itself). If I'm able to glean useful information about strengths and weaknesses from them, I'm sure the developers can get far more information from sustained exposure to the player feedback therein.

It's an odd decision not to treat your forums as important and part of the overall experience. Yes, maybe only 20-30% of players use them, but then, I've got games that have entire systems that 5% of players might ever use (using the in-game theatre system to put on plays for instance) and we're not going to be removing those.

--matt

[Michael Chui]

16.

Even normally less-than-astute posters will occasionally write a gem.

Yes, but some people have thin skin. =P That said, however...

- The ability to easily create a private ranking of users, as in: UserXYZ always posts random and meaningless posts (like this one) => ignore his posts, or UserABC has quality posts => read them.

I've seen this utility, shown in-game as "squelch" and in-forum as "ignore". I use neither, in part because I've taught myself to read and parse things quickly, in part because I have a reasonably thick skin, and in part because I want unaltered archives.

If I'm able to glean useful information about strengths and weaknesses from them, I'm sure the developers can get far more information from sustained exposure to the player feedback therein.

When Richard did his jump-to-level-60, he mentions that he did a lot of research beforehand, and in particular mentions forums.

When News Corp bought MySpace, someone asked Murdoch, "Why did you do it?" I love his answer... "To listen."

Beautiful. =)

I'd love to see a major game come out with something more like a forum wiki or a MySpace for their game that makes it very clear, "Here is where you go to talk about character optimization," and "here is where you go to bitch to the devs," and "here is where you go to ask noob questions," etc

Dragonrealms does have this. =P

Surely we can do better than that...

That's my current project, in fact. Apparently, this thing I'm building has a diverse set of applications. Note to self: market to community managers, too. Hey Andy, want a job? *ducks*

[Evangolis]

17.

At this years AGC, the most personally illuminating session for me was entitled Improv for Game Writers. I was actually about blue-sky team creation practices, using the techniques of Improv to demonstrate the value of accepting the ideas of other team members unconditionally. Once the point got across, one of the first responses from the writing-oriented audience was “Why isn’t this session being run for programmers?”

Now I am a (non-game) programmer, and my thought was, “Why isn’t this in the Production track?” I was struck by the value of unconditional acceptance of ideas, but the session leader brought out the major objection that I had as a programmer, when she pointed out the need for later editing and refining of such ideas. It is all very well to say yes to everything in a blue sky mode, but as a person charged with converting ideas into functional code, I also need to be able to say no to anything, however neat and wonderful it might be.

It seems to me that there is a great deal of this issue in developers taking meaning from the boards. Boards are in many was the ultimate in blue sky thinking, with little constraint to what posters can propose. At the same time, experience shows us that even the least committal of maybes will be taken as a firm promise of immediate action, and the power of the dev is generally assumed to be limitless; witness the frequent illusion by posters that it is possible to write defect free code, and failure to do so equates to permission to use such defects to advantage. Under those circumstances, engaging in any way with the public boards is inherently a very hostile situation for the developer.

As a result, it is impossible for the developer to actually engage in a creative dialogue on an equal footing on a public board, for fear of becoming either Abashi, in constant argument with posters, on the one hand, or GL-Jeff, agreeing to the most preposterous of concepts, on the other. Why should a developer see such a fundamentally hostile place as a good source of information, regardless of the actual value of the information there? And is there any way to change this dynamic?

[Allen Sligar]

18.

Actually one of the features we're releasing in January is a customizable and deployable DSS (Decision Support System) for game studios for direct targeting and data mining. How effective this tool will be is dependant on a number of factors (assuming I don’t have the data set you want already).
1. How much buy-in your player base gives the tool (this will likely be heavily dependant on the studios motivation and initiative)

2. How robust you want the data to be, meaning the client will drive the data to be captured (with guidance from us)

We'll be able to deploy this tool in less than 24 hours.

I get a fairly wide range of responses from the game industry developers when I talk/describe what we're doing. Some see absolutely little utility in having data about their player base/consumers while others generally want to know more and see it as a long sought after source of information.

As far as executive management/finance/marketing/distributors/publishers, I have yet to talk to one person in any of these fields who are associated with the game industry that does not want more data about gamers.

As of last week we had around 987 data points, across a number of strata.

I'd like to add that we're going to add more data points based on community suggestions, game industry, and academic suggestions (the latter two will be carved out private forums) as I know there are more data points that can be captured and I think the best means of vetting them will be community involvment.

As someone mentioned above there is a differance between vetting information and data mining.

[Allen Sligar]

19.

I'd also like to add that the hardest part of what we're doing in developing this is not building the backend, features, site, or figuring out data points to capture. The most difficult part is setting a reasonable price point for people to have access to the data, given the number of stakeholders and the demand for the data, I'm very commited to making sure its accessable for the industry, and virtually free for the academic community.

Also I'd like to apoligize if this came off as self promotion in any way, its not that so much as I wanted to make sure the TN community understood a bit more about what we're doing in relation to this subject.

The fact of the matter is the industry has little in the way of data about its own consumers, something we're trying to solve, and hopefully this results in better games.

[Nathan McKenzie]

20.

Are there any good, hard, data-filled tests anywhere that indicate the correlation between what forum posters think / want / desire, and what average players want? Or even between what forum posters say they want and what forum posters actually want after all the work has been put in to fulfill their desires?

I've worked on commercial multiplayer games, but never any persistent ones (think more in the counterstike vein). Over time, I found listening to forum posters a) seemed to give me profoundly skewed ideas about gamers wanted, compared to what I'd find when I would talk to random people who happened to like and play games and b) sapped my enthusiam for making what I was making. I happen to be pretty thinned-skinned, though, I will concede.

I wonder sometimes if forums are like the SAT or IQ tests - they're really damn good at measuring what certain collections of forum posters think at a given moment in time...

...which is perhaps why the blogger onslaught didn't push John Kerry into office with 85% of the vote and Snakes on a Plane didn't pull in 150 billion dollars on its opening weekend.

[magicback (Frank)]

21.

Here's a blog article on Netflix offering $1m contest for the best system of recommending film and the determination/valuation of that $1m + support cost to get that feedback in the perspective of The Long Tail.

Chris Anderson, the author of the book and the article, clearly demonstrates the economic value of "findability" and in this case via recommendations.

So if we view forum posts as a form of recommendation platform and apply Chris Anderson's analysis to this, interesting logical extrapolations results.

Frank

[Andy Havens]

22.

@Michael: I certainly am available to do marketing on a consultative basis... whether or not you can afford me? *ducks*

Evangolis said: "Why should a developer see such a fundamentally hostile place as a good source of information, regardless of the actual value of the information there? And is there any way to change this dynamic?"

It all depends on how you view the situation and how you train people to respond.

I like to tell the joke about the marketing guy, the programmer and the lawyer in the train who look out the window and see sheep being shorn on the mountainside. The marketing guy says, "Ooh. How quaint and pretty. The sheep on the mountain are being shorn." The programmer says, "Well. Be precise. All we can truly say is that the sheep on *this side* of the mountain are being shorn." And the lawyer says, "Be careful... All we can really say for certain is that *these sides* of the sheep on *this side* of the mountain are being shorn."

Yes. There is a place at which a representative of a corporation must stop short of making undue public promises of future service delivery on which he does not have license to proceed. Customers of that service might rely on said promise when considering one service option over another. Upon future lack of fulfilment, the corporation might then be liable for the difference between the two states; that which was promised and that which was delivered. IE, if you sell me a horse, but deliver a mule... you owe me an ass.

But a good training program will make clear how to handle all manner of customer communications. And when someone asks for something unreasonable, you don't have to say anything about whether or not you're going to *do* anything. You just mimic Murdock's statement above: You say simply, "I'm listening." Or, "That's a good point." Or, "You know, I've heard that before." Or, "I understand where you're coming from." Or, "You sound like you know what you're talking about."

Another joke... A politician who claimed he never lied was, at a community affair, purposefully introduced by his opponent to a woman with a horribly ugly baby. After sucking it up and kissing the wee pug, the not-so-honest pol, with a wry grin, asked his opponent, "Now, don't you think she's beautiful?" With a crowd of potential voters, including the doting parents looking on, the honest fellow simply said, "She looks just like her mother."

When a customer is talking to you, you're golden. Because it almost always means that they WANT to be in the conversation. 99.95% of the time, there's a way to turn it around. .05% of the time it's a psycho who's just spewing bile, but that's very, very rare. Even psychos have better things to do with their time. Usually, even the loudest, most obnoxious customers just want a bit of attention and understanding and empathy. Because they gave you some money for something that they either really like and expect some respect for that, or that they're disappointed in, and would like some acknowledgement.

The scariest noise you can get from a customer?

Silence.

Crazy don't bug me. Loud don't bug me. Rants don't bug me. Whining don't bug my. Syncophancy (is that a word?) don't bug me. Bring 'em on. The worst signal-to-noise ration in the world is a haaaaaeeeeellll of a lot better than a flatline.

[Richard Bartle]

23.

magicback>Richard, was your student able to create a macro or agent to help process the information? Doing it all manually sounds painful.

No, he wasn't, and yes it was. He started to read through postings manually to begin with in order that he could see what kind of information he'd need to put in the database and how to categorise it. It was an option to use some kind of keyword recognition to extract promising posts, but it soon became apparent that this was no good unless you knew what all the keywords were ever going to be. He didn't, and new bugs and exploits were appearing all the time. Also, some of the posts addressed several issues and needed to be referenced under two or three categories.

The point was, though, that he did read the posts (well over a thousand of which were worth categorising) and in doing so he gained more of an understanding of the player base than if he'd just looked at a summary. He could make recommendations as a result (and indeed wrote a report doing just that). Numbers of posts on a topic are a guide to importance, but they're not always an accurate reflection of what needs to be handled first: a mildly irritating bug experienced by a lot of players will generate a lot of posts, but a show-stopper experienced by just a small minority won't. He could assess how important a bug was (or was becoming) and raise an action flag if it looked serious.

>Also, did any of your students create a database that compare game features item by items.

No, but I know a group at Teesside that's doing this as a spin-off professional business: Strange Agency. The idea is that if you're developing a game and want to see whether similar games exist, you fill in a form and your criteria are matched with about 5,000 or so games in their database (which they built by scanning reviews from a number of game and game history sites). They're not so specific that they can tell you different features between virtual worlds, but they can tell you if there's been a gangster-based FPS for the PS2 (they have 50 or 60 criteria, I thing, obtained by some kind of cluster analysis).

Richard

[Thomas Malaby]

24.

A helpful example, Richard. This part struck me in connection with what Tim is saying:

Richard wrote:

It was an option to use some kind of keyword recognition to extract promising posts, but it soon became apparent that this was no good unless you knew what all the keywords were ever going to be.

This connects to the nature of qualitative, exploratory research. It is precisely because one doesn't know the range possibilities going in that the very hard work of first-hand review can't be left out. As Tim also notes, one does develop a practiced sense of what's relevant specific to the case at hand (though this remains too complex to systematize reliably), and one also develops skill at archival research that improves one's ability to do this across cases.

Both of these mean that it's not *quite* as daunting as it could be, but exploratory work is always, I think, going to seem a bit frustrating in an era (and industry) in which we are predisposed to think automation must be an option in every case.

[Andy Havens]

25.

Thomas said: "Both of these mean that it's not *quite* as daunting as it could be, but exploratory work is always, I think, going to seem a bit frustrating in an era (and industry) in which we are predisposed to think automation must be an option in every case."

Well... in thinking about this, and the phrasing of the term, "playing the forums," I wonder if you could devise a UI for forums that encouraged a more play-like use of them that would be easier to automate data collection, because the "forum players" would not dump everything into a wordy heap, but would do more "game like forum things" in order to get ranked, rated, bumped, etc., in terms of the "forum game?"

For example, if you simply just go off... fine. But if you take 5 seconds to use the company's tagging system to accurately categorize your forum post by topic, feature, plus/minus reaction, etc. etc., you've immediately put it in the context of "pre-automated" response. Doing enough of these things -- even if the content of your answers is not positive -- would be reflected in a positive rating. IE, I'm a really high-ranking forum player who always uses the forum correctly to say negative things about the game. The publisher should encourage exactly that; civilized, constructive, ontologically precise criticism.

eBay has a system that rewards "playing the forums," in a similar way... it's just that the forums, in the case of eBay, are the whole game. There ain't nothin' but the forums. And the end of many forum incidents/actions is a purchase. But there is a chance to rate that action. A similar system for rating the "meta behavior" on a forum would, I think, yield much more easily parsed information for the company.

It would also make it easier for the company to understand which qualitative posts to delve into more deeply. A forum-player with a high ranking and a high word-per-post ratio? Read those rants. A forum-player who just goes off all the time, yammity-yammity-ding-dong and doesn't care about using the system? Probably less likely to yield you good data. Why? Two reasons: one, they aren't bought into the social group idea; i.e., they aren't committed to making it better, just to biyatching. Two, they don't understand the game aspects you've put into place.

So. If people want to play the forums... Put up some goals, give people score cards, and rank those muthas.

[Allen Sligar]

26.

Thomas: "Both of these mean that it's not *quite* as daunting as it could be, but exploratory work is always, I think, going to seem a bit frustrating in an era (and industry) in which we are predisposed to think automation must be an option in every case"

Granted exploratory work is required, formulating that which must be captured as important as the capture of data. Sometimes more, sometimes less.

If the point of the data capture is research, and academic persuit, then the execution and actionability of the system (DSS) takes a back seat to the formulation and capture.

If the point of the data capture is actionability and execution of managerial decesions, then data needs to be delivered as quickly as possible to the client (else its not really a "support system")

Automation from the perspective of business is therefore a preferance, because it provides the organization with a competative advantage. And I dont mean to infer that exploratory research does not also provide competative advantage, what I am saying is that where time is an issue and actionable data (rather than academic expertise) is required, automated real time DSS are typically the prefered means of knowledge discovery (data mining).

The real trick is to serve all stakeholders, business, creative, academics and consumers. As I noted earlier the best means of ensuring this is community input.

[magicback (Frank)]

27.

Christopher Allen of Skotos fame and Shannon Appelcline has written about ways to manage creative input

So in combining the index system design by Chris and Shannon (among other useful designs) to forums, we can design Forum 2.0 that is less chaotic and more useful for all stakeholders.

Frank

[Thomas Malaby]

28.

Well put, Allen.

[Allen Sligar]

29.

Also I'd like to invite any TN readers (specifically acadmics as I'm trying to flesh out data capture that would be useful for the academic community a bit more) into our internal beta testing of the site/functions or for a preview just prior to launch, I can be contacted at: [email protected]

You dont need to do much more than contact me, and I'll send you out a login and such when we're ready next month.

[Michael Chui]

30.

Speaking of links, Shannon Appelcline also did a couple articles on Skotos. The latest is:

http://www.skotos.net/articles/TTnT_/TTnT_196.phtml

Previous articles are listed at the bottom.

[Allen Sligar]

31.

Ty for the linky Michael, nice stuff there

[CherryBomb]

32.

Andy Havens is a bit of a nut. I am starting to like him.

If you ever feel that a game publisher is ignoring stuff on the game forums, try posting something like "Calculation of exploit money as a percentage of total game output", or "Estimating concurrent users using in-game data." They will glom onto that in minutes, and prolly squelch it.

[Andy Havens]

33.

@CherryBomb:

1. "Andy Havens is a bit of a nut." I resent that. I am quite a bit more thant "a bit" of a nut. I am the whole f'in jar, a squirrel-belly-full, plus Lord Planter's sack stuffed to the draw-string... of nuts. Show respect.

2. "Starting to like me." Yummy.

3. Never said they weren't paying attention. But from some of the comments here and on Raph's blog and on the SOE forum itself, it seems that some of the attention is like a spanking; bad attention. And, again from some of the comments from people who work/have worked on the forums, they don't seem to get it that any conversation with a customer that doesn't end in "I'm outta here," is a winning conversation. And that any chance to gleen any kind of data that makes a product better is amazing.

Out in regular ol' retail land, it's getting harder and harder to do surveys. Why? People are getting tireder and tireder of taking them. In a direct marketing session I attended recently I heard that the price of administering a good, reliable survey went up about 20% over the past couple years specifically because people are "surveyed-out." They get hit on the web, in the mail, by phone. People (some of them) used to like to take them, now (more of them)... not so much. On the forums, you have people who are engaging in self-selected survey behavior at essentially zero cost. Yeah, I know. You need to pay for the server space and some moderator time. Compare that to what it takes to actually do a real retail survey for a product like automobiles or cell phones or soap or a TV show. It's free, basically. And the volume of info is huge.

It's not just signal-to-noise. It's money-for-nothing.

"Bit of a nut..." What's a guy gotta do around here to make it into the "tree dwelling loon" category?

[magicback (Frank)]

34.

The "surveyed-out" trend is spot on, so for the past 10 years people have been focused on behavioral-based analysis: behavioral finance, consumer behavioral analysis, advertising, Amazon, Google and host of other companies trying forecast what you want before you even wanted it (e.g. a fridge that more than just tell you what you are running out of inside, but recommends what to get).

However, while it's great to read the forums for observations and insights, it can lead to the wrong conclusions (many exampled already given) or based on wrong sample of the population.

[Andy Havens]

35.

@Frank, who said: "However, while it's great to read the forums for observations and insights, it can lead to the wrong conclusions (many exampled already given) or based on wrong sample of the population."

But at least you've got a sizeable sample of the user population with which to gather some data and make some conclusions. If you're making wrong conclusions... well, it's because you're using the data badly, or in not enough ways, or trying to make it fit a preconceived notion. There's no such thing as *bad data*, just dumb interpretations.

When Pepsi did the famous "Pepsi Challenge," they gave people two wee cups of cola. People overwhelmingly liked the Pepsi better. The ad campaign based on this fact did very well for Pepsi. Coke's research confirmed that the challenge was for real. This scared them so much that they blundered into the New Coke fiasco that almost killed them. What was the missing fact? The simply truth that people don't drink cola in tiny little wee sips. They drink it by the glass, the can, the bottle. In small sips, the sweeter, more mellow taste of Pepsi is more appealing. But as an actual beverage experience, hard-core cola drinkers can tell you that Pepsi has a funky aftertaste and that Coke has a "bite" that is quite specific and pleasureable. Coke drinkers didn't switch to Pepsi because of the taste, but because of the advertising. And which the die-hards were confronted with a possible change of their beloved brand... they went ape s**t.

So... I agree with Frank that there is never going to be a 100% correlation between what the folks who "play the forums" say/whine/rant/confess about the game, and what the great masses who never touch the forums are actually doing. But there will be, if you mine the data, enormously helpful connections to be made. Especially if you start matching data points in the forums against data points from other input/output systems, like in-game help querries, new player sign-ups, advertising response rates, game reviews in major mags, etc. etc. The forum's aren't a vaccuum; they're not the only answer. But it's a huge, uncensored wash of nearly free customer response data that's living right next to your product.

If I were the king of that forest, I'd hire a guy whose done 10+ years of "two-way-mirror" survey work and ask him to set up forum systems, situations and rules for my moderators that would draw out the most helpful data. I've watched these guys work. You give the nice people sandwiches and soda and you ask your way around the questions you really want answered, because if you simply walk up to somebody and ask, "What sucks the most about your cable TV service?" you will get less-than-helpful data. Mostly rants about down-time and price. Which you already know.

So... in order to properly mine the forum, you have to first design the shafts properly, hire good geologists, and be prepared to sift through the crud.

Or you can just let your customers whine and beeyatch and share exploits and hax pay no attention at all, and occasionally log on an vent yer own spleen. Either way.

[CherryBomb]

36.

I miss those 8 ounce bottles, actually. Nothing spoils the taste of a good pop like way, way too much of it.

And sorry, Andy, but my bar for lunacy is set pretty high. Go out and get yourself a Nobel Prize in physics, then get so drunk that you pee in my garden, and I may re-consider. (I am not making this up, btw.)

[Andy Havens]

37.

@CherryBomb: Does it have to be for physics? Cause that just ain't gonna happen... And does it have to be alchol related? Cause that ain't gonna happen, either. I'm lousy at math, and I don't drink. So either we'll have to accept that I'll never be your kind of loonie, or you'll have to open the tent a bit wider.

In the meantime, I will take "bit of a nut," from you, as high praise indeed. ; )

[I did hang out for a year at MIT during my then-best-friend's senior year, as a kind of "pet poet in residence." Stories from that period include props and characters such as; the melon trebuchet, tape web wars, robot grappling hooks, rubber-band gattling guns, vending machine hack wirez, live-Tetris-mimes, The Shaft of Disposable Technology, microwave oven horror fiction, the longest insult in the world, beards akimbo, hide-the-Harvard-Street-bridge, Confuse-a-Frosh, Peter Pan on Crack, God Has No Pants and "If You Love Her, Give Her an Elevator." Don't know if participation in any of which, the full description of same being a bad idea on an open forum, starts to nudge me closer to your definition. Keep in mind that this was in the late 80's. I've had 15+ years since then to indulge in other, various schemes. Offline, I'd be glad to provide my entire Ciriculum Dementae.]

[Prokofy Neva]

38.

This thread shouldn't go by without somebody denouncing Linden Lab's decision to close their substantive forums ("General," "Land and Economy," "Political Science" etc.) and replace it with this unwieldy company blog. The blog entries not uncommonly get 200 or 400 or even 700 replies to them, and the original Linden poster often never comes back on to talk. Some posters are summarily deleted, recognized on sight despite their content. Some Lindens then continue to argue with the poster whose post they've deleted. It's truly a wacky experiment.

The Lindens claimed they couldn't handle the volume of reading nor the negativity, which Philip Rosedale, CEO, complained was "de-motivating" his staff.

I totally agree with what Raph Koster has been writing on this subject -- that playing the forums is part and parcel of playing the game for some and allowances must be made for it.

This is a "we need more community managers" problem. Hire a few professional, paid, benefited, competent managers of forums not drawn from the player base to moderate and let the players/residents have at it. Surely LL and other companies that have shut down forums will come to see that while they once had loyal subjects who only complained within the confines of their walled gardens, when they ban people from forums (like me) or close the forums and essentially ban everybody, they only force them to be more voluble on their own or other blogs and get even more Google hits .

[Michael Chui]

39.

I recently found these guys--Alchemic Dream. I have no idea whether or not they're any good, but since I felt like it (it's 5am; I feel like anything) I thought I'd toss in some free PR for them. Go hire them. Or one of their competitors.

[Andy Havens]

40.

Prokofy said: "Hire a few professional, paid, benefited, competent managers of forums..."

Easier said than done. Every company where I've worked, there has to be a pretty decent defense of new headcount. And when it comes to a totally new job classification, that defense has to go up to the highest level. You have to basically prove that hiring, in this case, a new Forum Manager or three, would benefit the company more than hiring the same number of programmers, marketing honchos, PR dudes, customer service reps, etc. Because those are already positions where you can (probably) get some ROI numbers going.

I'm not saying that the Linden experiment in how to handle forums is good; not in any way. But if a majority of the compaints and discussions on the various forums (Linden managed and otherwise) are, let's say, "the code is buggy," and "in-game help is non-existent" then hiring forum managers to manage those discussions is like hiring more 911 operators when there ain't enough cops on the street.

[magicback (Frank)]

41.

@Andy

While your point about headcount is true, the late Peter Drucker said that the pursuit of profits as the only corporate objective is misguided and that the corporate entity should also focus on other areas of corporate survival.

He wasn't focus on the specifics of managing online game companies, but I would think that he would agree that community management would be an important area of corporate survival and also address the social responsibility aspect of his management philosophy.

Many have spoke of the underinvestment in consumer-side of the business, so whereas 1 GM per 5000 accounts maybe the right ratio, one or three professional community manager should be the minimum requirement for long-term survivial of the corporation.

Yeah, I'm currently reading a book on Peter Drucker :)

Frank

[RxPillFirst]

42.

A man walked into his doctors office, and asked him for 3 viagra pills.

The doctor asked, "Why only 3?"

The man said, "Well, Friday my secretary is coming over, Saturday my girlfriend is coming over, and Sunday my wife is coming home from her vacation.

The doc said, "Thats more than I wanted to know, but here's your 3 pills."

A week later the doctor saw the man at the gas station, his arm in a cast, and sling.

The doctor said, "What happened to you? Did the women all find out about one another?"

The man said, "No, Nobody showed up....."

;)

And serious news: We have launched new [url=http://pill1.info]generic viagra[/url] store.
Best prices for cialis, viagra and levitra.
You'll be greatly happy to see our exclusive suggestions.
Coming soon - Hoodia and Soma!

Enjoy!

[Russ C.]

43.

Very interesting article, and it's absolutely true; there is value, although I would tend to prefer an official moderated forum instead of the ranting 'fanb0i' forums which are sadly prevalant recently, especially with the masses of free forum software out there.

I've used a lot of forums and I've found the the signal to noise ratio is very good on the forums for EVE-Online; but in a seemingly (and possibly sadly) rare move, the Developers, CCP actually go out of their way to post very regularly, informing the forum populace of upcoming developments and even giving feedback on player suggestions.

This is cool twofold, it means that they get a lot of valid feedback and most importantly; because the (mostly mature) regulars enjoy and respect this connection, they keep the number of trolling doodspeak posts to a minimum.

Also fair cred to EVEs active forum moderators, who always seem to be on hand and answer their emails pretty quickly (also rare!), that obviously helps this situation.

On the otherhand though, as posted it does also encourage the amount of Sycophantic CCP does no wrong posts ... But I guess it's hard to strike a happy medium.

What I'm trying to say, with CCPs model is that you get out what you put in, if you have a forum that you don't use properly and in-turn only seems to exist for people to whine and GMs to hand out retribution; then don't expect to get a goldmine of realworld feedback on your product!

[Andy Havens]

44.

@Frank: You'll notice I didn't say the company shouldn't spend the money. Just that you've got to defend the expense of a forum employee compared to some other stuff. If your by-god customer service phone lines, for example, have people on hold for 10-15 minutes on average with tech issues... if your code is buggy... if your servers are crashing... if your ads and PR aren't getting spanked in time... you've got other personnel needs, too. So I'm not suggesting you pare to the bone on doing what needs to be done, and if you can make a customer service or brand or biz dev case for forum staff... here here! But if you've got other holes in the dam that need filling... product holes, for example, that's going to be a hard nut to crack with your management team, I think.

[Allen Sligar]

45.

Speaking of the Games and Data Mining segment:

Nielson was nice enough to announce a service similar to what we're doing. I guess I should be flattered when someone wants to compete before we even launch.

GamePlay Metrics?

Of course they get a serious F- for originality in naming.

[magicback (Frank)]

46.

@Allen: “Only the paranoid survives.” I’m sure your company is closer to the customer and thus will have a more customized and suitable service.

I just viewed a PARC forum http://www.parc.xerox.com/cms/get_article.php?id=589”>presentation by Peter Norvig, the Director of Research at Google (after I watched Nick Yee’s presentation), and it was really enlightening about their work in textual analysis (and now probably audio and video analysis) and the realization that using brute force text analysis is currently still better than using parsers.

BTW, have you written a write paper on the company's approach to game metrics?

@Andy: I agree with you in general. However, I am sensitive to the general tendency to focus on the quantitative (tangible value) and neglect the qualitative (intangible value) in the cost/benefit analysis. I am also sensitive to the tendency of underinvestment in the area of customer focus.

Therefore, I'm just adovcating wise, but minimum investment in the area.

Often time management cut areas that can not prove quantitiative on paper their value. I'm thinking it's because they're using the wrong benchmarks and metrics. A good example are certain outsourcing decisions that did not adequately factor the human component in the business case.

Frank

[Online Casino]

47.

Online Casino, Online Casino Games,Free Online Casino Games,Real Vegas Cnline Casino,Online Casino Reviews,Online Casino Gambling,Online Casino Blackjack,Online Casino Bonus,Royal Vegas Cnline Casino,all slots Online Casino,Grand Hotel Online Casino,Golden Palace Online Casino,Club Player Online Casino,Las Vegas Online Casino,Online Casino News,River Belle Online Casino,Carnival Online Casino,Platinum Online Casino, The Gaming Club Online Casino,Top Online Casino,Online Casino Free Bonus,

[wuyiya]

48.

Dear friend, if you play online games ,please contact them via:
AIM：gilssales
ICQ：108239029
Yahoo: sellgilstoyou 5051360 8379263 13107942609
Msn:[email protected]
http://www.gneedg.com

[Jeff Swan]

49.

High! Interesting site you have here... Thanks for it!
http://thecasinodirectory.sbn.bz/ - find out everything on gambling online