I was going to post a review of some papers from the Other Players conference (many very interesting ones btw) and then this entry but since a discussion got going over in another thread I think it's worth foregrounding now. One of the big conversation points that came up at the end of the conference was on the relationship of scholarly game studies to design. Eric Zimmerman raised a very good question and concern at the end of the last session which was why are we seeing so many academic studies include some kind of conclusion around "what this means for design" or "how designers can use these findings." I was particularly grateful for Eric's intervention because I've personally been feeling a growing unease with how often scholarly work in games seems to seek a kind of legitimacy by being able to sum up how the research can be used in future game implementations. It's of course not the case that we never want what happens in game studies to have real impact and influence but can we always anticipate what that will be? By making moves into predictive work are we quickly losing ground for good basic research to thrive? Are we beginning to foster a space in which research is primarily legitimated through its usefulness in building future MMOG markets?
There seems to be at least three levels at play here - what I might call critical game studies, critical or innovative design work, and market-driven design work. It seems to me that more and more "speaking to design" or designers is conflated to the third category - addressing market concerns (the need for more eyeballs, more subscribers, bigger games, etc). But interesting design may, in fact, be that which is political, provokes, offends, unsettles, runs counter to mainstream sensibilities, or even challenges commercial orientations and I worry that all too often this point is overlooked. My concerns are, in fact, not primarily directed at designers. While of course there are cases (most?) in which research is dismissed outright if it doesn't have clear, direct utility to building markets I'm less inclined to try and fight that battle. In fact, when I think of designers who approach this most thoughtfully they don't see it as a burden we in academia need carry. As even Richard Bartle notes in his thread post,
"I'm with Eric in that I don't think "games studies" researchers should have to justify themselves to designers. As a designer, it's more useful when they do justify themselves, because it means I don't have to go scouring the literature for interesting stuff - it's already flagged for me. They shouldn't feel obliged to do that, though (especially if it reduces the chance of their getting published in their "home" field)."
Of course, it's always good to think about how to translate one's work out, but I am not a designer in any fashion. I know what is critically important to my subfields and what I find analytically important, but I don't have the orientation or the skills to invoke innovative design. So my question is more at us, the academics. Are there ways we are fetishizing market-focused design which is concerned with that very narrow branch of the commercial MMOG world? Social science in particular has a long history of working in the service of marketing and while I don't want to suggest that's not a fine path for some, I feel like I've been seeing quite a few studies that try and talk to the most limited conception of "design." It's not a clear line of course. A lot of the work we do can certainly be used to retain players or find new demographics for games but I believe we have to retain a space for other kinds of work - often basic, often critical - to exist.
Does this mean I'm saying scholars cannot, should not, pay attention to design? Absolutely not. My training leads me to always want to inquire about structure and in games & virtual worlds that can go to the heart of design. Indeed I often find myself engaged with what I consider critical political issues that touch design spheres. But that can be very different than trying to then place oneself in the role of quick'n'dirty designer or marketer. There are those talented individuals, of course, that can have a foot in each world and do justice to both. This post is in no way meant to incriminate the work done by people in trying to translate their findings out to a broader community or embody it in concrete forms. But scholars increasingly feel significant pressure to legitimize their work only through industrial channels (not to mention those that can only secure funding by doing so). We often exist in cultures and organizations that continually say the only real means of validation can come through industry and usefulness to it. Unfortunately I think much of that argument rests on a very limited understanding of how influence and innovation can occur (for most of us our work will insinuate itself into practice and organizations via our students and not because the current scene takes it to heart). It's this rhetoric, and the future it suggests, I hope we can find pockets of resistance to. At the very least, maybe we can have some fruitful discussions about it along the way.
I agree, and I'm glad you pointed out that a large part of research will spread its influence through students and so on because it's a great point. I would also like to see more work at a more basic, critical level, and especially more that interacts with the wealth of existing theories in other disciplines. I think many people who have done, are doing or would like do this kind of foundational work are still put off by what can seem like a primarily pop culture and industry-oriented discipline riding in the wake of the media's increasing attention paid to the big commercial games.
What if we could convince the public that flashy 3d graphics, visceral violence and level-grinding isn't the end-all of gaming, by teaching them "how to play, and why"? There are plenty of things that need to be said about games that are of absolutely no use to designers, possibly even detrimental to them - for example, what if our research enlightened millions of gamers and convinced them to stop buying expensive video cards and all of the latest computer games and go spend their free time with a two dollar Nerf football, or a ten dollar chess set, or *gasp* a five dollar classic novel, for example? Or what if a distributed open source gaming movement began to dig a big hole into EA or Sony's market share, like a Linux of games? It could be a big loss for the industry but a big win for the academic discipline and for ludophiles in general.
It's useful sometimes to step back and think about the theoretical possibilities of games as well as look at games that aren't stuck tightly on the path of industry and technology. All research should be applicable to our further understanding of games, but certainly not all research is applicable to their further design.
Still, economic interests are always a guiding factor in any field of research because the people with money tend to want practical results on their investment so there will always be at least some pressure to remain commercially relevant.
Posted by: Will Jordan | Dec 10, 2004 at 22:27
I'll admit that I had hoped this discussion would end with our conversation in the ITU atrium. Nevertheless, I'll offer a few points I tried to make in our hallway conversation.
(1) The underlying implication that the Other Players papers wholely or largely attempted to "justify" themselves for designers is dubious.
As a machine for provoking discussion, this discussion is valid and useful. But those not in attendance at the conference may get the impression that it was replete with sentiments of wholesale pandering, a grave misconception.
(2) TL's point about funding concerns is valid, but it risks conflating design (an already squishy subject) with commercialization, which is dangerous.
In the MMOG world, it's certainly much harder to imagine artists or independents using virtual worlds for self expression... at the same time, the ITU has launched an open-source MMOG project that specifically intends to explore unexplored terrains of virtual worldness.
(3) The many specimens of "non-designer-justified" game research suggest that this problem is perhaps smaller, or at least different, from how it has been characterized in this and related discussions.
(4) Game researchers who see the practice of creating games (both in terms of material conventions its role as a market) as separate from and unrelated to game research are missing something.
This has nothing to do with applicability of research to practice; it speaks to requisite fluency in the medium qua medium.
(5) Whether it means to or not, this discussion risks striating game research, such that some research is "more research-like" than others (no matter who. This is an unproductive and even self-destructive trend.
I am a designer and a researcher. Some aspects of my research seeks to inform design directly, while other aspects seek to perform "disinterested" criticism. Still other aspects reside in a kind of vortex between questions of design, technology, criticism, and social issues. Given my own predispositions, I question whether there really are "two kinds" of research, as there would be in basic and applied sciences.
When I attend conferences, I see approaches that are very, very different from my own (for example, highly quantitative research, or ethnographic research). Such is the case in our broadly cross-disciplinary field.
(6) While understandable given our field's youth and its hyperpostmodernity, the number and frequency of metadiscursive research questions is troubling.
Similar discusssions came up in the annual DiGRA meeting, which took place at the conference. Now, we all have to draw our personal and institutional lines in the sand. I do this almost every day, and at Georgia Tech we certainly think about the question of what kind of game research we want to represent. But, I hope such efforts represent a minority of my mindshare.
Posted by: Ian Bogost | Dec 11, 2004 at 12:46
Ian Bogost>The underlying implication that the Other Players papers wholely or largely attempted to "justify" themselves for designers is dubious.
I was a little surprised by Eric's remarks, too, not because I disagreed with the sentiment but because I didn't know why he'd felt obliged to say it. Indeed, I felt that at times that I was fighting a rearguard action on behalf of designers, rather than bullying researchers to toe the line. I suppose it's just a matter of perception.
Richard
Posted by: Richard Bartle | Dec 11, 2004 at 14:32
TL said, "Are there ways we are fetishizing market-focused design which is concerned with that very narrow branch of the commercial MMOG world?"
Now, I'm not an academic, but I'm curious -- what would you say the corresponding very broad part is to the "very narrow branch of the commercial MMOG world"? Is there a universe of non-commercial MMOGs that dwarfs the commercial part? (I assume you're not talking about the myriad tiny hobbyist MUDs and such.)
While I agree with the idea that not all research should be constrained to being immediately applicable -- this is far too short a horizon for real research or innovation -- the fact is that MMOGs exist because they are commercially viable. MMOGs are a commercial creation. It's not surprising then that for research to be relevant it is typically viewed through a commercial lens.
Perhaps you could provide a couple examples of research that you believe is important (and why) and yet is disconnected from the commercial aspect of MMOGs?
Posted by: Mike Sellers | Dec 11, 2004 at 15:24
Just a quick reply (I want to hold off on any substantial comments until more hit the discussion) - I really didn't mean to imply anything about the quality of the conference or the papers there. It was in no way some kind of GDC junior ;) Jonas and Miguel, not to mention all the people who turned up (both audience and presenters) did a great job and I really enjoyed myself. I think the discussion just arose naturally in the face of many of us sorting through our own work, the relation to the field, and our institutional positions. One other quick point, I have certainly done my share of trying to provide "design implications" so it's not as if I'm outside of the system I am talking about. My apologies if the tone of the post sounded at all like finger pointing. It is meant to open some discussion.
Posted by: TL | Dec 11, 2004 at 15:43
Science was always about legitimacy and ways of gaining it. There are basically two ways of gaining it, either through recognition or economy. We don’t gain recognition or economy by shutting down the doors or start thinking how to illegitimate a research field.
I understand the concerns of T.L., however I would like to reverse the problem.
How many Game Researchers are actually collaborating with the industry? The numbers are quite few. So how big an impact does the research actually have? Little, if any.
So here we are studying something, which basically doesn’t do a lot whole lot of difference, yet. We can study ourselves to death.
But in the end we will have to prove our legitimacy in front of university boards, external funding partners, industry, politicians and so forth.
The IT-University, which hosted the Other Players Conference, was established because there was a demand for qualified IT-candidates in Denmark. The Danish game research is gradually growing as a field because politicians are realizing that there is a huge economical potential in this field. So how do we prove our legitimacy?
One answer is to do innovative research. However this is not happening as most of the significant innovation comes from the industry. So how do we prove our legitimacy?
Another answer is to produce candidates with the right competencies. The right competences is something defined by the surrounding world, not us. The surrounding world, especially the game industry, is screaming for people with solid knowledge on various aspects of games. While also screaming for people with applicable levels of knowledge.
One of the frequent arguments in this debate is that the research must also think 10 years ahead of now... My answer is: I don’t think they will be applying less in the future.
Posted by: Troels B. Folmann | Dec 11, 2004 at 18:06
TL> I really didn't mean to imply anything about the quality of the conference or the papers there.
Yep, I absolutely know you didn't. I just wanted to voice the requisite hedge since those folks who couldn't attend the conference might get a different idea.
Posted by: Ian Bogost | Dec 11, 2004 at 19:51
Mike Sellers asked: I'm curious -- what would you say the corresponding very broad part is to the "very narrow branch of the commercial MMOG world"? Is there a universe of non-commercial MMOGs that dwarfs the commercial part?
The universe of virtual worlds certainly does extend beyond commercial MMOGs to include worlds used by educational institutions and the military, to name a few. However I wouldn't say these non-commercial worlds dwarf the commercial worlds. The commercial worlds still dwarf the non-commercial these days and in all likelihood this will continue to be the case. But as virtual worlds become more common I expect we'll see growth in all sectors of the virtual world universe, giving academics plenty of research material. I do hope that many will choose to focus their attention on lesser-known non-commercial worlds no matter how the proportions of world types shift within the vw universe in the future.
Posted by: Betsy Book | Dec 12, 2004 at 09:48
> Mike Sellers sez: ...what would you say the corresponding very broad part is to the "very narrow branch of the commercial MMOG world"? ....Perhaps you could provide a couple examples of research that you believe is important (and why) and yet is disconnected from the commercial aspect of MMOGs?
***
Hoho. You mean sociology, social psychology, psychology, cognitive science, philosophy of mind... random stuff like that?
My assumption would be that there is a universe of non-commercial human behavior that dwarfs the commercial part -- and, further, that the study of "commercial worlds" is a small subset of the study of human beings.
From this point of view, it seems reasonable to describe (non-pejoratively) the study of the commercial aspects of MMOGs as narrow.
Posted by: dmyers | Dec 12, 2004 at 14:39
Dmyers said, "My assumption would be that there is a universe of non-commercial human behavior that dwarfs the commercial part -- and, further, that the study of "commercial worlds" is a small subset of the study of human beings."
Yes, but that's not what TL said, nor does it answer either of my questions. TL was talking about research involving MMOGs, not research into the entire spectrum of human behavior.
There seems to be an underlying assumption in what TL said that commercial MMOGs are just the tip of some iceberg -- but what remains unanswered is what is the rest of the iceberg? While there are innumerable text, VRML, and other small online worlds, I don't believe these (including those made for the military, which almost invariably have a commercial component) can be said in any way to be broader than the world of commercial MMOGs (and this is even with the current in-breeding of MMOG genre and style).
So I'm still not sure in what way current MMOGs are a "very narrow branch" of the MMOG world (commercial or non-commercial). If there isn't a broader MMOG world out there today, then issues of industrial legitimacy and relevance in research are more pointed. MMOG research doesn't have a broad non-commercial MMOG world to fall back on. Rather than turning away from this fact, I think academic researchers would be better off working more closely with industry, while understanding that the two groups have different, hopefully complementary, goals.
I am very interested in hearing examples of research that MMOG researchers believe to be important (and why) -- especially if it demonstrates important research that is disconnected from the commercial aspect of MMOGs.
Posted by: Mike Sellers | Dec 13, 2004 at 09:43
Not only are there lots of us out here interested in non-commercial issues revolving around social psych, etc., but some of us are interested in design as a causal variable.
Mike asks for examples, so here's one from my work: Do different guild or chat systems lead to different kinds of real-life connections between demographic groups who would normally not interact? Would the firm care? Probably not, but I would, and so would many comm scholars and political scientists.
So I'm not out to find or recommend design improvements to the industry as my main goal, but I still want to know what causes what, and design issues matter in ways they never really did for TV. Us media studies types are adapting to the new environment. Thus, I'm happy to work with firms on design issues even while we have different end goals: the uses and effects of them for me and the viability/retention/usability issues for them. It's easy to piggyback one on top of the other. Everybody wins. So if an MMRPG developer wants a low-cost set of answers to design and usability questions, they get high-quality data and analysis and I get material to publish articles and make progress towards tenure.
Posted by: Dmitri Williams | Dec 13, 2004 at 12:01
Dmitri gives a good piggy-back example, though I have to admit my first impression is always that commercially oriented research is, at least initially, rather counter-productive to broader theoretical concerns.
First of all, if the research really is commercially oriented, then it's probably proprietary or non-disclosed or some such anti-information thing.
Second (as I'm buzzing through TN) I read this...
*
The design-premises of MMOs are somewhat dysfunctional:
1. slow down the user, prevent the user from acquireing the content-space
2. make the user depend on others
Posted by: Ola Fosheim Grøstad | December 13, 2004 07:56 AM
*
...which I quite agree with.
These are design premises, I might argue, that have more commercial than theoretical justification. I would hate to villainously characterize all commercially oriented research as attempting to discover the most palatable mechanism for muting, distorting, or other muzzling free and individual play. ...but there you go.
And a third example: the Bartle player categories. Now, I will obviously bow to Richard's own account regarding this matter, but my recollection is that the original article describing these categories (explorer, socializer, etc.) was a piece of designer-motivated research trying to find a way to properly BALANCE player populations, i. e. solve a practical designer problem (of more than a little concern to commercial designers).
Nevertheless, the major contribution of the article, in my mind, is not its description of any commercially viable design innovation, but its description of genres of human play. That is, these categories are connected to play and to people, I would suggest, much more significantly and much more fundamentally than they are connected specifically to MUDs, or, even more narrowly perhaps, to the commercial world of MMOs.
Thus, any research -- whether aimed at commercially orentied MMOs or not -- will be most valued and valuable when, despite its aim, it hits a larger target.
I think that was what T.L. might have been getting at, and what Dmitri implies as well: It's just not enough to hit the commercially oriented MMOs designers' targets. Why? Well, one way of putting it is that those targets are too narrow.
Posted by: dmyers | Dec 13, 2004 at 13:20
Great to get some feedback on this stuff, very interesting comments from everyone. Weirdly enough it seems I posted right about the same time related conversations were springing up elsewhere in the blog world. Not sure what that means (is something in the air?) but I think it's worth pointing to the other discussions. Right now one of the most interesting is happening over at Intelligent Artifice (http://www.intelligent-artifice.com/2004/12/on_academia.html), where you should definitely check out the comments as well. There is also a post at Last Castle (http://mythical.blogspot.com/2004/12/last-castle.html) though I admit being pretty baffled by my post (heh, aka 'The Aristocracy' no less!) being characterized as as "Why won't anyone ever listen to us" given that I wasn't at all trying to make (rehash?) the old claim that if designers just read everything academics wrote the world would be fine. Which leads me a bit to a reply here.
I appreciate Mike's query that I provide some examples of research I think is important but disconnected from commerical aspects of MMOGs because my sense is he is one of those folks who really does value academic work in a variety of forms. But I was more intending to turn the discussion of 'usefulness' on its head a bit. I think 1) we don't always know what is useful (and that is not a bad thing), 2) we don't always know what /may/ be useful, and probably most important in this context, 3) sometimes 'useful' is far too narrowly defined. For example, one thing that might be 'useful' for a commercial world is a feature that produces more subscribers. But that may be 1) a fairly limited orientation for interesting design in a broader sense (I'd love to hear a designer chime in on some of this - for example, does hanging on "massive" produce some unfortunate design limitations?), 2) not particularly compelling from a research perspective, or 3) critically problematic. My concern was that we don't simply value research for its apparent 'usefulness,' but acknowledge the core value in work that is basic, exploratory, and maybe even produces findings that are oppositional to commercial (or other?) values.
But to add on another layer, I worry about the model of influence that unpins some of what I hear in Troels comments. If I read him right he is concerned not more academics collaborate with industry (and I just want to note again that I was trying to deliniate between industry and design because I think the conflation is a dangerous one and something I'm really trying to watch myself) and that our primary mode of impact is through innovative research. I just wonder which measuring stick we use to define "innovative research" and again, when you simply tie that back to industry collaboration you may be setting things up a bit. While it is certainly the case that some scholars will have this kind of direct impact, I think we should be careful to imagine it's the only, or primary, model. He also suggested that "Another answer is to produce candidates with the right competencies. The right competences is something defined by the surrounding world [who he in part defines as the industry], not us" and while that sentiment certainly matches part of my own in that it is our students that act as the main conduits of scholarship, I'm concerned that he is conceeding too much ground about who decides what is deemed good or useful. It's an old saw of course, but I still see the university as playing a major role in helping intervene and shape what is defined as valuable. The debate of course wouldn't be so animated for me if it weren't for the fact that there are serious moves being made (across multiple countries as far as I can tell) to require scholars to legitimate themselves primarily through what they contribute to industry in very simple immediately 'useful' ways. Again, I'm in no way criticizing those talents folks that blend their research/theory with their own design work, nor suggesting that working with industry is evil (please reread as I don't want that point lost), but it is only one model. As I mentioned in the post, while it'd be great if designers didn't impose the burden of applied immediacy, I'm really just as concerned about when we as scholars feel we have to come up with the quick 'how this can be used' formulation.
One last thing, I can understand Ian not wanting to create stratifications in the game studies community about who does 'real' this or that - it certainly isn't my intention either. I'm less convinced, however, that the way we avoid it is by not talking about these issues. I think if the very lively discussion at OP was any indication, many of us are really trying to sort through the ways these components relate.
(On a sidenote: I find myself thinking a lot about Helen's comments (over at IA) when she says she doesn't want to see us reify the old "thinkers" and "doers" dichtomy. While my inclination is to agree, I think there is also a more radical proposal (vs just taking those categories whole & intact and mashing them together) when she writes "I would want to argue for fluidity of transfer of knowledge/expertise/experience even when we dont really realise it is happening." I'd be curious if anyone has thoughts on that.)
Posted by: TL | Dec 14, 2004 at 12:34
Sorry, the title of the blog I named as Last Castle is Mythical. Also, IA notes a thread at GTA that I agree is worth checking out if you are interested in the subject - http://grandtextauto.gatech.edu/2004/11/20/keeping-it-real/ (and as usual, lots of good comments there).
Posted by: TL | Dec 14, 2004 at 12:38
Okay, I know I've now officially spammed this thread but there is just one more URL I want to toss in here, Damion Schubert's Zen of Design entry on the subject - http://booboo.phpwebhosting.com/~ubiq/index.php?p=162
Posted by: TL | Dec 14, 2004 at 12:48
TL, I agree with you about the overall usefulness of academic research to industry: if we knew what was going to be useful from the start, it wouldn't be research!
In this light I see MMOG reseach (whether academic or industrial, pure or applied) as sort of 'scouting' around the main body of where the industry is currently heading. Maybe someone will discover a better path one direction or another up ahead, or maybe someone off to the side will discover something entirely different that's useful in, say, traffic simulation or effective crop growth. Who knows?
But as far as MMOG research itself is concerned, I don't really see a way to get away from the fact that these are commercial artifacts, and so studying them is always going to make reference to the commercial body that forms the center of gravity of this area. Given that, questions of relevance and more or less immediate utility are always going to crop up.
Posted by: Mike Sellers | Dec 14, 2004 at 13:41
Sorry for the mischaracterization, TL. Updated with:
Update: 'Why won't anyone ever listen to us' unfairly characterizes the post by T.L. Taylor. His post is more to say (in very short form):
A lot of the work we do can certainly be used to retain players or find new demographics for games but I believe we have to retain a space for other kinds of work - often basic, often critical - to exist.
...which it is my intention to entirely agree with. Or more to the point: I don't feel it is the responsibility of academics to find the immediate applicability of their research, but rather that responsibility rests with the developers.
If we can't find the applicability, then so be it. That's our problem, not theirs.
'Reason for linking to that post in the first place is that - as I see it - it is a reaction to the pressure from without and within the academic community to justify their research, to "be more relevant". That is, to "get developers to listen to us".
Posted by: Jeff Freeman | Dec 14, 2004 at 17:22
This might be blunt but... social scientists have no business making design recommendations based on identified flaws. Suggesting a new avenue of design-oriented research would be o'so'much better...
What totally confuse me are papers that provide guidelines based on identified flaws! Heavens!! Guidelines are supposed to be based on what works! :-( I also hate all the design-oriented speculations. Got a new idea? Implement it and test it for f'godssake.
Posted by: Ola Fosheim Grøstad | Dec 14, 2004 at 18:49
Yes! T.L! I thought you ignored my poor attempt to light the fire to this debate! Thx.
Please note that I have no interest in evaluating or measuring different types of research. My comments should primarily be regarded in a political and economical context.
Allow me to elaborate on your reflections:
: )))))))
"...I worry about the model of influence that unpins some of what I hear in Troels comments. If I read him right he is concerned not more academics collaborate with industry..."
Yes and no. What worries me is when academics starts rambling about the ignorant industry, hollywood invasion and other xenophobic issues. I believe game studies suffers from lack of collaboration with the world outside the walls.
I have been involved in very large game productions and its been extremely valuable lessons for me - both as a researcher and composer. I have also been very fortunate to meet highly reflective people in the industry. I have learned equally as much from both worlds (academia and industry) and can only urge design oriented research to get their hands into the real deal. However not all research is design oriented, naturally. Nor should it be.
"...I just wonder which measuring stick we use to define "innovative research" and again, when you simply tie that back to industry collaboration you may be setting things up a bit..."
I believe almost all innovation - in games - comes from the industry. There is alot of interesting studies going on within game studies, however the impact is hardly measurable, since only few researchers are collaborating. Not all research is measurable, but parts of it should be - which is another reason to underline the need for more collaboration.
Its not because I advocate for measurement. But because I know that we are being measured by boards, politicians and the other systems keeping us alive. We can ignore these... For a while...
"...While it is certainly the case that some scholars will have this kind of direct impact, I think we should be careful to imagine it's the only, or primary, model..."
I totally agree. But the current problem is that we don't really have any clear examples of any scholars doing direct impact research. How are we gonna explain ourselves to boards, funds, politicians? How?
"...He also suggested that "Another answer is to produce candidates with the right competencies. The right competences is something defined by the surrounding world [who he in part defines as the industry], not us" and while that sentiment certainly matches part of my own in that it is our students that act as the main conduits of scholarship. I'm concerned that he is conceeding too much ground about who decides what is deemed good or useful..."
Universities come and go. We are currently seeing a trend (at least in Denmark) where researchers within humanities and social sciences are facing problems. Risoe National Labotory fired 70 people some weeks ago and University of Copenhagen fired 25. Why did this happen? One of the primary arguments was that both institutions are facing new challenges in order to innovate and collaborate more. Its also interesting to notice that the public debate in Denmark about the need for private universities is pretty hectic.
We can debate all these issues for years, but the fact is that the landscape of research is changing. The political demands are obviously going towards a more applied oriented types of research - and I fully comply with this concept. Naturally there will be winners and losers in this game. The winners will adapt and the losers will be glued to their xenophobic principles of non-applicable irrelevant research.
Research was always about legitimacy - and legitimacy is about recognition and/or money.
How do we gain this without collaborating with the world that keeps us alive? That is the main question - and it stills stands unanswered in this debate.
Hope my second attempt will scorch a bit - and let me state again ... not all research is design oriented, naturally. Nor should it be.
Posted by: Troels B. Folmann | Dec 14, 2004 at 19:01
Additionally allow me to elaborate on why the Risoe National Laboraty recently fired 10% of its staff (70-75 people) - here is the official argument (translated):
"...The administrating CEO at Risoe, Joergen Kjems, recently stated that Risoe fired 10% of its staff consisting of 750 people...
Risoe will start new research projects with the money saved. The new projects must promote the competive ability of the Danish industry.
The idea is that all future research must be better at creating new products and/or new companies..."
I am not saying I like this. But it clearly indicates a trend. Whether we like it or not.
from http://www.danmarks.net/nyhed.phtml?idxno=1472
Posted by: Troels B. Folmann | Dec 14, 2004 at 19:11
I think design-oriented research is best left to Industry, and Academia ought not concern itself with being 'immediately applicable'.
Commercial research is an investment, and yeh, it ought to be measured in terms of ROI. Produce, or get out (it also tends to build on academic research, too, doesn't it?).
Make a new blog called 'Terra Firma' and put that there.
Academic research is a different kettle of fish, though, don't you think?
Posted by: Jeff Freeman | Dec 14, 2004 at 19:26
"...I think design-oriented research is best left to Industry, and Academia ought not concern itself with being 'immediately applicable'..."
Oh? So design oriented research is not "real research" in the academical sense? Where is the line between design and academia? Academia cannot be immediately applicable? We don't have actual needs and problems to solve?
Please elaborate...
I would like to see your definition of academia.
: )
Posted by: Troels B. Folmann | Dec 14, 2004 at 20:17
LOL I guess Jeff Freeman only include the humanities/sociologists in Academia. Oh, I weep for the poor souls in the applied sciences (comp sci, HCI, information systems etc)...
Posted by: Ola Fosheim Grøstad | Dec 15, 2004 at 06:30
Research done for its own sake, as opposed to research done with the requirement that it be immediately applicable.
Academic research can be immediately applicable, but I don't think that should be a requirement.
Whereas, if a company is paying for the research, then I think there's nothing wrong with them demanding immediate results (if that's what thay want).
And I'm just talking about game design. What I meant was that the industry ought to do the research which is required to have an immediate practical application and academia ought not worry about it.
Glad I could give you two a chuckle though. REALLY.
Not now, you wouldn't.
Posted by: Jeff Freeman | Dec 15, 2004 at 11:14
Yeah. I was more thinking that Terranova and most MUD research aren't representative of academia. Those are far out in the soft meadows... Academic research can at best provide tools for game developers which they can use if they want to, it's not their business to optimize a paradigme, do free consulting or tell game developers what to do. Besides, your average hobbyist have more insight than your typical researcher anway.
Posted by: Ola Fosheim Grøstad | Dec 15, 2004 at 11:31
Hehehehe. This thread is gradually becoming more entertaining. Thx to T.L., Jeff and Ola...
Let me first state that "research for its own sake" is precisely what research should'nt be about. It ends up being an introvert product, which nobody (but the researcher and his poor family) can barely comprehend.
Allow me invent a few topics like that:
"...Marxistic discourse analysis of player behaviour in economic oriented games..."
"...An analysis of postmodernistic game design traditions..."
"...A Luhmann approach to games - understanding autopoesis of sexual relationships in MMORPG..."
Oh god. I could keep pumping them out. Let me know if you need more of these. I coúld write a program, which randomly generated topics like this. Such a program is likely to better then the outcome of the research.
: )))))))
However I think that Ola stretches it a bit too far by stating:
"...your average hobbyist have more insight than your typical researcher anway.
Insight in what? Game mechanics? Game Play? The average hobbyist is likely to be claiming that the type of game he loves is what all games should be like. However without knowing that pornographic games would really be the thing he had been waiting for for decades.
Merry X
Posted by: Troels B. Folmann | Dec 15, 2004 at 12:31
TL wrote:
>There seems to be at least three levels at play here
>what I might call critical game studies, critical or
>innovative design work, and market-driven design work.
>It seems to me that more and more "speaking to design"
>or designers is conflated to the third category
>addressing market concerns (the need for more
>eyeballs, more subscribers, bigger games, etc).
I think this dichotomy and between and rhetoric of "critical" vs. "commercial" contains many pitfalls.
First of all, I don't see any general move towards academia seeking commercial design legitimacy.
Secondly, I think the early years of academic game projects student projects and experimentations were marred by a kind of avant-garde or critical fetish.
I have seen many students who were encouraged to try making the game equivalent of Finnegan's Wake before they had learned to make even a clone of Space Invaders. This not only meant that most of the stuff produced had no commercial value, it also meant that it was simply bad work by any standard.
The dichotomy between critical and commercial additionally makes it easy to claim that if anything you have written or designed has no commercial potential, this must indicate that your work is critical and based on some deeper understanding of the world, the medium, and so on.
I think there is simply more to it than that - I think all good experiments have to understand the mainstream well and vice versa. There is no "us" (the good critical people) vs. "them" (the mindless industry who thinks only of money).
Posted by: Jesper | Dec 15, 2004 at 14:42
Troels> Insight in what? Game mechanics? Game Play? The average hobbyist is likely to be claiming that the type of game he loves is what all games should be like.
The overall effect of design choices.
I meant design-hobbyist, not game-player. More specifically, ideas that have already been suggested and beaten to death 3-7 years ago on MUD-Dev crop up as research or as "new ideas", but with no better empirical grounding... It is odd.
Merry Xmas anyway.
Posted by: Ola Fosheim Grøstad | Dec 15, 2004 at 14:43
Jesper wrote:
"...There is no "us" (the good critical people) vs. "them" (the mindless industry who thinks only of money)..."
Totally agree. Allow me to experiment with your idea and reverse it. It would sound something like: "...there IS a good and academic-critical industry and a bunch of mindless researchers that only thinks about staying alive..."
: )
But thats not the real issue. The issue is the coupling between the worlds.
If we - as academics - really have such a great and qualified insight and a critical and senseful bulk of knowledge - by all means lets get it out. Unless ofcourse - our public funded research is a secret.
Ola wrote:
"...I meant design-hobbyist, not game-player. More specifically, ideas that have already been suggested and beaten to death 3-7 years ago on MUD-Dev crop up as research or as "new ideas", but with no better empirical grounding... It is odd..."
Thx for clarifying. I could not agree more. I think this is one of the reasons that we don't see many major collaborations yet.
Posted by: Troels B. Folmann | Dec 15, 2004 at 16:24
Thanks for the clarification/context Jeff :)
Jesper wrote, "I think there is simply more to it than that - I think all good experiments have to understand the mainstream well and vice versa. There is no 'us' (the good critical people) vs. 'them' (the mindless industry who thinks only of money)."
I appreciate the chance to clarify because I absolutely don't want to convey the impression that I think those in the industry are simply mindless, uninteresting, unthoughtful... even uncritical. There are certainly those doing work in commerical arenas that produce interesting stuff (and I find just as important, may want to do innovative work but find themselves unable to really carry out their ideas given the structure of their organizations). Let me also be clear too, I also find it frustrating when academics come along and, after playing a handful of games or reading blogs for a year or two, presume to speak as if from on high about what would work in this or that MMOG. Despite the bite in Bartle's previous 'you've now reached MUD-Dev circa 1995' (or whatever year it was) comments I couldn't help but smile because it is certainly the case that far too many people think this stuff has never been done/talked about before. I think this is a problem though not just with game scholars but even some game designers who overlook the body of knowledge built up over the years on virtual worlds, internet research, CSCW, HCI... and work in traditional disciplines, etc. So there are certainly academics who produce work that is uncritical and I would not make the equation Jesper is worried about (that non-design = critical).
What I was trying to foreground with some of my examples, however, is that the commercial arena at a very basic level looks to the market first and foremost. If something won't make money (or often just as important, is /thought/ that it won't make money), will put the ownership or brand into question, would cost too much to implement/revamp, etc. it often gets shelved. What I want to suggest is that the /structures/ we operate in have profound impacts and have their own internal models of value. We have had a lot of good work around gender and race for years now and frankly, I've yet to see much of it really taken to heart by the commercial sector. What is even more devastating is that despite potentially having arguments to justify innovation in those sectors /in market terms/ it remains entrenched. What I'm then saying is that, for example, if you do innovative scholarly work on subjects like that (or IP, or fan communities, or etc), 1) just don't hold your breath it will be really taken to heart in commercial design in any immediate way and 2) let's make sure we still have ways to talk about, and /structurally acknowledge/, the value of that kind of work.
Posted by: TL | Dec 16, 2004 at 07:43
Sorry, just thinking about it a bit more and to follow through: Taking this idea that critical commercial work can occur (and are there limits? what are the deal-breaker areas?), what kinds of structures need to be in place to really foster it? Is it a matter of game companies starting to integrate basic research labs into their organizations? Is it an issue of diversifying the workforce? Just curious if people have thoughts on the issue. I know of handfuls of anthropologists who have over the years been hired into research labs at places like Sapient, Intel, and of course Parc and I wonder, is that a kind of future model for game companies? And if that feels completely unrealistic, is it tied to other structural constraints built into the developer-publisher-product cycle model?
Posted by: TL | Dec 16, 2004 at 08:00
TL>I know of handfuls of anthropologists who have over the years been hired into research labs at places like Sapient, Intel, and of course Parc and I wonder, is that a kind of future model for game companies?
Big game companies such as Sony will tend to put their blue sky money into universities rather than their own labs (although sadly not into my university).
As for game-friendly work in other research companies, look no further than Nate at BBN.
Richard
Posted by: Richard Bartle | Dec 16, 2004 at 08:10
a few more cents...
In my other life I work on the sociology of science (mostly physics and chemistry)... I hang out with scientists and even get to do some lab work but we would not dare (even if we might dream) of advising scientists on how to do their work let alone pretending to be able to do the work ourselves. There are a number of reasons for this but i'll mention just one that tends to be less obvious.
It is a methodological point specific to just one brand of social science... a science like physics works as a culture that produces knowledge (whether we like how it works or what knowledge it produces does not really matter) and the goal of the sociology of physics is to understand how the science works the way it does... to do this we need to be careful not to substitute our own analytical frame of reference for those of the physicists.
Note that this is not a divide between those who "do" the work (the physicists) and those who "understand" the work that is done (the sociologists). Physicists (especially) have very well developed analytic frames for understanding the hows and whys of what they do... they are their own theorists if you like (just as designers develop their own theories of design, and players develop their own theories of play). Where does that leave the sociologist? The task is to make sense of what the physicist/designer/gamer does as well as how they understand what they do.
That is, as a games researcher I want to try and generate some analytic distance between myself and the gamers/designers I study... In this mode, I don't want to do design or even advise or consult; nor should my work be accountable to designers or players. To be specific if game design culture doesn't employ concepts from formal semiotics then I really don't want them to even if I personally think the concepts would lead to better games (which I also enjoy playing).
With this in mind one goal of academic research on games would be to differentiate between actors' categories (the way gamers and designers understand their games - the objects of their social worlds) and analysts categories (the way academics understand the actor's catories).
Of course, this is too simple and game studies is too undifferentiated to accomplish this easily (even if we wanted to)... but perhaps it is one point of departure for pondering the value of a 'distinct but equal' approach to the relation of academic research, design and play in game studies.
- Bart
Posted by: Bart Simon | Dec 16, 2004 at 11:16
Weighing in a little late here. I think the context under which research funds are handed out is part of what drives the sense that research needs to be industry relevant. In Australia at least, anti-intellectualism is rife, and the need to justify one's research in terms of industry relevance pervades much of the discussion about academe.
There has been a breakout of discussion about this recently on the Australian cultural studies list, in response to yet another attack on humanities projects by journalists in the popular press. These journalists are very fond of holding up projects for ridicule and claiming the state is wasting its money funding them. In response, Graham Turner has written an interesting article which says in part:
"Every year at about this time, the Australian Research Council releases the results of its Discovery Projects; every year they are greeted by a customary attack from some quarters, keen to opine about the daftness of academe. This year was no different. According to some, this kind of stuff is just self-indulgent radicalism or the whim of fashion, with no possibility of a return to the community. ...
Last year, for instance, David English and Andrew Bolt laughed at a study of mobile phone culture by my colleague Gerard Goggin, characterising it as an example of the pursuit of self-indulgent theories and neo-Marxist fancies.
The mobile phone industry disagreed: the Australian Mobile Telecommunications Association -- the industry's peak body -- has since joined Australia's leading social scientists in a world first to develop a research agenda into the social and cultural impact of mobile phones. ...
You do not build a national research effort that is comprehensive and effective by responding to fashion or to a punditry that falls back on uninformed anti-intellectualism. ...
When we abandon process in favour of prejudice (or politics) to guide research decisions, the results are predictable. We have been badly served in the past by those outside the universities who think they know what is worthwhile research and what isn't. Too often they have failed to understand the short-term and partisan nature of their own interests. ...
An example of the consequences is our present incapacity to provide much in the way of expertise to help us understand the cultures of Islam. Australia's collective ignorance about Islam is significantly the result of successive governments' reluctance to fund teaching and research in then unfashionable languages and cultures. Our capacity in Islamic studies is so diminished that we are hard-pressed even to find suitable readers to assess new projects that aim to rebuild our knowledge."
Sorry for the excessively long quotes, but I think Turner makes useful points about long and short term benefits of research, about our inability to predict what will be useful in the future and the need to not fall for short term applicability as the guide to what should be researched.
A couple of other things I have been wondering about in relation to this thread are 1) the focus on industry as designers tends to erase the role of players as co-creators and innovators, so we limit our discussions by using that as a framework, and 2) I don't think my research will ever be industry relevant, but it may be policy relevant - there are more 'real world' relevancies than those embodied by industry.
Posted by: Sal Humphreys | Dec 16, 2004 at 22:15
Sal Humphreys>the focus on industry as designers tends to erase the role of players as co-creators and innovators
But the designers have to design the game in order for the players to be co-creators and innovators. A game intended for players to be co-creators but which was badly designed would stop the players from creating much, no matte rhow creative they are. In other words, co-creation doesn't just happen, it happens because the designers designed for it to happen.
Richard
Posted by: Richard Bartle | Dec 17, 2004 at 14:37
Some of the most well balanced games were actually done in collaboration with players - both during conceptual design, alpha and beta stages of testing. Bill Fulton is a great example. In cause you don't know the guy:
"...Bill Fulton was a founder of the User-Testing Group for Microsoft Games, which uses psychological research methods to get feedback that improves the usability and fun of games published by Microsoft. Since 1998, the group has tested 12,000+ gamers playing 80+ different games..."
In other words - the design of games should, ideally, involve collaboration between players and designers. AOE1-2, AOM, MS FS and other great games all went through this proces and it clearly shows on the quality. Both the alpha and beta test stages of WOW is another clear sign of this.
Its interesting to notice how Blizzard turned beta-testing into a commercial venture - as over 300.000 users already bought the game - in order to participate in final beta testing and later aquire the final release.
http://www.gamasutra.com/features/20020911/mclean_01.htm
Posted by: Troels B. Folmann | Dec 19, 2004 at 05:14
I think suspect it's running both ways to some degree in that designers like the idea of being someone's scholarly interes while scholars want to feel they're affecting someone's ideas in some positive way and that we're not just hiding in our ivy-covered closets playing Halo but pretending it's serious stuff. There is also a need to feel there is appropriate academic venue for our work as well as for our games. It's not all about anxiety, but there certainly is quite a bit of it in the mix. Of course, we're also quite fortunate in that we're endeavoring as much as possible to have our cake and gobble it down too . . . in that we want the legitimacy of the academic while the absolute pleasure of making games our area of study. Come on, how cool is that? We don't actually have to grow up, we just need to pretend we have in that instead of just playing games, we're learning about all sorts of neat concepts regarding the human condition while playing those games.
Posted by: Brian David Phillips, PhD, CH | Dec 19, 2004 at 09:37
>Troels wrote: Let me first state that "research for its own sake" is precisely what research should'nt be about. It ends up being an introvert product, which nobody (but the researcher and his poor family) can barely comprehend...
You seem to be focusing on two polarities without considering areas of academic research that might not be in immediate demand by industry but which could have significant repercussions in the development of MMOGs in an indirect fashion without ascribing to the rather specific examples you mention.
If research follows industry's immediate concerns it misses out on what I view as the most significant contribution academia can make to the field, which is, an informed yet detached perspective that asks questions which dig deeper into our relationship with virtual worlds. Questions which industry might not have the budget or long-term insight to ask.
Posted by: Gordon Calleja | Dec 19, 2004 at 17:13
the most significant contribution academia can make to the field [is] an informed yet detached perspective that asks questions which dig deeper into our relationship with virtual worlds
Good point. As TL said earlier, I really am interested in what such research can bring us. But I'm also skeptical of the value of something just because it's called "research."
So: what are some of the specifics of the more important informed yet detached perspectives about our relationships with (and within) virtual worlds that have been gained through non-industrial research?
I'm not at all being snide by asking this. I'm hoping that someone will chime in with a strong list of issues and ideas that have been so addressed; or alternatively, to hold the research community's feet to the fire: for this to be an area of important, relevant, legitimate research, there must be some fruit to show for it.
Posted by: Mike Sellers | Dec 19, 2004 at 17:29
Mike Sellers> I'm hoping that someone will chime in with a strong list of issues and ideas that have been so addressed;
In my opinion there is no such list. The main problem being that good descriptive accounts are undervalued, instead you get "theoretical" ideas that are either obvious, speculative, arbitrary-so-what, or just difficult to follow (the link to the data is up-in-the-blue). This is of course not unique for virtual world/games research, but ATM quite dominating.
For my part the list is quite short, and I value these more for the descriptions than the theoretical ideas:
1. Turkle for pointing out how complex and personal the motivation for playing MUDs can be. I.e. forget general motivation models for MMOs.
2. Suler for providing a number of perspective on a simple system like the Palace and making the link to transference.
If you want more design-oriented research then the list gets a bit longer, but not much :P.
Posted by: Ola Fosheim Grøstad | Dec 19, 2004 at 19:36
Hi,
Just a quick intro as I think our situatedness as researchers and industry practitioners is very important to this issue.
I recently submitted a PhD (University of Queensland, Australia) - an ethnographic study of the relationship between a game developer (Auran) and an online fan community. For much of the research (since 2000) I've also been employed by Auran (a medium size game developer) as online community relations manager, and more recently as Director of Online Communications.
Managing this relationship as a researcher/worker at Auran has been challenging and interesting. It most certainly has been a collaboration aimed at assisting Auran with improving their practices of community management. However, the dissertation also has research agendas independant of Auran that Auran has been supportive of. The research questions that I've been exploring include corporate governance of online fan community and the emerging shape of distributed production networks that enlist the labour of fans. This work is quite critical of current industry practice. So I'm not sure that the question of these collaborations being an opposition between independant research and say more commercially driven outcomes is necessarily the most useful way of framing the discussion.
I think it is at least arguable that the network of relations being explored between researchers and corporations is far more nuanced and complex than such a framework suggests. There are often overlaps, connections and disconnections between these different moments of research (the commercial and the noncomercial?). I've found that interesting and challenging research questions have emerged as I've been caught up in very bottom-line driven corporate strategies and projects. Commercial imperatives have also at times disrupted the research in uncomfortable ways.
My collaboration with Auran as a researcher has extended or perhaps the network has lengthened to now include Auran as an industry partner with an Australian Cooperative Research Centre (ACID - Australasian CRC for Interaction Design). This involves Auran collaborating with researchers and other industry partners across a range of projects including Virtual Heritage and Massively Multiuser Online Environments. The research is genuinely multidisciplinary involving people such as myself with cultural studies/media studies ethnographic methodology backgrounds, IT specialists and human factors/interaction design. It all makes for an interesting mix, as does the commercial and non-commercial research agendas.
These relationships between the commercial industry partners and the academic researchers must be carefully negotiated and renegotiated throughout the stages of the various projects. However, this is a research space that I enjoy occupying. Of course it is not the only way to undertake research and nor should it be. But I will continue to explore the possibilities of these research relationships next year as part of a Postdoc that continues to involve Auran as a key partner. Innovative research and commercial outcomes can both result from these relationships.
I believe that corporations are occasionally places where challenging and critical research can be written. After all much of my PhD research was undertaken and written from a desk here at Auran. Since 2000 I've spent far more time situated in industry then in academia. Or perhaps a more interesting way to consider it, situated at the intersection of industry and academia.
I'm interested in exploring ways of reframing these industry - researcher relationships in langauge other than oppositions between independant researcher and the commercial. This may also involve something of a rethinking about what research is in the context of these collaborative relationships.
This is a great discussion and I hope it continues.
Cheers,
-jb
John Banks
Posted by: John Banks | Dec 22, 2004 at 00:05
Does this mean that you have done or are planning to do "action research"? I.e. trying to study the organization/system by changing the practices?
Posted by: Ola Fosheim Grøstad | Dec 22, 2004 at 05:38
Hey John. Great to hear you chime in here. When I think of folks working in that hybrid space, you definitely come to mind as someone doing it in interesting ways ;) So I'm curious, what do you think is required (if at all) structurally, culturally, whatever, to have the kind of critical commercial work you see possible happen? Can you provide any good examples of those moments when researcher & company were unsettled by the collaboration? How it goes resolved? etc? And maybe as a follow-up, how typical is Auran?
Posted by: TL | Dec 22, 2004 at 05:59
Hi Ola,
I actually only belatedly discovered that my research is very close to an action research methodology. Throughout the PhD I largely worked from a cultural studies ethnographic/audience research methodology. I also draw extensively on actor-network theory (Bruno Latour etc). However more recently, after collaborating with researchers from human factors, interaction design and HCI backgrounds, I'm getting into action research approaches. I will be doing further work on this in 2005.
I'm increasingly interested in the possible intersection between HCI/Human factors and new media studies / audience studies approaches. Who knows what could come out of this for games research. It is something that I will explore further in the coming year.
Cheers,
-jb
Posted by: John Banks | Dec 22, 2004 at 19:00
Hey TL,
Good to hear from you. A great topic that you started here. What is required I think is an openness from both commercial and academic partners to work at shifting and renegotiating oppositions between the commercial and noncommercial. Rethinking somewhat as you negotiate the relationships where the boundaries are. This is very much about building trust relationships and takes time.
It requires a willingness on the part of the industry partner to understand the research needs of academic partners that may not necessarily have direct short-term commercial outcomes. But on the other hand it requires the academic researchers to be prepared to explore commercial opportunities that may emerge from the relationship, even if this does not quite fit their research agendas. There will be incommensurabilities and often irresolvable tensions that emerge in the relationship. The trick I think is to view these as opportunities, even conditions of possibility rather than obstacles to the relationship. I think it is around those tensions where the really challenging and difficult research questions emerge.
The risk here of course is overly generalising from my experience with Auran. It has taken a lot of work and been a relationship built since 1997. So yes it is unique. However, I think the experience and mode of practice I guess that I've refined by working with Auran can then be carried over to other potential industry-academic research collaborations. A lot of this though will need to be negotiated on a case by case basis.
Moments when the reseacher and company were unsettled - yes many :) Many of these issues were around Auran's relationship with managing essentially a network of voluntary fan labour. The fans have contributed significantly to the success of Auran's Train and Rail simulator Trainz. At times bottom line driven decisions were made that I was not comfortable with that impacted on the fans. However, I think Auran has also developed the relationship with the fans in an innovative direction - sharing IP, often sharing background technical information in a fairly open way, currently exploring revenue sharing models with teams of fan third-party content creators. How do these tensions get resolved? Often they don't :) But I continue working at those areas of tension - identifying them, discussing and exploring them with the team here at Auran, proposing different ways of working with the fans. It is an ongoing dialogue. However, it is through the dialogue that Auran also becomes comfortable with exploring different possibilities in working with academics. An outcome of the research for me is as much about building these relationships and experimenting with their possibilities. Here then research is not about maintaining a distance from the industry partner. The research participates in the process of making and building the networks and objects that are also the research subject.
Cheers,
-jb
Posted by: John Banks | Dec 22, 2004 at 19:45
Very interesting comments John - thanks from some reports from the field :) I look forward to hearing as your work progresses with this stuff. Makes me think we should pull a panel together on the topic sometime!
Posted by: TL | Jan 02, 2005 at 20:00
Hi TL,
Thanks. A panel/seminar to explore these issues would be great. I'll be in touch :)
Cheers,
-jb
John Banks
Posted by: John Banks | Jan 09, 2005 at 22:30
I want to do home data entry work. from Bangladesh.
Posted by: naila zobair | Jan 06, 2006 at 02:25