« I need a hero | Main | A Massive Matrix »

Nov 15, 2005

Comments

1.

My own take is that the two approaches, as they've been outlined above, actually dovetail much better than is often thought.

The value of quantitative work, such as the tremendous stuff being done by Nick, Dimitri, and Raph, among others, is important in two ways. It's important simply by virtue of being quantitative (or in Raph's case based on observations of large pools). It's also important because it responds to those who see quantitative studies as the only true 'scientific' approach.

Qualitative work on the other hand is often dismissed, particularly because it is seen as vulnerable to focusing on 'outliers' or being simply anecdotal. However, the power of qualitative research, to my mind, is that it allows much finer grade distinctions to be drawn among the subjects under study. Qualitative research can, for example, provide insights into motivations and the function of individuals and groups, which quantitative research has a terrible time ferreting out.

As a result, I see tremendous value to qualitative research that starts by examining individuals or groups that 'fit the ideal' of a given cohort identified by quantitative studies. Meanwhile, quantitative research can greatly benefit and draw inspiration for new ways of 'carving up the territory' suggested by close qualitative work.

2.

It's a great topic, and I'm surprised that you're seeing qualitative/micro/exploratory ("Qual+") work being pitched, given my impression that quant/macro/experimental ("Quant+")would have the upper hand in any bid for funding. What I'm curious about (and this would push again at the limits of what people are willing to disclose) is whether the intuition that quant+ is really the superior strategy in seeking funding is correct, given your comments.

I admit that I'm much more drawn to the qual+ paradigm in VW work -- which I appreciate the type of experimental work that we were urged to produce at the Ludium, for instance, and the data produced at PlayOn, sometimes I find it more interesting for the questions it raises. Reading smart qual+ (exploratory) works (e.g. books by Julian and TL) is much more helpful for advancing my thinking in this area. I think this is due to the fact that VWs are fictionally embedded environments, hence empirical approaches have some significant difficulties describing what makes them tick.

Of course, we're talking about funding explicitly, so there are some very old debates about the money devoted to science v humanities, as well as some practical political realities. There's too much to say about that, so I'll just say I'm encouraged by the fact that people have been bold enough to seek funding for qual+ work. :-)

3.

Just to clear up--the large grants that I have reviewed are to large a scale to be simply qual or quant. They typically involved the development and application of a new technology, with qualitative social science work at one or more points in the process, furthermore presented as a valid or even indispensible component. This is why I found it so striking--rather than individual researchers out there trying to make the case for qual+ work (and exploratory work, to a lesser extent), here were large grants where that approach was already integrated.

4.

Quant+ is always going to be more favoured by grant-awarding bodies because it needs the money. Qual+ can be done by someone sitting at home in the bath, but quant+ needs people to go and obtain data.

Personally, I see them as symbiotic. Quant+ is useless unless it is interpreted, which is qual+; qual+ is useless unless it is verified, which is quant+. Virtual world research is only small compared to other fields, and we are desperately short of quantitative data; if there were a virtual world fund-awarding body, it should be throwing money in that direction right now to provide us with facts we can use to propose theories. We'd then want to switch to the theoretical side so as to make sense of the data, making predictions that could be tested through more quantitative work. If we got big enough as a field, these two things would be going on it parallel.

The macro/micro distinction is a tricky one. Right now, micro would be studying one shard of one virtual world, whereas macro would be studying virtual worlds in general. We do both of these, and everything in between, too. However, that's not to say there isn't something more macro or something more micro that we don't yet study, either because it doesn't yet exist or because we have failed to recognise it so far. I try to keep abreast of the whole spectrum, but there's just too much work being done now for me to keep up with it all.

Experimental research is done all the time in virtual worlds, only it's not always recognised as such. Each new game is exploring a new branch of the paradigm, digging a little deeper in some direction in the hope of unearthing new sources of fun. That's only one kind of experimental research, though, and it only appears at one level (that of design). There are many other opportunities to undertake this kind of research within the virtual world itself, but there are moral issues here (those experiments are on real people, who need to give real permission to be guinea pigs). We do get exploratory research (Constance's work on guilds, for example), and as a designer I'd like to see more of this.

A question I'd like to ask is what is it we're hoping to gain from this research? I was at a seminar yesterday that concerned the study of the behaviour of collectors of vintage radios in the light of the arrival of eBay, and while it was fairly interesting to hear about this somewhat esoteric hobby it didn't say anything surprising about the effect of eBay at all. The only purpose of the research seemed to be to keep two academics employed for a year.

What I want from virtual world research is better virtual worlds. Other people may want a better understanding of human nature, or insights into better forms of government, or mechanisms for addressing real-world issues of diversity, or any of a host of other things. I just want better virtual worlds. What "better" means and what "virtual worlds" means are themselves part of the issue. If the answers come from quantitative, qualitative, micro, macro, experimental or exploratory research, I don't care - I just want the answers.

As for which tactic is the best to use for getting a research grant, I wish I could care but there's no grant-awarding body in the UK that will accept proposals for anything as trivial as "game design", so in my case the question is moot.

Richard

5.

I'd like to speak from the perspective of a target of all these funded studies -- a player, or a resident of a world. From a consumer perspective, and also referencing Richard Bartle's idea that such research should lead to better worlds (although I worry about prescribing social missions in advance), I just sense that quantity, if it means lots of sociological surveys weighted for geography and with margins for errors, etc. could be a more accurate tool than quality --when the quality of virtuality is always dubious. I worry when cultural anthropologists come in and study just one tribe or just one village in these ephemeral worlds. I see a group they are studying filled with alts, and I wonder if some minds are at work deliberately packing the anthropologist's study group with the alts to skew their research. There's just an awful lot of room for pranksterism and fake results in these worlds. What do you study when you study a group of anonymous people's avatars? It just seems common sense that more journalistic type research getting as many interviews as possible, as well as extensive polling are going to help these worlds know themselves a lot better than experimental research with just one in-depth localized study, whatever wealth of insights it might yield. These studies are needed particularly as an antidote to perfectly predictable superficial media stories.

6.

I absolutely agree with these methodological concerns--it is very difficult to do good qualitative work online (and that applies as much to interviews as to participant observation; it also applies to quantitative work!).

Actually, focusing the study on smaller groups, especially if they are also locatable offline (which helps establish the credibility of the actions in-world), remedies to a certain extent these concerns, while avoiding the potential for over-generalization of larger-scale studies. This is also why 'long-term' qualitative research is so important (and why journalists who spend a lot of time covering a particular world can do such good work), because the longer one is in place, the greater the opportunities to check hunches, observations, and accounts against one another and one's own experience as a culturally competent participant.

That said, it is always a challenge to understand the value of particularist accounts (so typical of history and cultural anthropology). At their best, however, as monkeysan pointed out, they reveal the subtle interplay of factors that give rise to potentially common patterns, or dynamics--there all along, but so taken for granted as to be virtaully unnoticeable--that we can then recognize beyond the case at hand, and incorporate into the design of all kinds of research.

7.

Thomas, I take your points, but I still worry about anthropologists or journalists becoming hostage to group think and to the "group is its own worst enemy" kind of dynamics where they come to see the object of their study as "right" because they start inevitably to imbibe the group referencing of the others and the outsiders as "they". So much for checking hunches when your hunch is bombarded with messages that it shouldn't be hunching.

Re: "because the longer one is in place, the greater the opportunities to check hunches, observations, and accounts against one another and one's own experience as a culturally competent participant

I'd generally hold this to be true, and I'm all for cultural competence as a RL translator, but there's also a need for correctives and checks and balances, which I think the constant flow of the data and the field view give you from the quantitative study. There are different schools of thought now on this. Some countries like the UK believe that you put a "hand" in a country and that "hand" becomes an expert and lives there for 20 years as an ex-pat and can be reliably expected to know his turf. But the U.S. State Dept. has a different concept and is almost guarantee to put you in Japan if your expertise is Nigeria, to constantly mix up people and rotate them around. This is seen as a weakness but also it destroys the "handism" or "clientalism" that develops when someone is in a place too long. U.S. foreign news bureaus follow the same line, removing bureau chiefs after a tour of duty, whereas British newspapers might keep them in place for years. And who has the better foreign news coverage?

I know in the field of what used to be called Soviet Studies now, in bodies like the American Association for the Advancement of Slavic Studies, there is a crisis of identity and mission, especiall with trends in political science now moving to build models of universality, able to be abstractly applied to any situation, and away from regional studies and country expertise. With the breakup of the Soviet Union in particular, and the end of the Cold War people can more easily get grants to study cross-disciplinary thematic issues like AIDS transmission and health care systems rather than, say, dialects of Ukrainian. It seems there is a fad in other areas too, where localism is dead and thought to be irrelevant. I find this misguided. Yet I could also say that in an interconnected and interdependent world, focusing on only one locality is short-sighted.

BTW, I think U.S. government grant-giving agencies tend to have to show their own performance levels quantitatively with 90-day impact studies showing various metrics, so their tendency is to like the quant vs. qual study.

8.

Greg said (pardon the selective excerpting):

It's a great topic, and I'm surprised that you're seeing qualitative/micro/exploratory ("Qual+") work being pitched, given my impression that quant/macro/experimental ("Quant+")would have the upper hand in any bid for funding...

I admit that I'm much more drawn to the qual+ paradigm in VW work...I find it more interesting for the questions it raises...I think this is due to the fact that VWs are fictionally embedded environments, hence empirical approaches have some significant difficulties describing what makes them tick.

I want to point back to Greg's comments here in order to see whether we can think a bit further about why, if my impression is correct, qual+exploratory work on technology may have gained some prominence recently.

My impression of this comes as much from my own experience of looking for funding from the NSF as from reviewing for them. (Btw, Prokofy, there are many federal grants that do not follow quantitative forms of oversight and follow-up; periodic reports from principal investigators are common, and in some cases ["standard grants"] there is no follow-up from NSF.) After realizing that the cultural anthropology program at NSF would never fund my project (a topic for another day), I looked for funding programs that focused on technology, and found many that were effectively interdisciplinary (even if they came out of a discipline-specific program) because they were asking deep questions about the societal implications of technology. I think this is part of the reason why qual+exploratory research on tech has a place at NSF (and elsewhere): technology's self-evident potential and power lead to urgent questions that are by their nature exploratory and difficult to quantify, such as those of the Ethics and Value Studies Program [in Technology] that funded me.

I don't see this possibility as at odds with Greg's experience, because it is the singular and ineffable power of these environments that he notes which he (and surely many of us) as a user encounters regularly and which seems difficult to capture in research. At the public policy level, that power or potential (which just as often inspires fear and puzzlement as it does something immersive and engaging) translates into questions that, in their breadth and depth, call for a wide range of approaches.

9.

BTW, Thomas, you might be surprised to know I have worked in RL on large projects similar to, and even related to, NSF in RL, involving principle investigators. I know the default on the Internet with someone you don't know is to assume they are talking through their hat, but look me up some time in SL or on email and I'll be glad to share with you my experience. In fact quantitative metrics are used on a huge percent of USAID type grants. I don't have a scientific answer as to what *percentage* of grants overall, of all types of grant-giving agencies, but I think it is *very* safe to say that the default of the U.S. bureaucracy is to look to numbers of some type that they can easily use in their own reporting. Even when they measure quantity like "how much democracy did we succeed in bring about overseas this year" (!) they use opinion polls or sampling of grantees not just consultants' analytical assessments (though they end up relying on the latter). I know, because working for organizations that were recipients of some of these types of grants in the past, I've seen them both use quantitative as well as qualitative methods myself. When the USG relies on a PI, they can do so confident in the world that the PI is already peer-reviewed and presumably showing his qual as well as his quant somewhere else if not to the granting agency. When follow-up reports aren't sought, it's because at the get-go, a determination was made, based on a variety of factors like existing reputation of an institution -- factors that somewhere had to produce numerical metrics to somebody. Yes, I agree that "asking deep questions about the societal implications of technology" was the trigger to use.

10.

Thomas> I want to point back to Greg's comments here in order to see whether we can think a bit further about why, if my impression is correct, qual+exploratory work on technology may have gained some prominence recently.

My take on this is that technology is changing so rapidly that we remain in a lengthy pre-paradigmatic phase (Kuhn). Before a science is self-aware, there aren't any strong rules delineating who a scientist is, or what's a discovery even. Just lots of exploration and accumulation of thoughts that will eventually gel or coagulate or normalize into regular science, at which point there will be much stronger rules about what is publishable. In short, this is the best possible time to be working on this stuff. It's been good for me, anyway. Though strictly trained in quantitative research, I've been forced to deploy pseudo-ethnographic methods when talking about the economies of these places. The existing paradigm of macroeconomic policy just does not fit ("fun" as a policy goal? No way, not in normal economics!). And so to begin talking about a new paradigm, we just have to grasp the phenomenon better. I call my method pseuod-ethnographic because in truth I have no training whatsoever in qualitative methods, and so I have no idea if I am/was handling things properly. I've just been doing "experience and describe" work. And to the extent that others are doing that, they are eminently publishable right now, PhD or no. It's just a wonderful time, I feel very lucky.

The comments to this entry are closed.