Anti Anti-Anecdotalism

The recent flurry of attention to SL and its numbers (here, here, here, and, most recently, here) leads me to think that folks might be interested in having a chance to chew through some methodological stuff, along the lines of the "Methodologies and Metrics" panel on which Nic, Dmitri, and I served at the State of Play/Terra Nova Symposium early this month. Below the fold, some tweaked ideas from some emails I circulated among the panelists in preparation for the panel. While I'm not discussing virtual worlds and the methodologies we'd use to understand them specifically, I hope this will be helpful background for such a discussion.

It is hard to get away from a common conception, both within and outside academia, that numbers are the one, true path to understanding. This is part of a set of cultural expectations that are reproduced precisely because they are so rarely challenged. Most commonly, one hears that claims with numbers are "grounded" or otherwise true in a way that other kinds of claims (such as the ones based on the kind of research that Tim talked about here), are not. Claims based primarily on those other kinds of research, particularly on interviews and participant observation, often get branded as "anecdotes", with the suggestion that they hold no real value as reliable claims. Here I would like to push against this association, and help clarify our understanding of what qualitative social science research methods (ethnographic research ones in particular) bring to the table. In short, they are not "anecdotes", and they can form the basis of reliable claims, even without numbers, although as Dmitri and I never tire of saying, having both is better than having just one.

No social scientist, of course, would want to "generalize from anecdotes," but the problem is that often we do not really understand what that means; or perhaps it is more accurate to say that across the academy many scholars (not to mention the public at large and policy makers) do not know enough about methodology (this is true of both qualitative and quantitative methods, and more broadly about exploratory versus experimental research), and therefore these charges are in essence a political move meant to marginalize the other side's research that can succeed because of that lack of broad grounding. From my conversations with everyone involved with TN I have never felt that we (as a group of authors) were particularly prone to make these errors, but there is no question that it finds its way into the discussions on TN, as in the recent threads.

The goal of all social science is "generalization" in a sense, but the legacy of positivist thinking about society (that it is governed by discoverable and universal laws) has left us in the habit of thinking that the only generalization that counts is universal. It is always interesting to me how some work (especially that done by the more publicly-legitimized fields, such as economics) can proclaim itself to be about the universal despite the fact that only a moment's thinking reveals the application of the ideas to be narrow (to industrialized, capitalist contexts, etc). The strange thing is that this doesn't end up being a problem for those already-legitimate fields; instead, it is largely ignored -- this is what being well situated on the landscape of policy and academic relations of power gets you (to be Foucauldian for a moment).

But of course generalization, in the more limited sense of seeking a bridgehead of understanding across times and spaces, has long been the hallmark of history (the first social science, in a way). The strange thing is how difficult it seems to be for those who would like to criticize methods such as participant observation and interviewing to see the projects those methods support in a similar light to history and its efforts. There is nothing inherently problematic with such claims; they are just as able to inform policy as universal ones, and have the benefit of incorporating more nuance.

So then what is an anecdote? It is a description of an event isolated from its broader context, so no wonder all of us would like to shy away from the suggestion that we are drawing our conclusions in isolation of the broader context. But ethnography (meaning principally participant observation, along with interviewing, surveying, and other methods), to speak of that relevant methodology most familiar to me, quite distinctly does not treat these events in isolation. Brief descriptions are often presented in the course of ethnographic writing in order to illustrate a point concretely, but the point made is only as sound as the degree to which we trust the author's command of the broad array of processes ongoing in the context at hand. How is this credibility established? Through a complex of many, many, many techniques of writing, thick description, peer review (always including experts in that period or place), solid reasoning itself, track record of previous research, etc, etc. This form of generating reliable claims is not somehow "less" viable than other ones, and its strengths and weaknesses of similar scope (though differing in their particulars).

So one of the tropes that one finds in the recent spate of posts about SL and its numbers is the suggestion that only when numbers that we trust are present do we feel that the claims authors make are "grounded". This is not true. As anyone with much experience with statistics knows, the numbers say nothing without the ability to interpret them provided by other kinds of interpretive research. In fact, given the above, if any research has a claim to being "grounded" it is the first-hand research of participant observation.

Even when this kind of contribution from qualitative research methods is acknowledged, however, there is still a tendency to see the claims of work based on them as always and severely limited to a "niche", at least until numbers come along. But a social history or ethnography of a place and time is not this narrow. They are able to make general claims at the level of locale, region, or even nation, and they often do (when done well). The idea for ethnographies is that the ethnographic research method, at root, inculcates in the researcher a degree of cultural competence such that he or she can act capably (and sensibly) as a member of that culture. Supported by observation, archival research, surveys, or interviews (usually some combination), as well as (possibly) prior work, this learned disposition informs an account of the shared disposition of the actors on the ground, and is laid out in the published work (as best one can in writing) as representative of a worldview from a particular time and place. Thus, my claims about gambling in Greece were made beyond the level of the city where I did my research, and I argued for the existence of a cultural disposition that characterizes Greek attitudes toward contingency at something like the national level (without holding too much to hard boundaries).

Of course, these claims are further bolstered by the broadening of one's research methods, whether through surveys, demographic data, archival research, media studies, or any other means that support the big picture. Relatedly, there is nothing about quantitative methods that dictates that they must "stay big." They can be productively focused and narrow as well.

This is not to say that there isn't a limit to the level of generalization for qualitative research that is exceeded by quantitative methods. So, for example, while an ethnography could make reliable claims about Greek culture, I don't think it could about American culture. The reason for this connects to what culture is -- a set of shared expectations, based on shared experience and continually re-made through shared practices -- and why it is far too fragmented and varied across the US for an ethnography to make such claims. But while this is true, the important point is that qualitative methods' levels of claims are not as particularist as they are sometimes made out to be.

I become, I confess, a bit sad whenever I encounter this kind of marginalization in action (for me, it most often happens on interdisciplinary fellowship review panels and the like), because at root it bespeaks a lack of trust across the academy. There is little doubt that there have been excesses across the gamut of methodologies and theories that the social sciences use (reductions to representation, or materiality, or power all come to mind), and perhaps this accounts for the parochialism and suspicion, but let's hope that we don't fall prey to what are more often, in my view, essentially not battles over the nature of sound inquiry but instead part of gambits meant to direct or redirect institutional resources.


Comments on Anti Anti-Anecdotalism:

Ordinal Malaprop says:

Are there any particular examples that you are thinking of here where anecdotes have been criticised despite an author's ethnographic knowledge?

Posted Dec 30, 2006 3:57:01 PM | link

IAmEric says:

Hi Thomas,

I completely agree with your main point. One thing I don't understand is why you feel the need to say it when it seems self evident to me. Why do you care if others emphasize quantitative research over ethnographic research? If I were you, I would just concentrate on what you think is important and screw everyone else. If you're right, which I think you are, then let's build on it and see where it takes us. People can only ignore the truth for so long (although sometimes it is amazing how long it takes).

In regard to virtual worlds, I've said in another thread that I think the appropriate way to proceed in their study is to treat them as if they truly were newly discovered economies. Economists rely on numbers such as GDP, inflation, unemployment, wages, etc but where do those numbers come from? They come from ethnographic research. Still when you perform a survey, it helps to know what percentage of the population your sample represents. I think that is one of things that Clay is saying. We don't even know what the population is, so how can we have faith in a survey?

Virtual economies pose an interesting difference though. Unlike real economies where it is impossible to trace every microtransaction so that reliance on ethnographic research is essential, in virtual economies, at least in principle, you CAN get at the numbers purely from querying a database. Nonetheless, it seems obvious to me that there will need to be some level of ethnographic research to study virtual economies.

So how should we proceed with ethnogrpahic research? Should we hire avatars to roam the virtual landscape to conduct virtual surveys? May not be a bad idea, but I'm not sure how to motivate people to do so.

Ultimately, the state of ethnographic research may need to advance to the point that the results can be aggregated into meaningful numbers that can then be studied by virtual economists. I don't know many economists who actually go out and perform ethnographic research themselves, but they rely intensely on the ethnographic research of others.

Best regards,
Eric

Posted Dec 30, 2006 4:00:34 PM | link

Thomas Malaby says:

@Ordinal: I don't really understand the question. The issue is not that "anecdotes" have been criticized, it is that some work has been labeled as "anecdotal" to start with, when it is not.

I'm not speaking about my own work, actually (at least, not that I know of -- there just haven't been that many people who've read Gambling Life! ;-) ), but I hesitate to point to the particular examples of which I am aware, since I'm not interested in specific recriminations. I'm more interested in developing our understanding of what qualitative research methodology is and what it can accomplish. Labeling qualitative work as "anecdotal" is common enough around the academy that it provides a useful frame through which to get at this issue.

Posted Dec 30, 2006 4:08:28 PM | link

Thomas Malaby says:

@Eric: Great questions about how to move forward. My short answer is that while for some time I was quite a skeptic about the prospects for doing ethnographic research that is primarily situated in virtual worlds, I have recently come to feel that the bandwidth for human action within them (through better and better avatars, plus lots of other old and new stuff, including text-based interaction, etc) has increased enough that we can begin to think about these domains as generating distinctive dispositions for their participants (that is, generating distinctive cultures, if you will). This means that, from an ethnographer's point of view, the same process of enculturation can be followed, with the right training, by a researcher in order to develop a reliable picture of that cultural point of view.

But challenges remain. One of the biggest for online communities in general is still true of most MMOs, and that is the "person-to-person" quality of a lot of communication. In Greece, I knew if two customers at a cafe were whispering to each other, even if I didn't know what they were saying, and this meant that it wasn't lost (I knew I could follow up about it with them, or others, etc). In most MMOs, private tells and the like between third parties are essentially invisible to the ethnographer. This is something that is an issue, but, again, becomes less of a dealbreaker as the more visible dimensions of action widen in scope.

Another related challenge is the relatively low fidelity of the environments. If something big and bad went down on the other side of the old harbor in Chania, where I did my research in Greece, it was immediately apparent, and one could quickly head to that kind of "diagnostic event" (as Sally Falk Moore called them) and try to capture as much as one can. In MMOs, the situation is both better and worse. The graphical fidelity (and scope of action, above) are so narrow that if a large-scale event goes down in, say, a WoW server, it would be harder to stand somewhere and feel like you could follow a number of simultaneous developments (and check them up with contacts later). But, MMOs do provide the opportunity to marshal technological resources (such as add-ons in WoW and the like) that could be used to aid the ethnographer (a possible analog is the "gargoyle" from Snow Crash :-), though sort of like having one in the Metaverse). Nic's presentation for the panel above demo'd the beginnings of some incredibly cool tools for just this kind of application.

Posted Dec 30, 2006 4:26:12 PM | link

Ordinal Malaprop says:

Thomas: Fair enough; I was just wondering whether there were any good examples that one could get one's teeth into, as it were.

In the context of Second Life, I would say that qualifying one's ethnographic experience would be quite a vague process, in that it would depend on displaying sufficient familiarity with the culture and history. There are certain historical events of which displaying knowledge might be sufficient, but on the other hand, a new resident could have "got it" without knowing anything about the history. Accurately reporting and analysing certain aspects of resident interaction would be a sign for me... how people react in different situations, where you've entered their land by mistake, where you're in a club, where you're in a store, displaying some knowledge of what happens there.

To be honest almost anything above the usual journalistic "SL is all about sex and you can make money there too!" would give a researcher some credibility.

Posted Dec 30, 2006 4:36:36 PM | link

Allen Sligar says:

@Thomas

"the numbers say nothing without the ability to interpret them provided by other kinds of interpretive research."

Exactly

@Eric

"Ultimately, the state of ethnographic research may need to advance to the point that the results can be aggregated into meaningful numbers that can then be studied by virtual economists."

And if one had the ability to capture both qualitative and quantitative data, aggregated, and historically trended, it would be even better....

Good insights in this post. Of course I'm more focused on the quant rather than the qual, how to capture meaningful qualatative data has been an issue I've had trouble with for awhile now because its not my area of expertise.


Posted Dec 30, 2006 6:41:29 PM | link

Prokofy Neva says:

I'm all for getting away from obsessive and narrow literalist number-crunching -- it's never enough, anyway, and I suspect that even if the Lindens handed Shirky figures showing the actual, true, list of accounts defatted of alts and mess-ups, and showed him the actual and true percentage of log-ons after 90 days from sign-ups, it wouldn't be enough and he wouldn't like SL anyway.

However, I get very wary of new-fangled anthropological and sociological and ethnographical studies of SL when they begin to throw around the usual campus Marxist jargon about "empowerment" and "neoliberalism" and "capitalist" this and that. Just study the stuff, please, without the scrim of ideology.

And here's where I just don't get your field of study, Thomas, I'm sorry. Writing an ethnography of Linden Lab may be a very worthy and interesting (and obviously already funded) endeavor likely to produce a fascinating book. I don't know what you are quite calling this genre. Corporate ethnography?

But for me, studying LL without studying its main product, the world and the people in it is, it like being the court scribe and writing about the French Revolution from the point of view of the king. It would be only a partial account.

Posted Dec 30, 2006 7:45:13 PM | link

Thomas Malaby says:

@Prok:
I agree in general with the critique of the overdetermination of anthropological (and plenty of other social scientific!) conclusions by the vulgar application of theory. This happens all the time, because it's easier than doing the empirical work. The postmodern "turn" in anthropology of the eighties was just the most recent incarnation. I don't see how that relates to what I'm saying here, though I'd be happy to hear how.

Re: what kind of ethnography I'm doing -- I answered that question in the other thread. It is not "corporate ethnography"; it belongs to a subgenre of ethnography called the "ethnography of organizations", if anywhere, and this has a long and distinguished history, in fact. It is concerned with the social processes that characterize how organizations work (and think).

Yes, my account *will* be a partial account, just as my account of Greek cultural attitudes toward the unexpected was itself a partial account (and of course it was a partial account of Greek life!). But that work doesn't exist on its own; there are a raft of ethnographies of Greece, and reading the lot would take one a long way toward understanding Greeece (along with other kinds of studies -- histories, etc). The same will be true of the Linden book.The essential point is that there is a *lot* going on with LL and SL (as there is with any human endeavor involving more than, say, 8 people ;-), and therefore there is plenty of room for multiple studies on all the aspects of this that interest us. Your interests may be partially served by my ethnography, maybe not, but there is a lot more work of all kinds to do about all these spaces.

Posted Dec 30, 2006 8:07:28 PM | link

magicback (Frank) says:

Peter Drucker contributed greatly to the study of corporations as a major social entity. I think you guys will fill the same role for VWs as an emerging social entity.

He was a bit of iconcast and didn't follow the standards of the day and did pronounce much "dogma" or "qualitative claims", but after a few decades he got credibility when many of his claims were found to be "golden wisdom".

His principles and claims worked for the corporate organization and I think much of it will work for VWs. I thinik we should look to him as a role model and his practice as a process model for VW.

One aspect I liked about his approach is that he looked for practical application and results. He gain acceptance and traction because his "dogmas" were practical, applicable, and focused on results (which businesses can readily understands).

Keep at it Thomas and Co. Happy to contribute whatever I can.

Frank

Posted Dec 30, 2006 9:43:28 PM | link

Prokofy Neva says:

Thomas, there are now so many threads going on this subject of the SL numbers that I can't find your previous answer -- I even went searching for your name specifically. I do take your point. I quite understand, having read up on it that "corporate ethnography" is more a term involving a corporation's own use of ethnography to do something like determine how its customers are adapting to a product. What you are doing is a study of an organization, hence, ethnography of an organization.

Again, I feel we are very, very far from Greece when we take up an object of study like "Linden Lab". Greece has lots of books out on the subject already; Greece is a kind of known area of knowledge.


The Lab, on the other hand, has never been studied (except upside down from underneath the microscope slide by its sometimes unsuspecting amoebae residents). So either you are studying "just another company" or you are studying something that has revolutionary or evolutionary new potentials and actualizties (i.e. all that Love Machine stuff, the fact that they've made a virtual world kicking off the Metaverse to a new level, etc.) or you are studying something that may be all too ordinary but has made something extraordinary -- or however you put it.

No doubt my pesky questioning makes you uncomfortable, but since you *are* a public intellectual who is going to be publishing a book in the publice eye, I can only plead a kind of consumer's plea: look beyond whatever flora and fauna you find at 1100 Sansome Street into the world and the residents' feedback, where I think you *do* have to look to understand the company and what they are doing.

For example, if you read a text like this:
http://blog.secondlife.com/2005/04/13/vote/

about Philip Rosedale's own notions of voting, organizations, companies, the market, etc. and his idealistic and enthusiastic blog about it, what context could you understand it in? Only in the reality of the feedback not only of the residents right on that blog, but looking at things like the feature voting tool -- where you can't vote "no"!

Or looking at his idea of the "wisdom of crowds" as applied to SL literally -- as if a lot of SL crowds in fact aren't so smart lol -- take where it is absurdly played out as a justification for ramming p2p back into the world (the excuse that it was once there during a smaller earlier phase doesn't cut it)and ripping out telehubs where they made sense (infohubs) and keeping them enabled where they didn't make sense (private islands).

Posted Dec 31, 2006 12:30:47 AM | link

Thomas Malaby says:

Lol, Prok. The voting system(s). Perhaps you may find my ethnography interesting after all :-).

And I'm not uncomfortable by the questions at all, though I find it a bit strange that you presume I haven't found much of the same material and researched it in my own way (such as Philip's entry, above). Look, why don't you read it when it comes out, okay? I'll look forward to your reaction, though I'm sure that at least in some ways it won't measure up to your own view from under the microscope slide. :-)

P.S.: The reply to your first asking of the corporate ethnography question is actually in this thread (it's not a Shirky one; that may have tripped you up).

Posted Dec 31, 2006 12:41:12 AM | link

Thomas Malaby says:

@Frank: Your comments are very much appreciated. Drucker's ideas are very, very close to my own in many ways -- I chalk it up to a (presumed) common background in American pragmatist thought. Though I've always wondered if he was influenced by some German thought given his background -- but he's so optimistic that he really sounds a lot like Dewey in the end. I'd love to read a good biography of him, if you have a recommendation.

Posted Dec 31, 2006 12:44:54 AM | link

Douglas Thomas says:

I am not sure I am ready to dismiss the anecdote so quickly. In a lot of cases, anecdote is the home of nuance, it has incredible power to provide illustration and illumination. We generalize from anecdotes all the time and if you look at the work of someone like Kenneth Burke, he makes a case that certain anecdotes (or types of anecdotes) are "representative" (if not generalizable).

More importantly, anecdotes capture something that academic research has traditionally had trouble with, which is the performative nature of gaming as an "event."

We've all lived through one of the most significant anecdotes of our lifetimes in Sept 11th. It would seem absurd to argue that we need not study or pay attention to it because it was only an "isolated event." It was an anecdote and I would argue it was an incredibly powerful one which provided an opportunity to see, discuss, and debate things that were not part of our national consciousness prior to it.

In fact, I will go out on a limb and contend that illustrative anecdotes are better at getting at *some* things in games than many other approaches.

@Eric: "Ultimately, the state of ethnographic research may need to advance to the point that the results can be aggregated into meaningful numbers."

This seems to me to be precisely the problem. I would contend that ethnographic research and a host of other qualitative approaches cannot be translated in the way you suggest. Doing so creates a kind of category error. It is not possible to test out certain kinds of knowledge. Put differently, if you take a particular event ("WMDs in Iraq") you can measure it as an historical question (were there or weren't there), as a social scientific question ("Did people believe there were WMDs" or "How did major newspapers report WMDs?"). Translating one from the other, potentially results in research which begins with interviewing a few key people in Iraq and reviewing documents and then turning to public opinion polls to see if the documents were right, you know, giving meaningful numbers to a single interview or text.


@Prok: "usual campus Marxist jargon about "empowerment" and "neoliberalism" and "capitalist" this and that." I am curious why you would have an objection to this kind of specialized language? You do realize that these are words and ideas that have rich theoretical heritage and meaning, that they are words that do work to help us understand things happening in the world (both real and virtual). I never quite understand the reaction to "jargon," as if it only exists among left-wing academics. It seems to me a way to dismiss ideas without engaging them.

Signed,
Your postmodernist, anecdotal pal

Posted Dec 31, 2006 9:03:44 AM | link

Thomas Malaby says:

I agree with everything you've said, Doug, except that I'm not sure "anecdote" in the sense you use it can be recuperated. Besides, when anecdotes *are* effective in the way you describe, it's because the context that makes them sensible is implicitly shared among the interlocutors to such a degree that they do not even need to state it. The danger in that direction is the reproduction of a kind of class (or generational) view of things. Thick description, by contrast, was in my view part of an effort to ensure that we don't rely overmuch on the implicit understandings between, say, the ethnographer and the reading audience when we talk about events.

And events *are* vitally important; you're absolutely right. Sally Falk Moore talks about the events that are powerful in the way you describe as "diagnostic events", because their ruptures allow us to see social cleavages otherwise kept offstage.

Perhaps the confusion here, then, results from two ways of thinking about the term. If an anecdote is simply a tool of writing that presents a brief account of an event amid a broader analytical and descriptive work, then certainly ethnographers (and many others) use anecdotes all the time, and use them well. Beyond the diagnostic events above, there is also the written "mise-en-scène" that begins many an ethnography (including mine) as an example of this kind of writing tool. It is incredibly effective at evoking a sense of place and time.

But I think the dismissive use of "anecdote", meaning a brief description that is isolated from its context, is how the term is most commonly used around the academy today. If we could bring it back to a less politically-charged meaning, that would be great, but I'm afraid I'm not sanguine about it.

Posted Dec 31, 2006 9:21:47 AM | link

Tom Boellstorff says:

Excellent, inspirational post, Thomas, and great discussion too, as I'm pounding away at the manuscript for "Virtually Human: An Anthropologist in Second Life." There's a lot I would like to add to this discussion, but in the interest of pressing on with the book, I'll hold off and just provide a couple sentences from my book The Gay Archipelago (Princeton University Press, 2005) where I briefly address this issue. Thanks again for this great exchange.

"Qualitative methods are not more micrological or localizing than quantitative ones; they illuminate different aspects of social life. When explaining this to students I tell the story of a quantitative researcher and a qualitative researcher who go to Japan to study the Japanese language. The quantitative researcher prepares a survey that can be distributed to a large, random sample, providing valuable data concerning, say, varieties of Japanese dialects. The qualitative researcher studies a far smaller sample; he or she could not learn every vocabulary item of Japanese, for instance, or every dialect. Yet through immersion in daily life this person could learn to speak Japanese: from a handful of individuals one can learn how to communicate with millions. Qualitative research can be effective in drawing out cultural logics that, like languages, are shared: it is an approach suited to the study of similitude. There is no necessary relationship between methods and spatial scale: it is not true that qualitative methods like ethnography are 'more local' and quantitative methods 'more global.' As globalization becomes seen as the new default state of affairs in the world, there is a real need for qualitative studies of the global." (p. 20)

Posted Dec 31, 2006 10:00:23 AM | link

Thomas Malaby says:

Thanks, Tom!

Posted Dec 31, 2006 10:25:43 AM | link

Dmitri Williams says:

A couple thoughts: Generalizability isn't always about a universal scope. It's about knowing how your sample is or isn't representative of a larger population. That population might also be fractional. That's OK.

The qual/quant process is indeed best when done concurrently, but let's be honest about how science works. Due to people, projects, funding and the like, it's usually iterative. I like to draw a census as step 1, but the census-taking can certainly be informed and improved by qualitative insights and empiricism.

For example, in game research, I like to play the thing and talk to people to understand local customs and language before I develop any kind of survey or census. If I had better ethnographic skills, I'd have better outcomes, which would in turn lead to a better quantitative survey.

Now, the dreaded anecdote. Doug is right that it is very important to be able to freelance and develop new thoughts outside of sampling frames and generalizability. That's very often the start of crucial insights and should never be dismissed. I'm there with you.

So what's the problem then? Well, every method has its abusers. Just like there are legion of statisticians screwing the pooch, there are qualitative researchers who do that purely anecdotal research AND THEN imply generalizability. That's where I have a problem.

Someone discovers the Gor subculture in SL? Cool. Tell me all about it. But don't at the end of the paper then tell me it's "common," "rare" or anything else that connatates frequency without also telling me something about the sampling used so I can evaluate the claim. If the goal isn't generalizability, don't then imply that it's there. If it is, do the legwork to back it up, or at least give us the famous "future research should investigate . . ." it.

An example of how to do this well is in Thomas' own work on Greek gambling subcultures. As he described it to me, he learned a ton from anecdotal and systematic ethnographic information. He then had a sense of how and why his subjects did or didn't generalize to a larger culture. Given enough detail about that step, the reader can evaluate it and decide if he/she agrees, and it's all good.

Posted Dec 31, 2006 11:20:30 AM | link

Rex says:

It's also worth stating that one of the theoretical approaches associated with ethnographic research is an idiographic, rather than nomothetic, approach to data -- a concern with the particular, rather than the general. Many ethnographers -- and even more humanists -- would argue that the goal of inquiry is _not_ generalizable results but detailed knowledge of a particular thing. Particularizing research can be done both qualitatively and quantitatively.

Posted Dec 31, 2006 12:49:34 PM | link

Allen Sligar says:

.@Douglas

I agree with you, about avoiding category errors in qualitative analysis that is aggregated.

That’s what’s so frustrating, its hard to do accurately. Especially hard for an old fashioned data mechanic like myself.


@Dimitri

"For example, in game research, I like to play the thing and talk to people to understand local customs and language before I develop any kind of survey or census. If I had better ethnographic skills, I'd have better outcomes, which would in turn lead to a better quantitative survey."

You touch on a key concept here, to get at the quantitative data one must understand the underlying context or rather system from which the data occurs. Then one must describe components of the system, assign it a language. As you know all languages that aren’t dead evolve over time through meaningful contribution.

The language may be rudimentary and need work to begin with, but a descriptive language of a system that allows for quantitative study, and data, is better than nothing at all.

In this I mean a descriptive language that allows for symbolic annotation. Much like Koster has described in Grammar of Games.

This brings me to another point, it takes many actors to form a language, and give it a common meaning. A casual gamer may assign meaning much differently than a developer, an academic or members of the press (is this where the media disconnect with SL went wrong? Could be). So the description must encompass an aspect of common utility, this allows it to be used over again, and in fact to evolve. Advanced concepts can be extrapolated from this but not the reverse. Otherwise we would all be speaking the Kings English rather than American no?

Is it not true that if I say to you: I phwn noobs ir l33t, lolz… you will likely understand that as the common dialect and assign it meaning as you understand it.

IMO academics do not desire the reverse of this relationship in any field of study, however if those you study do not understand what your asking them your likely to get both qual/quant errors coming and going.

“So what's the problem then? Well, every method has its abusers. Just like there are legion of statisticians screwing the pooch, there are qualitative researchers who do that purely anecdotal research AND THEN imply generalizability. That's where I have a problem.”

And so here the age old argument’s of the subjectivist vs. the objectivist begin…..

Implied generalization is fine, if one wants to exile themselves to the ivory tower…
http://gamasutra.com/features/20061110/hopson_01.shtml

Posted Dec 31, 2006 1:08:44 PM | link

Luis Villa says:

As much as I hate to agree with Prokofy...

You do realize that these are words and ideas that have rich theoretical heritage and meaning...

The words have a rich theoretical heritage and meaning, and many of them also carry a lot of baggage and unquestioned assumptions from the sixties about power and cultural relationships- assumptions often just as biased and skewed (though in different ways) as the relationships which these meanings arose in opposition to. It could, of course, be possible to use the terms critically, and to resituate them in a less value-laden context. For better or for worse, though, I think Prokofy's unspoken assumption that this is very rare is likely correct, which means that any work using that jargon is often tainted by association.

Tangentially, and speaking of learning and teaching by anecdote, I'll give an anecdote about the jargon-creators: at my undergrad institution, there was a course which purported to be an introduction to Marxism, taught by a professor of literature whose work you've no doubt read, Douglas. Unusually, it was cross-listed in six different departments, including literature, sociology, etc. None of those six departments were political science or economics.

I shall forever be curious as to how one teaches an intro to Marx which can't qualify for cross-listing in political science or economics. My hunch was that even the actual Marxists in the political science department wanted nothing to do with what passed for Marxism over in the litcrit department. Sadly, since the course wasn't cross-listed in political science, it didn't help me with my political philosophy requirements, so I was never able to squeeze the course into my schedule and confirm that hunch myself.

Posted Jan 1, 2007 8:32:48 AM | link

Thomas Malaby says:

May I suggest that anyone who would like to gleefully engage in theory-bashing for its own sake head over to this thread? Much of that territory was covered there.

P.S. to Frank: The good news is that in re-checking that thread, I found your recommendation of Peter Drucker: Shaping the Managerial Mind by John E. Flaherty. I'll be picking that one up.

Posted Jan 1, 2007 10:06:22 AM | link

Prokofy Neva says:

>You do realize that these are words and ideas that have rich theoretical heritage and meaning, that they are words that do work to help us understand things happening in the world (both real and virtual).

No, these aren't richly theoretical, they're just lame, one-dimensional, shrill phrases from standard lexicons of Marxist cant and rant. They aren't really tools for thinking or discourse, but just ideological memes that have infected lots of stuff. Yes, it's all about this warmed-over Marxism.

Posted Jan 1, 2007 5:20:44 PM | link

Thomas Malaby says:

@Prok: I do find your opinion valuable, but on this point you're not making sense. You said,

I get very wary of new-fangled anthropological and sociological and ethnographical studies of SL when they begin to throw around the usual campus Marxist jargon about "empowerment" and "neoliberalism" and "capitalist" this and that. Just study the stuff, please, without the scrim of ideology.

"Neoliberalism" and "capitalist", at least ;-), certainly do have rich theoretical heritage and meaning, although of course this doesn't mean that they're well-used and applied just by being invoked. In any human endeavor there is always plenty of bad craftsmanship around; it's like a law of gravity. This is exacerbated by new and interesting phenomena that draw the interest of those with too much time on their hands coupled with just enough education to be a danger to themselves and others. Add in a dash of youthful idealism amongst the connected and well-heeled, and you've got a recipe for disaster. So if you're thinking of a particular set of people's work, then fine, but it should be obvious, if I may say so, that it's possible to use these terms in effective and nuanced ways, just as Doug described.

Posted Jan 1, 2007 5:33:17 PM | link

Andy Havens says:

One of the things I really love about my chosen career -- marketing -- is that it has a couple absolutes; at least in my book, and when it's done well. One of these is that you don't study nothin' without first establishing a reason for doing so (goals) and the metrics by which you'll know if those goals are getting closer or further away.

We're currently beating this dead horse on all the various "SL Numbers Are Crap" posts, but I'm not sure the fundamental point has been made in terms that could (maybe?) relate to academic research, i.e., anecdotalism vs. quantitative research.

In marketing, there's plenty of good reasons to do both quantitative and qualitative research, as long as you set those reasons up ahead of time and know what you're going to do with the data after the fact. It's like planning a vacation. You can either ask a bunch of people, "What fun beaches have you been to?" (qualitative), or you can measure distances to various beaches, prices of hotels nearby, likelihood of bad weather, etc. (quantitative). One method isn't better, they're just different, depending on what you want to know.

It's that last part that trips up lots of investigation, I think, outside the (somewhat) more focused world of business/marketing. In business, at the end of the day, we usually have a couple of pretty clear "root goals." Make more money, increase customer satisfaction, keep employees happy (that'd be a "Balanced Scorecard" approach anyway).

But if you want to "study stuff for the fun of it," or to get a graduate degree or to make an argument for a religious or political debate (and no slight intended on any of those pursuits), the "goal" of the research may often be much trickier, and, thus, the relative need for qualitative vs. quantitative data (and types of such) ends up being balanced (I've often found) AFTER the data has been gathered.

Sometimes, when a particular subject is just so new and fuzzy and weird... yes. You wander in and start learning. You measure everything you can and take whatever learning out of the situation is possible. Whether anecdotal or quantitative, however, that kind of initial "soaking in it" kind of investigation needs to be backed up by reproducibility to be true knowledge.

The job of a poet is to make us think deeply about things that may not be. Which is incredibly useful. It is an anecdotal pursuit of limitless, fictional proportion. On the other end of the spectrum, the census taker should try to be purely quantitative. An unruly, metaphoric survey is worse than none at all.

When we know the goals of what we are looking for -- or at least know what kind of knowledge we are trying to increase -- it's much more effective. Measuring for the sake of measuring is often like running in place; it may feel like there's a lot going on around you, but you're really not going anywhere.

Posted Jan 1, 2007 6:56:05 PM | link

magicback (Frank) says:

Andy Said: "Sometimes, when a particular subject is just so new and fuzzy and weird... yes. You wander in and start learning. "

Fuzzy logic and its usages are useful in this discussion. Marketing and corporate innovation are two functional areas that uses both quantitative and qualitative approaches. Qualitative research usually yields insights while Quantitative research usually yields proof. Feeling you way around is an unstructured approach while hypothesis and parameters is a structured approach. I think both have their uses.

@Thomas: hope you will find the book useful. Check out Bruce Nussbaum (BusinessWeek) and his blogposts on Ethnography. He has a couple of articles on the subject as related to design and innovation. A key article is "Ethnography is the new Core Competency" or just do a search on Ethnography within BW's website.


Frank

Posted Jan 3, 2007 4:41:43 AM | link

Douglas Thomas says:

@ prok "No, these aren't richly theoretical, they're just lame, one-dimensional, shrill phrases from standard lexicons of Marxist cant and rant. They aren't really tools for thinking or discourse, but just ideological memes that have infected lots of stuff. Yes, it's all about this warmed-over Marxism."

Capitalism?

Ummm. I guess I don't really have a response to this. You are right. There is nothing to capitalism. It doesn't exist as anything other than a meme used by warmed over Marxists. It would be silly to mention it at all in the context of Virtual Worlds. I think we should also abandon those warmed over economic memes, made popular and overuse by the whole "Free Market" crowd. Supply, demand, inflation, profit, market. Nothing but silliness. So, we agree, such terms should be abandoned.

I had no idea it was so easy to dismiss ideas. You simply claim they aren't serious and they go away. What a relief, I can't wait to visit sweatshops in the global south. If they only knew that oppression, surplus value, and exploitation were worn out Marxist tropes they would feel so much better. I know I will now.

@Dmitri I don't think we have a difference of opinion in principle. No one likes it when anyone makes claims they can't support. On the flip side, there seems to be a hierarchy emerging, perpetuated in part by the practices of some academics, suggesting that the kind of knowledge which generalizes is better than kinds that don't, or that there is a rational progression from non-generalizable knowledge (which may be interesting, but is trivial) to generalizable knowledge, which is "True."

The point I wanted to make about category errors is a big one. There are certain things that are knowable in a generalizable sort of way and certain things that aren't. I am amazed that Columbine produced a movement to try to find out if aggression and violence from video games was generalizable (it seems obvious that if the Columbine kids' experience was generalizable, we'd have millions of shootings). In fact, it seems that understanding what was in Dylan and Eric's heads that day is precisely the kind of thing that you don't want to generalize. It doesn't make sense to (does it?)

But that is exactly what we did, we said "Ohhhh, kids played video games and shot people," is there a causal connection. That strikes me as an idiotic question. And worse, it leads to a bunch of research that we think is answering the question, when it is measuring something completely different.

Do we really care if there is a statistically significant causal connection between video game playing and aggression? I think we only do if there are consequences to such aggression. Are there? Of course there are, look at Columbine. . . . There is your big fat category error. And it is a bad one.

@Luis

"I shall forever be curious as to how one teaches an intro to Marx which can't qualify for cross-listing in political science or economics."

Would you make the same argument if it had been a class in econ not cross listed in literature?

More important. I can think of lots of reasons:

1. Economic: poli sci and econ may be in different schools, programs, or areas which divide up revenue differently.

2. Personal. Said professor may not like the econ or poli sci people

3. Perspective. I can think of quite a few works that Marx wrote or were inspired by him that have little to do with Econ or Poli Sci as it is taught at many universities.

4. Student cross over. Said professor may work regularly with students in those programs, serve on their committees, or have joint appointments there. Would make sense to cross list so they could count his classes toward their degrees.

5. Politics. There may be turf issues among different departments

6. Methodologies. Econ and Poli Sci tend heavily toward quant methods, the others have a bent toward qualitative methods. That seems as good a way to divide things as any. (I don't know the uni, prof, or class, so that is speculation, but you ask how it could be possible).

7. Maybe said professor tried? And maybe the Poli Sci and Econ departments weren't interested?

8. Perhaps the class was on Marxism and literature, which would seem deadly boring to Econ types, but not to the others?

That is the beginnings of a very long list of reasons. But I have to say, the idea of dismissing a class in anything based on where it is not cross listed seems like a fairly weak argument. If you are a scientist you could make the same argument from Theology classes not cross-listing in Astronomy and Biology. Or Econ departments not cross listing in Psychology or Sociology. Most knowledge is multidisciplinary, so the list of potential cross listing is boundless.

A better question to me is "Was the class any good?" And if it was why would it not fulfilling some checklist for a degree stop you from taking it?

Posted Jan 3, 2007 6:17:35 PM | link

Neils Clark says:

Thomas, I agree that really great research shows a thoughtful interplay between qual/quant.

I was nervous when I first saw the title of this post. The foundational authors for "addiction," to the interweb and online games, still use unsupported anecdotes in the worst possible way. They talk about marital distress and junkie adolescents, marginalizing anyone who plays or, conversely, has an interest in experiencing a persistent world of contrived contingency.

I'm ridiculously thrilled to see that TN is actively pursuing standards for games. Right now brilliant people are using addiction and dependency standards which lack face validity. I would ask that you not overlook studies in addiction and dependency, as along with fears of online violence, unrepresentative anecdotes here present one of the greatest obstacles to social acceptance and a resultant understanding of these worlds.

Posted Jan 4, 2007 5:21:44 PM | link

Thomas Malaby says:

@Neils: Thank you for the comment. You may be interested to hear where the title comes from. It's a reference to Clifford Geertz's famous distinguished lecture, "Anti Anti-Relativism", in which he took pains to note that the title was meant to indicate his opposition to the shrill denunciations of anti-relativists, rather than that he was in favor of relativism, which he called a "drained term." As he put it, "My through-the-looking-glass title is intended to suggest such an effort to counter a view rather than to defend the view it claims to be counter to." :-)

Posted Jan 5, 2007 4:08:08 PM | link