Maxwell's Hammer

From time to time here on TN I've delved into methodological territory, and in my last effort, quite some time ago, I focused on the charges of "anecdotalism" that qualitative research in the social sciences sometimes faces, and argued that generalizable claims can be generated out of such methods. But, in retrospect, that piece did not confront the root of the problem directly, given the degree to which I do not there question generalizability itself as the core aim of scientific inquiry.[fn 1] As research on virtual worlds continues to increase, and as the different parts of the academy ramp up their efforts to fight for their funding (and perhaps thereby seek to discredit other approaches), it seems worthwhile (and consistent with the ecumenical spirit that largely characterizes TN) to consider how scientific the pursuit of other kinds of claims apart from the general are.[fn 2] And that's where James Clerk Maxwell comes in...

When it came to generalizability, Maxwell (yes, that Maxwell) was ready to wield a not-so-subtle hammer against those he saw as seeking to hitch science to a positivist view of the world. He said (in a speech the text of which is available here):

It is a metaphysical doctrine that from the same antecedents follow the same consequents...[I]t is not of much use in a world like this, in which the same antecedents never again concur, and nothing ever happens twice.

By highlighting the irreducibly contingent nature of the world, Maxwell joined Charles Darwin in a view of scientific inquiry that saw its provisionality as perfectly consistent with a world that was not, in the last analysis, law-driven and ordered. Instead, they argued that the proper aim of science was to explore the processes that are in place under different conditions, with an awareness that those conditions never perfectly reproduce themselves (for Maxwell, this anti-positivism was also tied to his religious views).[fn 3]

In a sense, all academic research is based on critical observation of such situated events and circumstances. It may be concerned, yes, with making provisional comparisons across them when possible, but it is just as often concerned with understanding the specific processes in place that led to unique outcomes not generalizable elsewhere. For this reason attempts to trumpet generalizability as the primary (or exclusive) aim of the social sciences (where I see it happening quite often) not only marginalize particularist work by historians, sociologists, anthropologists, economists, and others, but (ironically, to me) thereby also seek to exclude a vast swath of the natural sciences (such as much work in paleontology, geology, and biology, to name a few).

As work in the sociology of science has shown, expert critical evaluation (usually associated with the humanities), observation, and hypothesis-testing are all used by all branches of the natural sciences. Efforts to claim special status for the natural sciences (or any field) by pointing to hypothesis-testing ignore not only this, but also the fact that, as Maxwell suggests, hypothesis-testing in the absolute sense does not, in fact, exist (what you have instead are very very very very close approximations of it, and this is only possible for certain kinds of conditions).

What this means for research on virtual worlds is that we must be wary of how the drive to fight for resources may prompt researchers to claim that a certain kind of project (generalization, particularization), or a certain kind of methodology is "scientific" (or, one might imagine, "humanistic," although the comparative lack of money makes this more of a localized danger!), while others are not. A broad view of science, in all its variety, and, ultimately, of academic inquiry, should inoculate us from this kind of divisive maneuvering. Critical observation, exploratory research, and hypothesis-driven work are all going to be vital components of understanding what virtual worlds are all about.

[fn 1] Alert TN reader "Rex" (aka Alex Golub) pointed out this issue in the comments on that post, and I have long wanted to give that observation a proper response.

[fn 2] I am also moved to write this because there is something of an ongoing conversation about scientific "truth" and methodologies here on TN (one example).

[fn 3] For further critical discussions of the limitations of generalizability see the Preface of Anthony Giddens' The Constitution of Society, and Chapter 8 of Alasdair MacIntyre's After Virtue.

Comments

[Andy Havens]

1.

I remember sitting in a meeting, back in 1998 or so, with a bunch of high-energy, nearly vibrating with enthusiasm, Young Media Turks from our ad agency who wanted a metric ton of money to start doing "Web Things" that were going to change the world. Their examples of the campaigns they'd done showed some wildly successful response rates and really good conversion success for the services they'd already done web campaigns for.

Problem is, all the stuff they'd sold up to that point had been free. Free email services, free Web games, free content packages, free software upgrades... free free free.

So my boss asks, "Have you done any of this stuff for products that cost, you know... money."

They had not. We gave them only 1/4 metric ton of money and, compared to our sister regions, only got 1/4 screwed on the net returns of the campaign. Since, it turns out, selling free stuff tends to be easier than selling stuff for money, regardless of media.

My point is this: I agree that generalizations can be very helpful. And that, in a chaotic, random world, nothing can really be said to be completely reproducible. However... the difference between a generalization about "better response rates" for a web campaign and the specifics of years and years of analytics based on TV, radio, print and direct mail experience and observation boiled down -- in this case -- to the generalist not knowing a tinker's cuss about the basic rules of marketing.

Meaning only this: I'd almost always rather have some specific data, even if it's flawed and irreproducible, than a generalization, even if it's pretty good. Because data, even if imperfect, can be used to help *create* a generalization in the future when combined with other data. Whereas a study that has, as its goal, general observations aren't going to help move the ball forward as much.

I'm not saying there's anything inherently wrong with that kind of research -- it is, as Thomas says, a necessary part of the whole picture. I just think we have to carefully proceed towards generalizations that are helpful vs. those that are based on a lack of rigor.

[Mike Sellers]

2.

I agree with you, Andy. In the early days of usability testing the line was "an n of 1 is much better than an n of 0."

Thomas, I'm not sure of the underlying motivating factors for your post, but it sounds to me like an appeal along the lines of, "please don't take my funding, I'm doing science!" combined with an appeal to broaden "science" to include almost any form of presumptively critical inquiry (do I have to bring up Sokal?). A "broad view of science" is necessary particularly in the early stages of any new field, but broadening that too much renders it facile and useless.

I think also that Maxwell's claim that "the same antecedents never again concur," and thus that attempts at replication and generalization are futile, can be easily dismissed as a straw man. Precisely the same train of antecedents may well never occur twice, and yet pretty much every time things happen at the macroscopic or microscopic scale, the same chain of events proceeds in a predictable fashion. We have to go to the smallest scales of quantum events to see significant variance (and even processes like photosynthesis that make use of quantum effects nevertheless are not capricious in their functioning). That "hypothesis-testing in the absolute sense" is not possible is immaterial to any actual study of the human or natural world.

All that said, I do agree with your statement that Critical observation, exploratory research, and hypothesis-driven work are all going to be vital components of understanding what virtual worlds are all about. Ethnographies and quantitative populational studies will both continue to be informative. We must be continually careful, however, to generalize only very tentatively from anecdote, and to avoid making our functional definitions of science and rigorous inquiry so broad that they lose all meaning and predictive power. As my mom used to say, "keep an open mind, but not so open that every wind blows freely through." The same goes for our methods of inquiry, wherever they take us.

[Thomas Malaby]

3.

@Andy: Thanks for the great comment.

@Mike: I've no worries about my funding. But I do sit on a number of review committees and the like for science foundations, etc. For the most part these are people who, rightly, have a broad view of what science is, because these are people who know what scientists actually *do*. They know that exploratory work, and critical observation, are part of the natural sciences, and that attempts to represent the natural sciences as purely driven by hypothesis-testing are fictions. They also know that, when it comes to human beings, the romance of replicability is also to a great extent a fiction.

The fact is, replicability and generalizability *would* be the primary aim of inquiry if we lived in a static universe. But we don't, and whenever humans are involved, the scope of contingencies ramps up so quickly as to make studies that claim to be experiments effectively explorations under another name. This is the point that has been made by Giddens, MacIntyre, and Ian Hacking in addition to Maxwell's implciations noted above.

That's all fine that this is exploratory work, because we can do good work that way, but let's dispense with the notion that hypothesis-testing generates knowledge of a different order from other kinds of claims. Under certain circumstances, for the purposes of generalization, they may be more reliable, but it is a difference of degree, not of kind.

[Thomas Malaby]

4.

Oh, I should add, I more or less agree that "pretty much every time things happen at the macroscopic or microscopic scale, the same chain of events proceeds in a predictable fashion," but the point is that "pretty much every time" is *not* every time, and large-scale consequences can happen as a result of accidents or other contingencies. An obvious example would be how the impact of an asteroid led to the destruction of most major dinosaur species, opening the way for mammalian evolution. These particularist avenues of inquiry are also the proper subject matter of science, and much of the science done to pursue them is not (often cannot) be done through hypothesis-testing.

Once we no longer are blinded by the notion (strange to many scientists, especially field scientists) that hypothesis-testing and generalization are the only worthy pursuits and aims, we start to see science for what it is and has always been. (I could add in here lots of information about how critical evaluation is also a part of science practice, in ways that tend to be overlooked, but it's too late in the evening for a super-long post. :) )

[Mike Sellers]

5.

@Thomas: I've no worries about my funding.

Right - I didn't mean my comments above to be referencing you or anyone else in particular; more the tenor of the argument as I read it seemed to be a defense of those doing broadly based work -- scientific or not, while claiming the imprimatur of "science."

let's dispense with the notion that hypothesis-testing generates knowledge of a different order from other kinds of claims. Under certain circumstances, for the purposes of generalization, they may be more reliable, but it is a difference of degree, not of kind.

I'm not sure I agree, as this seems to open the door to all kinds of potentially dubious truth-claims. This would, for example, allow some to put Aristotelian or astrological methods on equal footing (or at most, seeing them as having differences of degree, not kind) with empirical methods.

That said, I definitely agree that when talking about any system involving humans, hypothesis-testing needs to be handled carefully and creatively. I have done hypothesis-based generalizing research using humans in usability tests (it is possible to generalize from even subjective data if you're careful), and have also had to endure not being able to do much in the way of hypothesis-based investigation on complex AI models in social situations. At some point, as you say, the complexity outweighs our ability to form clear hypotheses in all but the narrowest of domains, and then we have to fall back on other methods of generalizations, ultimately that something "feels right" to participants (this last as applied to whether certain AI models conformed to users' typical expectations of human action). Even this is amenable to hypothesis-based generalization with a sufficient population, but this level of statistical testing is often not feasible.

[Thomas Malaby]

6.

@Mike:

This would, for example, allow some to put Aristotelian or astrological methods on equal footing (or at most, seeing them as having differences of degree, not kind) with empirical methods.

Not at all. I'm only including here empirically-ground claims as being of the same kind. Whenever this point is made about how science is done, it seems to run the risk of being read as a claim that scientific claims are no better than any speculation by anyone. This couldn't be more untrue. The empiricism is very much a part of this portrayal of science. It is so empirical that it seeks to dismantle the romantic and exceptionalist portrait of hypothesis testing that doesn't accord with the empirical facts, and instead seeks to ground it as a human endeavor.

[CherryBomb]

7.

"Precisely the same train of antecedents may well never occur twice, and yet pretty much every time things happen at the macroscopic or microscopic scale, the same chain of events proceeds in a predictable fashion."

Mike, this is simply wrong. Not even getting into chaos theory, contigency is the elephant in the room in historical sciences like geology and paleontology (my background). I really enjoyed reading this post by Thomas, most scientists don't "get" contingency. They may understand is on a rational level, but they do not feel it in their very bones.

Why do we in America use a querty keyboard? Contingency. Why do we drive on the right side of the road? Contingency. Very much of what actually happens cannot be understood with a simply reductionist model.

[Mike Sellers]

8.

CherryBomb, contingency is important in areas involving human interaction, but that's not what I was talking about in the part you quoted from me above, and not what Maxwell was talking about.

I think Thomas has an excellent point. Still, I would be concerned, in a bookend or opposing buttress way, about things like contingency or other methods being used to justify anecdotal or other methods based more on reasoning from arguable but not falsifiable base assumptions. It doesn't take going down that path very far to get mired in intellectual fashion and dogma -- dangers that are hardly unknown in the academy.

[Thomas Malaby]

9.

@CherryBomb: Thank you.

@Mike: The best bulwark against bad work (whether bad because unsupported, or having poor methodology, or poor reasoning, etc) is to dispose of a categorical view of what counts as good work and instead recognize that all scientific (all empirical) claims are measured in their reliability as a basis for further action. There are more reliable claims, and less reliable ones. This realistic (or even better, pragmatic) view of empirical inquiry does not leave us in a position where we are unable to evaluate the claims we encounter. On the contrary, it recognizes that the proof of the good work is in the extent of its reliable application.

So, you see, this does not in any way open the door to an inability to know what good work is. Of course we will still want to have our bridges built by structural engineers, and take astronomy's claims about the arrangement of heavenly bodies over astrologers'. But I'll also take the canon of historical and anthropological work on the Middle East over (most) political science claims if I need to make a good guess about what will happen there next.

More broadly (that is, beyond whatever the basis of your objections may be), the frequency with which this reactionary view is seen whenever a clear-eyed view of science is put forth points only, I'm sorry to say, to the extent to which in the vast support and pursuit of experimental science, many have lost sight of what science actually is, in all its glorious provisionality.

[dmyers]

10.

"Why do we in America use a querty keyboard? Contingency. Why do we drive on the right side of the road? Contingency. Very much of what actually happens cannot be understood with a simply reductionist model."

***

Why do we in America know that we use a qwerty keyboard because of contingency? Hint: not because of contingency. Why do cars that drive on the right side of the road have a steering wheel on the left side -- and vice versa? Hint: not because of contingency.

Too often, perhaps, the silver hammer is waved attractively overhead like a white plastic baton tracing sparkly little things that then so soon fade away. The silver hammer best serves when it comes crashing down. And makes sure you are dead.

***

I do not know what a "basis for further action" is. Is it, for instance, life?

[Bart Simon]

11.

I'll bite on this one... amongst some of us anecdotalists (if we can be such a thing) there is some important subtly in methodology worth mentioning here.

Crucially I'd argue for a distinction between anecdote and case study. A case study in interpretive social science research is a methodology which privileges the particular over the general in a form of reasoning which descends from Casuisty (a kind of situation based moral reasoning but there is much more to it then that).

The case study approach presumes that there is much to be gleaned from the very particularity (or contingency if you like) of the phenomenon being studied in opposition to the search for generalities, commonalities, laws, behaviors and such. I dunno if Maxwell is that helpful in making a point like this... might need to go back to Pascal rather than point to the law like nature of contingency (i.e. it is a law of nature that there can be no big L "Laws" of nature)

Anecdote and the idea of anecdotal evidence is perfectly suspect in case study approaches precisely because the anecdotes need a context in order to be made meaningful so in fact there is a lot of work and debate that goes in to making sense of anecdotal evidence when pursued from a case study perspective.

The biggest problem arises, and I think Thomas will agree, is when anecdotalists try to make generalizable claims from their situated analyses. If you are a hard core anecdotalist nothing exists (in any meaningful way) beyond the situatedness of the phenomenon anyway. The fallacy appears when folks try to match up verbatim anecdotes (suddenly called evidence) with demographic data say in an effort to say that the anecdote is particular support for a generality (pattern, hypothesis tec...) or vice versa -- when anecdotalists do this they can be justly accused of making all sorts of validity errors IMO.

So if I say "a funny thing happened to me on the way to work the other day..." and you use that to support your survey data that 90.235% of white, male professionals think that the world is a funny place then well Houston, we have a epistemological problem.

[Thomas Malaby]

12.

Agreed on all counts, Bart. You might find this earlier thread on anecdotalism specifically to be of interest.

[Bart Simon]

13.

Ah ok... I had a vague memory of this. After rereading the old thread I am sorry to see that Doug Thomas' defense of the anecdote qua anecdote was not taken up with more vigor (i'd be up for this on a different thread) but leaving that aside I am curious about where Thomas' Maxwellianism is meant to take us.

There's nothing wrong with the ecumenical plea for mutual respect in game studies... why we've just finished with the Charles Taylor report on 'reasonable accommodation' here in Quebec (http://en.wikipedia.org/wiki/Reasonable_accommodation). Nothing wrong with making reasonable accommodation part of the charter of game/VW studies but I would suggest there are some sticky problems with this when it comes to carving out the institutional backbone for this new field we are in.

1. Epistemological tolerance is nice but the liberal "let 1000 flowers bloom" approach does tend to hide the fact that some folks hold the purse strings. This amounts to a situation in which some scholars say go ahead and do that arty farty Maxwellian stuff but not in my backyard (or vice versa). Well the history of Anthropology testifies to this form of institutional liberalism and the field is severely balkanized into cultural and physical anthropologies as a result (see #2 below)

2. In this age of shrinking page numbers and quicky blog posts we are tending to wear our methodological presuppositions too far down under our shirts. There is no room or time left, it seems, for casuists, anecdotalists (is Doug the last one?), naturalistic ethnographers, logical positivists, empiricists of all stripes and heaven knows who else to explain the terms of their engagement to audiences who do not share those terms.... such a crucial thing in this interdisciplinary melting pot. This 'need for speed' condition will probably lead to a kind methodological balkanization also and to the victor go the spoils.

3. So far the interpretivists (my label for the entire artsy fartsy anecdotalistic collective of which I am one)in game/VW studies have been the nicer ones; smiling politely if knowingly when the jabs come. Well i'm feeling plucky today... why plead tolerance when we should just fight back. Hey back in the day, I did a tour in the "science wars" (http://en.wikipedia.org/wiki/Science_wars) and I gots me a high level toon that can pick off ADDs from any direction. You pull Thomas I got your back :)

Naw just kidding... peace out man!

[Cunzy1 1]

14.

I can't decide whether or not to:

a) Start my lists with letters or

2) Do it with numbers.

I think numbers carries more weight. It seems like I am asserting myself, whereas letters just seem to be offering up suggestions.

III) No. That just doesn't work at all.

[Dmitri Williams]

15.

Bart wrote:
"The biggest problem arises, and I think Thomas will agree, is when anecdotalists try to make generalizable claims from their situated analyses. If you are a hard core anecdotalist nothing exists (in any meaningful way) beyond the situatedness of the phenomenon anyway. The fallacy appears when folks try to match up verbatim anecdotes (suddenly called evidence) with demographic data say in an effort to say that the anecdote is particular support for a generality (pattern, hypothesis tec...) or vice versa -- when anecdotalists do this they can be justly accused of making all sorts of validity errors IMO."

and Dmitri cheered.

In virtual worlds, life is pushed into less human and organic spaces because it's governed by code. That makes sampling potentially easier, although the human condition of course remains messy even when segmented into servers and factions.

As always, I see great value in anecdotes and case studies and really only ever get my back up when those studies are used in the way Bart flags. It's so valuable to see what is happening and what is possible. It's also a different question than how often it happens. Thus, triangulating methods remains a very good thing.

[Thomas Malaby]

16.

That's good to hear, Dmitri, and I'm sure that means you'll be as ready to object as I am the next time someone says that hypothesis-testing, or quantitative research, has a corner on the market in "Big 'T' truth." ;-)

[Dmitri Williams]

17.

I would be.

Every field has its folks who are sure that theirs is the only possible method. I have as much patience for them as for dogmatic religious zealots. Ironically, there's less need for tolerance when dealing with the intolerant.

[Richard Bartle]

18.

Thomas>What this means for research on virtual worlds is that we must be wary of how the drive to fight for resources may prompt researchers to claim that a certain kind of project ... or a certain kind of methodology is "scientific" ... while others are not.

Hmm, I have two points to make here.

Firstly, some of the methodologies people employ really are unscientific or involve very poor scholarship by any objective measure. Yes, I agree, we may see some people claiming that something isn't scientific when it is; however, they may actually be right, and there is a lack of scientific rigour that should concern us.

Secondly, I wish some of those scarce resources could be allocated directly to the study of virtual worlds for their own sakes, rather than indirectly through benefiting some external academic discipline. I just want better virtual worlds; to people like me, being able to criticise one another's research for being unscientific is a luxury we'd like to have.

That doesn't mean I think this is a pointless discussion, I hasten to add; rather, it's just a reminder that the public perception of the worthiness of a subject can be just as big an obstacle as that of the quality of its science.

Richard

[Thomas Malaby]

19.

@Richard: I don't think we disagree about your first point. My point was about categorical rejections of one or more of the methodologies employed throughout the academy (and specifically those employed, as noted above, in the natural sciences as well as elsewhere). Of course, a categorical rejection of a widely discredited research methodology could be valid as a kind of shorthand, but anyone making such a rejection should be able to give specific reasons as to the shortcomings, rather than simply point to something as "not science."

Sadly, in my opinion, the latter tendency has become more in evidence as even those within the academy, drawn to devote their energies into specialization, lose sight of what makes empirical inquiry what it is (witness the anemic answer to intelligent design -- the last thing the scientific community should have done was to attempt to answer the ID'rs certainty with certainty).

[Susan]

20.

Thanks, Thomas. That was a really thought-provking piece. A couple of reactions:

Structured versus unstructured data

In many studies in the social sciences, the most revealing data is relatively unstructured, and doesn't easily fit into a standard statistical test.

Suppose you're asking all your participants about their gender and ethnicity. A structured form might have tick boxes for "male" vs "female", and "white" vs "black". But your your interview subject might self-report their ethnicity as "Irish traveller", and emphasize that they consider this to be distinct from "Roma". This might might well be more relevant than just ticking the box labelled "white". (e.g. the purpose of your study is to investigate potential racist bias in policing, and the officer who arrested your interview subject has said to you "I hate gypsies" - which is another piece of unstructured data).

You can gather "unstructured" data in a relatively systematic way - for example, it's common to have a flow-chart of questions to bring up in ocnversation with each interview subject.

Unstructured data is also compatabile with statistical sampling. For example, your sample population might be everyone who is arrested for drug-related offenses in a particular city within a particular time period, and you can interview a random sample of this population.

You can't step into the same river twice

However well they were designed, statistical samples of Second Life users carried out last year can't be repeated this year, because contigent factors have changed. For example, the gambling ban started to be enforced last Easter, and now there's no way another researcher can re-measure Second Life with gambling. This is a big problem for reprodcability of results.

I have a related concern about long-tailed distributions. How big a sample size do you need to reliably estimate the mean? For example, suppose you are measuring the number of IP datagrams sent between Europe and the US per unit item. There may be very long frequency fluctuations in the rate, such that your sample period isn't long enough. (Or worse: such that no sample period is long enough).

The researcher's personal experience may not be typical

The researcher's personal anecdote may not be typical of some hypothetical population that is being studied.

But you get this effect anyway, even if you try to do random sampling:

- Some interviewers may be better than others at getting interview subjects to be forthcoming (think of KInsey, for example). So even if you random sample interview subjects, the results may not be reproducible with a different interviewer.

- Bonnie's cybersex research records how people have cybersex with her: the possibility exists that they may behave differently having cybersex with someone else

- Psychiatric diagnoses are notoriously dependent on the psychiatrist who is making the diagnosis. So if you want to establish, for example, that there is a difference in the rate of autism between two populations, you'd better make sure the psychiatrists doing the diagnoses on the two popualtion are comparable (e.g. the same, or suyitably random sampled from the same population of psychiatrists)

[Thomas Malaby]

21.

Great stuff, Susan. Thanks.