Standard Metrics of Use

I'm AFK for a couple weeks and I miss a crazy series of threads on SL. Well, as a sociologist/statistician, I'm going to avoid the politics here and simply state what I would like to see for universal metrics of virtual world use--something to get us in the ballpark of apples to apples. The intent, after all, is to have agreed-upon baseline numbers which you then argue about being indicative of something.

I'd like to have measures of these basic things for any given world:
Duration
Intensity
Economic activity
Multiplicity
Active users

Here are my potshots on what those would look like:

The first three are cummulative for all accounts/avatars owned/used.

Duration (could also be called "stickyness"): The amount of time the person has been a user of the world, expressed as a population mean with a standard deviation. This would be retroactive time from today, telescoping backwards. The user need not be on every day. Gaps between logins are allowable, with the maximum allowable time gap between logins being 90 days. Use example: "The average user of MegaWorld has a duration of 83 days." The standard deviation of the mean would help explain whether the typical person logged in once and quit or actually hugs the mean.

Intensity: Average hours per day per user. Sure, subject to all kinds of distortions for people who leave accounts logged on for various purposes (selling goods in Lineage 2, e.g.). Nevertheless, pretty useful as an indicator of bandwidth and interest levels and roughly comparable across worlds.

Economic activity A: A binary variable reporting whether the user does or does not pay for access to the virtual world.

Economic activity B: The average amount paid per user per week across the world population.

Multiplicity: Another population mean, an indicator of the number of accounts or avatars maintained by the user. This number would then be used for the previous three to obtain per-character averages. Note that the number could be either more than 1 or a fraction of 1 if the account is shared.

Lastly, I would like to see the term "active users" simply be anyone who has logged in to the world within the past 30 days. Not perfect, but simple and pretty good for an apples-to-apples measure across VWs.

With these numbers, you could calculate a range of use statistics for different users' needs. And, of course, having the raw numbers themselves (rather than means & SDs) to plot distributions would be even better.

I understand the incentives to supply, distort or hide these figures and I just don't care. I want the raw numbers based on these criteria (even if it makes Thomas think less of me ;) ). And sure, we're all subject to using what companies actually decide to disclose, but that's no reason to not spell out a better standard.

No doubt the excellent collection of minds perusing this site can improve on my starter list. The goal is to come up with some series of metrics that make sense across virtual worlds.

And if you want to pile on Clay, LL or anyone else, there are three other threads open that would love to hear from you.

Comments

[Prokofy Neva]

Sounds good, but I'm not liking the concept of "economic activity" meaning only the payment to access the world. The economic activity in the world is important, too. I think those are actually the numbers to watch -- the "GDP and GNP" if you will of SL, how much economic activity within the world, how much is repatriated to real life, etc.

It seems LL does count transactions twice, and there is some funny stuff going on with their numbers on the statistics page. I've also always protested against their removal of land sales from "business" and "profitable" avatar lists. People selling land count as a business too. It's insane to cut out the main business of SL from those figures. They do that for ideological reasons, wishing to privilege what they think of as "compelling content".

[Matt Mihaly]

How about picking a different phase from 'active users' to denote all logins within the last 30 days? It's pretty misleading to categorize someone who logged in for a few minutes and left, never to return, an 'active user.'

--matt

[Raph]

So a typical MMORPG would show something like

Duration: 9mos (or whatever -- I have seen anything from 6 to 18 mos here, and I don't know that anyone is measuring this well right now)
Intensity: 2 1/2 hours
A: Yes
B: $15
Multiplicity: 2+ (yes, that's the average)
Active users: 90,000

So the thing this doesn't capture is that there's 100,000 paying. So I wouldn't toss that metric yet. :)

[Andy Havens]

Some good thoughts previously here:

http://terranova.blogs.com/terra_nova/2006/01/100k_group_list.html

[csven]

First off, thank you.

Second, with regards to Economic Activity B ("The average amount paid per user per week across the world population."), I'm wondering if you wouldn't mind explaining that a bit more.

[Thomas Malaby]

Hehe, they all look like good starting points to me, Dmitri. After all, I said here that "more data (of all kinds) is helpful". Of course, the devil will be in understanding what these numbers mean (as always), but that's the fun part ;-).

[Dmitri Williams]

@Prokofy. We can come up with another label, but my intent is a measure of money given by users to creators, not actual activity within the world. That'd be a separate metric, and one I wasn't trying to tackle (yet).

@Raph. Quite right, and I wasn't clear in stating that I'd want not just the binary paying/not paying, but the resulting % from the population. This then lets you calculate the gross income as well as the number of payers vs. non-payers (plus some other fun stuff).

It'd be great to see a side-by-side with those kind of numbers, wouldn't it?

@csven. Just some time-based metric of money paid to the game maker/hoster. Given smaller units of time, you can extrapolate upwards, but you can't gain granularity going in the other direction.

@Andy. Right also. I just thought it was time to goose the subject given the, err, intense interest in the SL/Shirky threads. And the recent activity in the dreaded back channel.

@Thomas. If I get numbers, you are one of the guys I ask about for their meaning. Triangulation FTW.

@Matt. OK. If you don't like "active users," provide some alternatives. I think it's fine as it doesn't imply *how* active, but if you can do better, great.

[Prokofy Neva]

We can come up with another label, but my intent is a measure of money given by users to creators,

Well, here's where you are coming at it with the game/MMORPG bias, assuming all creation and content is in the hands of game gods, and payments to them in RL outside the arch of the game are the only metric that counts.

But the creators are inside the world in SL too because it's user-content. The sales in world are largely of user-made content; the game-gods only sell the land.

[Allen Sligar]

This is a good start, I'll come back to this later in the weekend but it's important to understand that taken alone, those data points could be rendered useless. Meaning regression to the mean, and/or sampling issues dependant/indepandant etc.

What it takes in actuality is multiple data points trended over a period, this is hard to expalin with brevity. However, what I'm getting at is using your base data points, and using associted data points, and capturing data related to an initial, and growing population over time would give you the ability to trend your analysis of these particular metrics.

The eventual goal or rather outcome would enable predictive modeling of the population under consideration as it related to the particular data points youve specified.

You could, in fact, predictively model many things related to your subject depending on the granularity of your data points as well as historical data.

I'll revisit this thread later, and get into more depth related to what you've proposed Dimitri, mainly because I'm looking for input on standardization of dimensions, measures and data points from the Academic's and Dev's but I need to get sleep right now....7 days until we start beta, times currently at a premium for me :)

[Clay Shirky]

@Dmitri,

How about "Visitors and users" for people who've logged in at least once in the last 30 days, and "Active Users" for people who have logged in in the last 30 days who also logged in the previous 30 days? (This is close to what IAmEric was suggesting in a previous thread, minus the hourly count per user.)

[blaze@blaze.com]

I think a big problem here is that people are focusing far too much on people numbers.

Nobody will provide accurate published numbers, simply because it's in no one's best interest to do so.

Not SL, WOW, EQ .. anyone.

If I wanted to, I could tell you all the lies that are being told in the drilled down numbers (how islands built are including void islands, how the transactions are meaningless, how I've ran across herds of automated libsecondlife bots on at one time, etc etc).

... but then I can also tell you that I make more than 8000 USD seling virtual goods.

But then the same could be said in WoW .. how much of that concurrency is people farming gold or plat or whatever it is?

The fact is, only the highly technical among us really understand what's going on, and it ain't pretty.

What makes SL or WoW or whatever compelling will never be the numbers, because the numbers will always lie.

What makes them compelling is what they provide. How I can go in world and collaboratively study and build complex systems with friends, students, and customers. How we can interact in a deeply immersive way, in a way that is going to only get more immersive every day.

There was a great post awhile back on Terra Nova. It was about group IM chat. This was very insightful and in many ways, quite brilliants, because groups in SL are the government .. are the over arching social structure .. especially now that forums are gone.

Study this and I think you'll get a lot more insight into the sociology of virtual worlds.

Study the numbers and you'll just be counting the number of angles dancing on the head of a pin.

[Clay Shirky]

If I wanted to, I could tell you all the lies that are being told in the drilled down numbers (how islands built are including void islands, how the transactions are meaningless, how I've ran across herds of automated libsecondlife bots on at one time, etc etc).

Yes please -- tell us that, or point us to other places telling us that.

Study the numbers and you'll just be counting the number of angles dancing on the head of a pin.

Blaze, the 'angels on the head of a pin' reference is inapt, because, unlike the number of unique users in Second Life, no one knows the answer about angels. Second Life calculates the number of unique users daily for it's economic stats, but only publishes the number with positive balances -- it could easily answer the more general question if it wanted to.

And the idea of doing research on sophisticated sociological problems without knowing how many people you are looking at is simply daft. Size matters -- villages are different kinds of things than towns are different kinds of things than cities. You have to understand the prevalence and density of social options, like "how many people are there to talk to or collaborate with or trade with?", in order to even begin thinking about second order effects. Economies or governance structures can't be meaningfully looked at without answering the basic demographic questions, which is what Dmitri is trying to do.

[Thomas Malaby]

Clay wrote:

the idea of doing research on sophisticated sociological problems without knowing how many people you are looking at is simply daft

And it is just as daft to believe that you can confidently interpret those numbers without understanding things that don't always show up within them, Clay. It's a two-way street, as Dmitri and I understand it. One of the things that means is that micro-level qualitative (and quantitative) studies are just as essential to the long-term effort to develop our understanding of these spaces.

Yes, these numbers would be cool to have, and it would be a lot of fun to how to adjust (in terms of the formal characteristics) and interpret the stats Dmitri suggests in order to *begin* to attempt meaningful comparisons across virtual worlds. As I said in the other thread, the numbers we would need would have to be new ones, not the old categories that have been worked from (and worked over) in the past. Your dismissal of Blaze's comments and your other comments bespeak to me a certain lack of interest in the broad and open-ended inquiry in which we are engaged.

I'm not sure you will respond to my comments here, since you did not deign to in the other thread, but I continue to wonder why we should be interested in your point of view if it is so uninterested in nuance and a balance between the qualitative and quantitative, between meaning and materiality. The ongoing effort to understand SL and other VWs is a process that requires a lot of hard and new conceptual work from people on all sides. Part of what this means is that banging the cheap drum of "Linden knows but won't tell us" starts to sound like a message that is meant to play well (especially since your criticism was always more powerful, imho, as a critique of the MSM). This is especially true given the fact that it's pretty clear to most that are familiar with SL and LL that Linden Lab quite pointedly is *not* in a position of total understanding of what it has wrought, and therefore even if we knew everything they know (which would be nice), we still would be a long way from the whole story.

[IAmEric]

Hi Dmitri,

Thanks for taking the question seriously and getting some real concrete suggestions started. If the tasks was to come up with some apples to apples numbers for comparison across virtual worlds, this is a great start.

I'm still relatively new to this area, but I am more interested in Prokofy's thoughts. I'm not so much interested in the economics of the game from the perspective of an outside observer trying to compute say the impact of virtual worlds on the GDP of the host country or countries. I am more interested in things along the lines of Castronova's original papers regarding the economies within the game.

You said you haven't tried to tackle this problem (yet), so I am very much looking forward to hearing what you have to say when you do.

Perhaps jumping the gun a little bit, to study the economies within these virtual worlds, it makes sense to me to treat them as if they truly were newly discovered economies. What would a virtual census burough do to construct a meaningful number for the population of the virtual world?

For example, I wish more people who are better at economics than I am could reproduce this type of work but for various virtual economies. The following was taking from an article in Walrus Magazine a while back. I can't find the original article because the link I have is no longer active, but I saved this quote:

"Things got even more interesting when Castronova learned about the "player auctions." EverQuest players would sometimes tire of the game, and decide to sell off their characters or virtual possessions at an on-line auction site such as eBay. When Castronova checked the auction sites, he saw that a Belt of the Great Turtle or a Robe of Primordial Waters might fetch forty dollars; powerful characters would go for several hundred or more. And sometimes people would sell off 500,000-fold bags of platinum pieces for as much as $1,000.

As Castronova stared at the auction listings, he recognized with a shock what he was looking at. It was a form of currency trading. Each item had a value in virtual "platinum pieces"; when it was sold on eBay, someone was paying cold hard American cash for it. That meant the platinum piece was worth something in real currency. EverQuest's economy actually had real-world value.

He began calculating frantically. He gathered data on 616 auctions, observing how much each item sold for in U.S. dollars. When he averaged the results, he was stunned to discover that the EverQuest platinum piece was worth about one cent U.S. — higher than the Japanese yen or the Italian lira. With that information, he could figure out how fast the EverQuest economy was growing. Since players were killing monsters or skinning bunnies every day, they were, in effect, creating wealth. Crunching more numbers, Castronova found that the average player was generating 319 platinum pieces each hour he or she was in the game — the equivalent of $3.42 (U.S.) per hour. "That's higher than the minimum wage in most countries," he marvelled.

Then he performed one final analysis: The Gross National Product of EverQuest, measured by how much wealth all the players together created in a single year inside the game. It turned out to be $2,266 U.S. per capita. By World Bank rankings, that made EverQuest richer than India, Bulgaria, or China, and nearly as wealthy as Russia.

It was the seventy-seventh richest country in the world. And it didn't even exist.
Castronova sat back in his chair in his cramped home office, and the weird enormity of his findings dawned on him. Many economists define their careers by studying a country. He had discovered one."

I consider Castronova's work to be ground breaking, but I am a little perplexed why he hasn't become more involved with Second Life. Are you around Professor Castronova? :)

The economy of Second Life doesn't need discovering. It was there front and center all along.

In the case of EverQuest, it seems like the economy emerged as a byproduct. Is that true? How is it now? Are there other virtual worlds where the virtual economy is an important aspect of the "game"?

Dmitri, your ideas represent a great start, but in the end, I think to compare virtual economies, we should start taking the economies seriously. How do you compare real economies? Perhaps we should just try to compute GDP, inflation, unemployment rate, wages, etc.

Is it farfetched to view Second Life and other virtual worlds with thriving virtual economies as "emerging markets"?

Best regards,
Eric

[Clay Shirky]

@Thomas,

No dis intended in the other thread, I stopped reading it in favor of later ones, and only just now noticed that it was still ongoing.

You say "And it is just as daft to believe that you can confidently interpret those numbers without understanding things that don't always show up within them, Clay."

I don't disagree. I'm certainly not saying that once we know the population, all our questions will be answered. I am however saying that none of the nuanced questions will be terribly well grounded if we don't have population figures as a start.

[Prokofy Neva]

Eric,

Having read the work of Castronova over the years, as you cite, I've constantly been perplexed why he never tackled the economy of Second Life; I really wish he had. It seemed like a perfect match. I've asked that question publicly in forums a dozen times, including here. Curiously, he's never answered. Oh, I know he's been cited as a talking head on this or that general article on Second Life where he provides the requisite sound-byte about how lots of people are online these days and some of them monetarize their time on line.

But he's never visible inworld giving public lectures, etc. (though his university's program has various activities in the Kurrian Expedition) and he's never posting or commenting on this amazing synthetic economy -- you think he would.

Now, how to explain that? Traditionally around things SL, one can explain these silences by possibly 1) an NDA 2) working on a competitor for SL still under wraps 3) work on a non-public alt or 4) simply having concluded that SL isn't a game, and even its synthetic world status is questionable (i.e. as was stated above there may only be tokens standing in for a real-world gambling economy). I have no idea if any of these apply, but I have to conclude that something about SL is either very compelling, or not compelling enough for Castronova, and that's an indicator all its own.

Clay, I can only keep repeating as others have repeated: 36,000 paid subscriptions, which support some 100,000 regular log-ons. That's never enough for you, but that's what it is.

The news of breaking the barrier to reach 20,000 concommitant log-ons were announced yesterday mainly to disdain from all the SL-related blogs, who remarked that if SL was down half a dozen times in that day due to asset server problems, these were mainly 20,000 cranky customers.

blaze, I've seen the herds of bots, too, but since only 40 or 50 can be logged on a mainland or island sim at a time, I don't think this is a significant dent in the numbers -- alts are way more significant. Libsecondlife has had a program for months, related to Copybot, which can log on multiple bots and run them through IMs. These clone drones are flooding camp-chair establishments and training them of thousands of Linden dollars by ordering the bots to sit and occupy all the chairs to get the camp pay-out, but then not play the casinos. The camp-chair kings are howling, and wondering why LL doesn't shut this down as an exploit -- though casinos with camping are hated by a vocal minority of tekkies and oldbies, still, botting the owners is theft, if they are drained of all their money.

There are also groups like ESC logging on herds of bots to test them for various things, and of course I at the Herald have asked the pointed question whether the non-copyable sheep avatars on all these bots were paid for, and gotten the not-completely-satisfying answer that yes, they are (but then, the avatar's creator is under an NDA).

NDAs explain a lot of the mysteries of our world, as I've commented:
http://secondthoughts.typepad.com/second_thoughts/2006/11/nondisclosure.html

I don't *at all* buy this concept of the scientists at the Lab having created a monster they can't track or understand. They just aren't talking. And you wouldn't either, if you had invented a thingie that was going to make you and your friends a bundle.

LL tracks how many objects are created constantly; remember the ticker they used to run on the front page "last 10 things purchased in world" that they finally removed because it was endlessly showing gambling objects like "Skunk Money"? Run that program, and you could tell lots and lots about the nature of the economy and whether the figure of the US $ spent is just washing in and out of casino objects.

LL now has the ability to remove an object from the world at one fell swoop on multiple sims when a griefer puts it out (or maybe they always had that but didn't use it or couldn't use it). That means they have (or had) the ability to track any object inworld across all sims whenever they wish.

When I asked Philip once -- to get past the gamed traffic numbers -- to indicate what the parcels were that people actually picked and put on their "Picks," he had the database query and numbers within a day or two, showing a different "top 20" than the gamed "top 20". I know that all the Lindens are fascinated with statistics, and put up as many as they can and still remain a private corporation -- or so it seems. It's never enough, however,

If we're going to make up wish lists of numbers, I'd really like to have a number that shows people who log-off *but come back after longer periods*. This is the most interesting number to me. As noted, I have a list of tenants each month who return after long absences from SL. It's a handful, but a growing handful because they must read about SL and decide that their tryme or even lapsed premium account they never fully cancelled can be reactivated. While some percent log-off due to poor performance, I would say most leave because of a bad relationship or a move in RL to a country or situation where they can't get DSL or log-on. So a figure that lit up with the emigrants who return would be interesting to me to see -- log-ons after 6 months or 12 months. And there's no reason that couldn't be tracked to see if it offsets any of that 90 percent or whatever the large figure is that are looky-loos who then leave.

The Herald and Tateru have reported on the waves that occur -- ebbing and flowing. These waves are tied to SL's performance, the patch days, news stories, etc. Some days you feel as if the entire place is empty. Others you put out an ad and 36 people arrive to look at a rental within an hour, and you can't possibly wait on them all. There is definitely always a huge ebb and flow, and I guess what is important is that each wave does leave behind more starfish on the beach.

[Thomas Malaby]

@Clay: That's fine put that way, of course, but you put things more strongly frequently. Such as when you say,

You have to understand the prevalence and density of social options, like "how many people are there to talk to or collaborate with or trade with?", in order to even begin thinking about second order effects.

This is not really true, if by this you're suggesting that inquiry can't even begin until the broad-level demographic questions are answered. The point I'm trying to make sure you're on board with (and frankly, I've seen little indication of it) is that the numbers do not precede other kinds of understanding, such as what Blaze was pointing to. These are all ingredients of an ongoing inquiry, so I would ask that you shy away from the rhetoric which suggests that we can't say anything until we understand everything from a numerical point of view. That's impossible.

[Dmitri Williams]

@Clay. There are a lot of levels of nuance and granularity we could suggest, but there's always a tradeoff. At one end of the spectrum you get something pretty abstract and broad and of minimal use. At the other end you get a perfect detailed replica. Or, as Steven Wright used to intone: At home I have a map of the United States. . . Actual size . . . At the bottom it says 'One mile equals one mile.' So I'd stick with the 30 days and "active" since that's a billing standard and might actually see the light of day. Plus, if you have the duration numbers (mean and SD) and intensity, you can make some qualitative inferences about activity stretching past that 30 days.

Alan is right to suggest repeated measures, and anyone can be right to suggest more depth to any one of the measures. Personally, I'd just like a massive data dump of everything ever, which I'd then use to calculate trends via time series analysis and use hierarchical linear models to look at groups compared to individuals and multi level models over time. But that'd be a bit much for the quasi-experts who need a level of understanding that is broad, but still a bit detailed enough the get some nuance. And it'd be unrealistic to ask press or even most investors to understand most of that. They'd just ask statisticians to process the numbers and spit out some coarser stat anyway.

@blaze. I disagree. Numbers matter for scope and scale. A census is a platform upon which everything else gets built. Once you establish it, you can then drill down and know how the people or situations you are looking at are or aren't representative of the whole or some portion of the whole. If Thomas or you or I identify some interesting new phenomenon (Gor groups, RMT, cybering, VoIP, guild dynamics, etc., etc., etc.), it can then be studied both in depth and context, but also with an understanding of how common it is. Meaning is of course important, but when it's only .01% of a population it's well, less meaningful to me. And media research tells us that this sense of proportion is one of the things the press is the worst about in most news formats.

For the researcher or analyst, this is also about allocating research resources. If you want to understand as much as possible, you devote a large chunk toward the largest pools of phenomena and then keep several moderate chunks for rare and emerging phenomena. Imagine a corporation investing in movies. They'd want to diversify by going both the big-studio route, but also investing in some Miramax-level and Sundance stuff. You spread, but spread based on baseline information.

@Prokofy. I'm after game-exogenous money, not internal. Do I think that internal money transactions and user creations are important and sociologically interesting? Yes, and I am actively engaged in research projects measuring them. But it's simply a different topic for a different post. A base statistic of who pays how much for access is worthwhile for a range of people. If you want to propose some universal measure for in-game economic generation, trade or arbitrage, go for it. You're probably in a good position to take your own starter pot shot.

@Eric. We (Ted, Dan, Nick and myself) have this kind of very detailed data on a popular MMO and are working toward getting it usable. The good news and bad news about modeling an entire economy is that the data volume is massive. We're talking about terrabyte of data on transactions, and that is tough to instrument and to query. It's taken a full-time programmer and the assistance of the National Center for Supercomputing Applications here at Illinois about 5 months to even get the data into an analyzable format. We're about a month and half away from having it ready to use, and then some time after that to crunch numbers and make sense of what we find. But again, that is a topic for a different post. Maybe you should be emailing Prokofy and working up a system and a post for us to host . . .

[IAmEric]

Hi Dmitri,

It's great to know that people at my alma mater are thinking about this stuff. Go Illini! I got my doctorate on the other side of Green St though :)

Will you make that data available once it is digestable? One of the reasons I am so interested in virtual economies is that they provide the kind of experiments whose absence constitutes what von Neumann claimed made the study of economics so difficult. I would love to have that kind of data for SL. If you are interested in collaborating, there is a chance I may be able to make some things happen on my end.

Best regards,
Eric

[Dmitri Williams]

It isn't SL data, and the idea is that long-term, it'd be made available to a broader research community once we take a good whack at it and make sure we comply with an NDA. It's going to be a little while, though. You're welcome to ping me off-TN.

And no worries, I do work on both sides of Green St. ;)

[Clay Shirky]

@Prok:

Clay, I can only keep repeating as others have repeated: 36,000 paid subscriptions, which support some 100,000 regular log-ons. That's never enough for you, but that's what it is.

It's not enough for me, because neither metric counts people. I wonder why you are willing to accept such unrepresentative data.

[Clay Shirky]

@Thomas, since my tone seems to grate, I'll let Dmitri post for me:

Numbers matter for scope and scale. A census is a platform upon which everything else gets built. Once you establish it, you can then drill down and know how the people or situations you are looking at are or aren't representative of the whole or some portion of the whole.

This is what I believe.

@Dmitri, re: "So I'd stick with the 30 days and "active" since that's a billing standard and might actually see the light of day."

Point taken. As with Matt and Eric, I am interested in numbers that discount the 'storm surge' effect, but since that can't last long, the 30 day number will settle down over time.

[Dmitri Williams]

Right, and if you were actually getting that 30-day number regularly, you'd get the surge numbers and trends. In my fantasy world of data, we see these numbers monthly.

[Thomas Malaby]

@Clay: Again, that's fine, but if that's all you believe then you haven't been reading Dmitri carefully. You seem to take this idea about numbers as grounds for generalization as a reason to stop the presses until they are determined once and for all. This suggestion that this is what needs to be done first saturates your claims here, but not Dmitri's. He knows that all of this work needs to be done in tandem; that's what process means. To say otherwise to score rhetorical points is disingenuous.

What is more (and here Dmitri and I sometimes end up quibblinga little over language), claims about "some portion of the whole" can be made without extensive statistics. If this interests you, then I invite you to see my post to TN to come.

[csven]

@csven. Just some time-based metric of money paid to the game maker/hoster. Given smaller units of time, you can extrapolate upwards, but you can't gain granularity going in the other direction.

And I'd argue that one can't adequately discuss virtual worlds on an "apples-to-apples" basis without looking beyond the standard game developer model. Eric's quote from Castronova's study should remind us all that what was so startling to many people *wasn't* what the developer was pulling in, but what was happening among the users outside the space. Consequently, I agree with others here that the economic numbers need to reflect something other than simply what the developer is collecting. If it doesn't, I don't see how this will ever begin to approach an apples-to-apples system.

My suggestion for economic activity would be something more like this:

Economic activity A (Access Gatekeeper):

1) A binary variable reporting whether the user does or does not pay for access to the virtual world.

2) The average amount paid to the Access Gatekeeper per user per week across the world's population.

Economic activity B (World Provider):

1) The average amount paid to the World Provider per user per week across the world's population.

Economic activity C (Real World Developer Centric):

1) The average amount paid to a Developer outside the virtual world per user per week across the world population.

Economic activity D (In-world Developer Centric):

1) The average amount paid to a Developer inside the virtual world per user per week across the world population.

Economic activity E (User Centric):

1) The average amount paid to anyone for anything related to the virtual space per user per week across the world population.

-

{Note 1: The Access Gatekeeper, World Provider and Developer may all the be same entity. I'm considering here that a virtual world might be accessed through a large umbrella gatekeeper (Multiverse 2 or AT&T Worlds or something), provided to users by a separate World Provider who both contracts with the Access Gatekeeper to be a part of the network and with a Real World Developer to build the underlying world content.}

{Note 2: A Developer can be a high-level entity contracted to build the underlying world assets for the World Provider, a low-level entity creating content specifically for users inside the virtual space (e.g. There Developer Program), or anything in-between. The differentiator is in which "space" the economic exchange takes place: in the Real World using traditional currencies or in the Virtual World using the embedded currency system.}

-

Tear it up.

[randolfe_]

There is no legitimate notion of GDP, GNP, NIA, current accounts, deficits, or even real rates or PPP within any of the current generation of VWs/MMOGs.

How does one measure monetary metrics? I've read most but not all the various academic notes on VW economies (as of summer 06), and find little of it convincing.

These "economies" aren't bona fide economies at all. They are, at best, quasi economic abstractions. They more resemble parasites (some might say symbiotic ones) upon the real-world economy, not unlike the "drug economy". The drug economy is not an economy. It contributes nothing to the Solow model (or whatever your particular choice of doctrine) independent of the host economy. It merely distorts the host economy's national income accounting. The same is true of SecondLife et. al.

In fact, all that really matters is the definition(s) offered by Dmitri.

I also don't care to know about the "GDP" of Harrah's. The fact that people are exchanging some kind of money-for-service-entertainment therein doesn't mean that they are generating economic activity deserving to be studied as an economy. All it means is that Harrah's has a P&L.

Last I checked the SL "virtual world" still falls squarely under the sovereignty of the US for both it's corporate operations and a vast majority of it's customers activities. States have a SNP because they enjoy some form of limited sovereignty. They represent legitimate economies, even if they are sub-economies. But SL enjoys no sovereignty, and is not a sub-economy any more than the drug market.

Arguments about VWs like SL somehow representing transnational entities, therefore granting them some kind of special consideration are also not convincing. Modern transnational corporations have existed for over 40 years now. Although some sovereignty/jurisdiction issues remain, by and large these things are old hat. The only reason disciples of VWs think they are different is because the "powers that be" largely do not yet understand what VWs are. Once they do -- probably fairly soon -- they'll be treated no different than any other multi jurisdictional business. It took them a while to catch onto internet poker too.

Better that we just call these things games, measure them as games, and quit pretending they are something greater.

[Clay Shirky]

@Thomas,

You seem to take this idea about numbers as grounds for generalization as a reason to stop the presses until they are determined once and for all.

Not at all. I'd settle for them being determined once.

[Thomas Malaby]

Well, at least now we understand each other.

[Andy Havens]

@Clay, who said: "I am however saying that none of the nuanced questions will be terribly well grounded if we don't have population figures as a start."

I must strongly disagree.

I've been involved in all kinds of marketing research where we weren't able to know raw or absolute "population" metrics (especially for competitors), and you can do insane amounts of good work that doesn't require an understanding of the total size of your audience. And you certainly don't need to know the size of all the audiences in all the competitors in a space in order to compare and analyze all kinds of qualitative AND quantitative aspects of their offerings.

There's brand and name recognition testing. There's "decision process" testing to determine how/when customers move through the pipeline. There's test marketing of any game/world feature (current or planned) using split-run tests and small customer sample groups. There's concept testing and mystery shopping of both operational and customer service capabilities. There's user acceptance testing, UI testing, customer lifecycle research. All kinds of price testing including elasticity, sell-through, coop possibilities, reseller acceptance, affiliate tests, win-back programs, etc. There's segmentation research to see the demo/psychographics of different service's customers, regardless of total numbers (which is another issue in-and-of itself).

None of this -- none of it -- requires gross numbers. Unless all you want to do is make a specific judgment about, "If X number of people use a service, it must be linearly X important. If Y service has 1/100th the users of X, it must be 1% as important."

Which, again, I must disagree with.

You've said a couple times now that you aren't interested in the "guts" of SL. The experience. We know you want to know about the raw pop numbers of SL. Lots of this other stuff we're talking about here -- even many of these numerics -- are nuance, and will require an understanding of the guts.

Because the differences between WoW and SL involve those "inside experiential thingamabobs" that you don't want to deal with. And that includes the fact that WoW doesn't allow for a monetarily free experience that does, however, add value to the overall platform.

So, if you get an "apples-to-apples" paid account comparison in WoW and SL, what you end up with is one of two things:

1. Either a complete dismissal of SL (compared to WoW), because 36,000 paid subscriptions is so much bloody lower than 7 million paid subscriptions. Or;

2. A need to now "ask the nuanced questions" (what I've been calling "context" for the last week or so) that will explain why the press, and many of the foine folks here at TN are so interested in SL, despite the low (compared to WoW) numbers of (paid) accounts.

I pick #2.

[IAmEric]

Hi Randolf,

If you take a snapshot of the world today, it is difficult to disagree with you. No one could debate your statement: "There is no legitimate notion of GDP, GNP, NIA, current accounts, deficits, or even real rates or PPP within any of the current generation of VWs/MMOGs."

Does that render the concepts meaningless? If we extrapolate (on a log scale) the past 5-10 years into the next 5-10 years, is it really so inconceivable to think that virtual economics as a legitimate field of study will emerge?

Maybe I've drunk the virtual koolaid, but I can see no outcome other than massive growth of virtual worlds in the coming years. Virtual economic transactions will soon rival that of some countries. The central bank of China is concerned that virtual currencies will undermine the value of the Yuan. That is a pretty powerful thought.

I'm suggesting that we study the economies of virtual worlds as true legitimate economies. It is certainly fair to criticize the idea, but the field seems very fertile to do some really good work. If I were still in academics, I'd be all over it.

The comparison of virtual economies to black economies is a fair one, but I don't think it negates the idea. The drug economy is an economy that distorts the real economy in ways similar to the way virtual economies will distort real economies. Economists will eventually love to have a handle on both.

It is not necessarily the right thing to read the literature when we should be producing it.

Eric

[Allen Sligar]

I'm back, no longer sleep deprived and well caffeinated. I'm even going to use my spell checker today because I'm feeling frisky.

"Numbers matter for scope and scale. A census is a platform upon which everything else gets built. Once you establish it, you can then drill down and know how the people or situations you are looking at are or aren't representative of the whole or some portion of the whole."

This in essence represents our goal in building a benchmark data set for the video game industry. I haven't gone public with what we're doing in its entirety, although a few developers we're sent an email recently, although I only got 2 responses, :( that had more detail.

@Dmitri: In reference to this:

"Alan is right to suggest repeated measures, and anyone can be right to suggest more depth to any one of the measures... And it'd be unrealistic to ask press or even most investors to understand most of that. They'd just ask statisticians to process the numbers and spit out some coarser stat anyway."

I'll go into how to apply what you’re after, in various levels of utility and granularity depending on the end user needs, later but let’s deal with your proposal now:
-------------------------------------------------------
"I would like to see for universal metrics of virtual world use--something to get us in the ballpark of apples to apples. The intent, after all, is to have agreed-upon baseline numbers which you then argue about being indicative of something."

I'd like to have measures of these basic things for any given world:
Duration
Intensity
Economic activity
Multiplicity
Active users
-------------------------------------------------------

From a less academic and more utilitarian data mining perspective (because that’s what I do) when beginning to tackle this issue I realized the industry lacks standard measures and dimensions. The underlying reason is related to a fundamental lack of a descriptive well defined language. Various academics lurking around here have made some great progress, but let us compare the ability to data mine other industries for actionable data sets for a moment:

Health Care, (descriptive, definitive language, allows for the extrapolation of things like incidence of disease, when cross referenced with demographic information, disease management when cross referenced against financial cost metrics, and predictive modeling for increased ROI on disease management when coupled with programmatic interventions)

Petroleum, (same as above, geography, satellite imaging, drilling technology, allow for understanding, measurement, management and interventions which increase ROI)

Retail, (same as above, demographics, purchase patterns, production and distribution costs etc)

But what of the Video Game industry and VW space?

Where are the standardized metrics? From whence do they issue? My conclusion is we must start with the basics first. Let’s understand that we know that we have an apple tree before picking the fruit....as it were

After all before we can get to higher strata of analysis, to find the nuance, we must first understand the basics.

The first order of definition is demographics of a valid sample size, I'll avoid the debate related to sampling size and validity for now, but go from the assumption that I'm fairly conservative in this.

Demographic strata are fairly easy to lay out; in roughly 15 minutes one can define what ones needs to measure in people. 25 data point suffice.

Now we can get into:

Duration
Intensity
Economic activity
Multiplicity
Active users

Duration:

Without reliance on data from a company (which is limiting and only measures duration of 1 population against 1 space) let us examine duration from a macro level. This will be difficult initially, yes, but constructed correctly provide greater insight later on, and be less limiting.

What I mean here is to get overall measures of market duration for this population segment we'll be finding what their duration was previously, and measure it going forward. The only subjectivity involved is that of population self identification with a given value set. Aka bias.

However, this metric becomes more powerful when we look at intensity as you've defined. It becomes telling when measured against things like churn rate, platform, and type of space.

We are limited to softer numbers here unless we get actual data dumps from companies, have a dashboard that allows for monitoring, like Business Objects, or use SAS, (this assumes they've bothered building measurement mechanisms) or build an integrated software platform (Decision Support System) for companies and that is currently underway actually....because hey why not have our macro analysis (cake) and micro analysis (and eat it too) as well :)

Intensity is a fairly straight forward and soft target to capture, what is elusive is capturing the nature of that intensity, this I've defined internally as "Interactivity", or rather "What we're they doing or what do they do most of the time" there can be A LOT of granularity captured in this area, and I'm certain this will be of interest to designers in particular, for various reasons. I've developed quite a few measures in this area, admittedly I am not an expert, there’s likely 3x more that I have not thought of, we'll get to them as we can and based on input....and maybe a few people will stop by with some helpful suggestions :)

I’ve actually set up a channel (forums) for direct input and suggestions for this purpose from closed beta onward, feel free to drop me a line.

Economic Activity A and B: Here I also believe inclusion of population consumption of games and VW's yields even more value. Churn rates related to interactivity, demographics, duration, intensity, etc.

More importantly we begin to understand more than the "What" of the population but also the "Why" and "How".

Much is owed here to the efforts of Castronova/Malaby/Koster in this area.

Multiplicity: Easy enough and straight forward. There are initially about 10 data points for this. What it comes down to is number of accounts maintained by individual and household, coupled with cost metrics, method of payment etc. and when you add metrics related to price sensitivity, adoption, conversion rates and etc. This area would begin to bear apple to apple understanding as it related to cost, per household, per anum, by VW/game type. And from a more business perspective, where to set price points, when and why....for in depth understanding on data mining this area we owe much to our local grocery store....

Active Users: A hard an elusive target again without those mechanisms’s in place as mentioned before:
Dashboard, SAS or DSS

One has to rely on self-selection related to value sets to get a macro understanding of this measure.

I'm sorry for the length of this post, but it's important to many people, and well, actually it's important to our start up :)

Finally:

"I understand the incentives to supply distort or hide these figures and I just don't care."

Neither do I, its more important than any one company's interests, its important to a large industry, its stakeholders and more important to those who enjoy VW's and Games. The people, who create, spend time, and money deserve to have better information, that’s relevant and actionable.

"And sure, we're all subject to using what companies actually decide to disclose, but that's no reason to not spell out a better standard."

I agree. What we're (GMM) doing is a pretty risky venture, we're going for it anyway. We're apt to make mistakes, because frankly we're learning and testing what will/wont work for our user base, and while I've got a pretty good idea about demand, getting validation is much harder when ones untested and proposing a different approach.

Either way the data will be generated, proprietary interests and obscurity not with standing, because the tools and mechanisms are in place to power the data acquisition, we're not innovating so much as aggregating possibilities and tools, and reliance on dated methods of analysis, (besides being costly) has failed to yield
results. This is our contribution, as gamers, to the solution.

People deserve better VW's and games.....

[randolfe_]

IAmEric,

Those are fair points, and I certainly don't mean to cast anyone interested in studying the economic aspects of VWs as cultists. My primary point is that those immersed in all things VW/MMO can easily miss real-world constraints and frictions.

My contention is with legitimizing the reporting, study, analysis and eventual prediction of things economic using VW economies as a basis. It is folly for myriad reasons. Perhaps with enough advancement of VWs they will evolve into full-fledged economies.

But for now, and I'm betting for a very long time and maybe always, VWs represent *markets* and not *economies*. These terms are not interchangeable, one being a subset of the other. As a study, it is the black _market_, not the black _economy_, even though we may call it that for popular clarity. VWs are also merely markets; specifically in terms of RMT.

China and others worry about this because transactions currently fall outside of established regulatory mechanisms. The US also worried about internet barter for a while, but that was just another market. Once they established tax code and GAAP rules regarding virtual transactions, no one much cared anymore (and most of those profiting off of the novelty of it evaporated).

I will revise my earlier indictment of studying VW economies as such: study of microeconomics within VWs is perfectly legitimate. However, there is no macroeconomic there there.

[Dmitri Williams]

@Andy. I still disagree, but probably not in the way you think. You mentioned the importance of test marketing, focus groups, acceptance testing, UI testing, customer lifecycle research, reseller acceptance, affiliate tests, win-back programs, segmentation research, etc. All of those things are valuable, but none of them are as valuable as they can be if you don't know the total population first.

You get some interesting leverage on phenomena, but then you don't know if it's generalizable or not. And that's a waste of a good opportunity. Any one of the tests you mention here can be made more useful if you also know if the sample used is representative or not. And you can't know that until you've had a census--outright or sampled itself.

As to using or not using some system because it will or won't make SL or other worlds look good or bad, that is for SL to worry about. If honest brokers report fair numbers and also give information about how to interpret them, then people will make of them what they will. I look at your point #1 and think, "Well, SL has fewer subscriptions, yet seems to have a lot of interest. I wonder how LL is making their money. I think I'll look further." And then I'd notice the numbers for Duration, Intensity, etc. and start thinking about how and why that world is or isn't fundamentally different. Will ignorant or malicious people misuse statistics? Of course. That's where experts are useful. But you still need to at least give the experts the data, or they're guessing in vaccuums.

Lastly, none of these statistics should be compiled or reported without some understanding of how the virtual world works. Without that, the numbers can fail at telling the whole story. This thread is simply about getting the numbers right.

Hi Dmitri,

I'd like to throw a thought out there and see what happens. You said, "I look at your point #1 and think, "Well, SL has fewer subscriptions, yet seems to have a lot of interest. I wonder how LL is making their money. I think I'll look further." And then I'd notice the numbers for Duration, Intensity, etc. and start thinking about how and why that world is or isn't fundamentally different."

One of the reasons why I think SL is different (admitting that I know very little, i.e. nothing, about other virtual worlds) is that I think it has grown to be bigger than LL already. When you talk to executives at LL, very few of them have a clue what is going on in SL. I believe I learned from one of Castronova's papers, but can't recall the reference, that there are precedents for the residents of a virtual world to actually take control of that virtual world even if the parent company goes bankrupt. If SL hasn't reached that critical mass yet, I think it will very soon.

How might that kind of dynamic be captured in numbers? I don't think the metrics proposed so far address that (granting they were not intended to). However, I think that kind of internal economic, social, political dynamic within a virtual world is something equally worthy of study. In the case of SL, I actually think it is the dominant factor to consider. I think the parent companies will become less significant to the future of the virtual worlds they created as the populations increase. In some sense I think the internal dynamic is more interesting and more important than the external dynamic, e.g. how much the parent company is making, etc.

Best regards,
Eric

[Dmitri Williams]

I think it'd be very unlikely that SL would continue if LL were to fold (which, BTW, is not about to happen). It's an expensive service to provide, and the population hasn't shown the willingness to pay for that service. Those very numbers about paid subscriptions/total subscriptions have to at least be considered a hint that the cash flow in from subscriptions alone won't suffice.

Now whether some internal engine (or an exogenous exchange-based one) would suffice is another question, and it's something that I'm sure could be measured.

As to whether the internal numbers would be more interesting or relevant for SL, I'm not arguing. They might be. I'm still interested in getting some basic population numbers as a starting point. Consider the premise of your suggestion: the users get to critical mass and take ownership of the world. OK. How do you know what critical mass is? Is it a % of a whole, some absolute number, what? You still need the data to make the argument concrete.

[Andy Havens]

@Dimitri: You make very good points. And, in the abstract, one would also like to be "the invisible eye" and be able to do testing that doesn't let the subjects know you're doing testing. We'd also like to be able to do research that eliminates all variables that are unlike from system to system, so that we can study, for example, "is voice chat a compelling feature in MMOs/VWs regardless of in-game task, and what (if any) price elasticity might be related to providing that feature?" It's very tough, though to remove all "sideways" variables from any single-point study.

And that includes gross user stats.

Again (and I seem to repeat this in every comment), I'm not arguing *against* getting good pop numbers. They'd be great. I just want to make sure we don't sing (as Clay seems to be doing) a chorus of, "Head Count Above All Else."

I spent 10 years in marketing in cellular telephony during the time when it went from "rich toy" to "everyman necessity." During that period, we learned a lot of hard lessons about how to count (and how not to count) heads, and how different kinds of heads meant very different things.

One example, for example; prepaid phone service. For a long time, in the earlier days (until about 1997), it took about 6 months on average for us to earn back in ARPU the up-front cost of acquiring a new subscriber. That's one of the reasons we required a 2-year commitment. Regardless of the cost of advertising, the buy-down on the phone (we gave them away free; still do in many cases) plus other set-up and sales spiffs was that high. And that doesn't count advertising COA, which you don't figure into your earn-back, because every product has an ad COA (ours, at the time, was around $60, which net-net, would add around another 2 months to the earn-back; like I said, though... don't count that; think 6 months).

So... if it takes 6-months to make any profit on a new sub, and we want a 2-year commitment, we required a credit check. Bad enough credit? You don't get a free phone and you don't get to burn us. Then along comes prepaid service; you pay for the phone up-front, you pay for a block of minutes, when they run out, you pay for more. Much less of a credit risk, because we only pay for about (if I remember correctly) about 2-months up front in sales/co-op licensing and other fees.

So, at first, our big thought was, "Here's the answer for credit-risk customers that were falling out of the bottom of our current, standard, advertising/sales pitch. You come in for the regular deal-i-o, don't make the credit score, we sell you on prepaid." Made perfect sense to us.

It took us quite a few months to figure out that, though those folks were a small portion of the audience for prepaid, the biggest was, in fact, NOT the "credit challenged," but the "I hate getting a surprise cellular bill every month and want to control my costs even if those costs are a bit higher" crowd. And that crowd is a lot, lot bigger than the low-credit gang.

We were looking at a "gross number" -- folks who didn't make our credit score for post-paid -- and thinking, "this is our audience for pre-paid." We had the number right. We did all kinds of research based on that number. We asked those folks a ton of questions about how much they'd use prepaid, what they'd pay for it, how they'd like to use it, where they'd buy it, etc. etc.

But it was the wrong number. And it was dinky compared to the real number. And we ended up marketing to the wrong people; bad-credit folks with a little money to spend on a low-end product for the credit-challenged, instead of a shiny, happy product that you can use to control costs (even if that control-level is as high as some post-paid bills), give to your kids, etc. etc.

Good research on the wrong number, even if the number is accurate, is as much of a stank as a bad number.

And this is my problem with Clay's "Big Beef."

Ya, ya, ya. Linden's 2-million is bogus. But he also seems to want to compare whatever the "real number" is to WoW, LambdaMOO (?), "social software" (email? blogs? Mososo? dating services? wikis?) in general using the raw number.

And that, to me, is ridiculous. As is the assertion that he doesn't care what the experience is like; i.e, the "features." Because that is, essentially, an assertion that all social software is the same.

Right. Just like all books are the same. And all web sites. And all blogs. It's just words and pictures. So we can judge them based on one gross metric: popularity.

Social software is *not* all the same and serves incredibly different purposes depending on the audiences, the intentions, the features, the function and the goals. Good grief... Does anyone here really... REALLY think that WoW should be compared in any serious way to eHarmony.com (which has over 12 million registered users, btw)? Even though there are stories about romances that bloom inside Azeroth? That particular "social feature" of WoW is incidental to its function, which is to play an RPG.

Hell, I'm sure that there are romances that have jumped off Wikipedia. I fell in love with my wife in my high-school cafeteria. That doesn't make the lunch lady a yenta.

You want "standard metrics" for MMOs and VWs, my point is that you need to decide the *goals* for those metrics. Back to my prepaid story; we got the goal of our prepaid metric deeply, deeply wrong: we counted potential add-on customers from our bad-credit pool. Nope. Wrong number. And by counting that number very well, we failed (for awhile) to count the right number: good credit customers who would spend MORE money with us if given an alternate billing option.

So... Here are my questions for the learned crowd:

1. Do you really want to count a person who is contributing content to a VW the same way that you count a person who is playing an MMO? Because the loss of an MMO player is, essentially, straight churn (w/ the usual "may drag a few buddies with him" caveat... then again, for a griefer...). The loss of a "creative customer" in a VW will actually bleed off your content. Example: what would the death of Anshe "as player" do to SL? Is she really even considered a player at this point? If she decided, "Screw it. I'm outta here," what would the affect to SL be, compared to a newbie land-holder? But under the current proposals being discussed, her leaving SL would count as, "-1 to active users" of the system. Seems, to me, to be an "accurate count of a wrong thing."

2. If a VW is earning revenue through the sale of virtual items or land, how do you (or do you) track that against revenue from account fees? For years, we gave away free phones and made a metric ton of money on the airtime. "Give 'em the pipe, sell 'em the tobacco." Very old tune. So... if you want to measure economic activity for reasons of investment, financial health of the company, etc. etc., you need a way to really, really look at the "dollars go in, dollars come out" of a system like SL vs WoW. And "value goes in, dollars come out," as well. Because lots of what happens in SL that costs the users nothing, still generates "value for Linden" that, in turn, generates eventual dollars for Linden. It is the equivalent of "earned media." I'd call it "earned content," or some such nonsense.

3. Since we are talking about the "social-ness" of these spaces, some kind of metrics to include the ability to interact with others in the space would be, I think, a good idea. Is there a way, for example, to form groups, clans, guilds, etc. in game that persist from session-to-session? (binary yes/no). If so, is there a limit? If so, what is the average usage of such? There have been for example, stories of "Playing alone together" about WoW; i.e, how it is possible to play much of the game as a single-player adventure. I've played lots of my characters up to 15-20th level without doing much grouping. So... is there a way to say, "Most users of this service engage, on average, in X number of in-game relationships?"

If, for example, we found out (makin' stuff up, here) that yer average SLifer was a member of 12 groups and had 22 friends, but that WoW players were a member of one guild that had 9 members... Well, it's a bigger game, yeah... but much less "social." That's good to know from a potential customers standpoint, eh? A larger city, but much less friendly, eh?

4. Economic potential: Can you make money in-game. And I don't mean out-of-bounds RMT. If you need to use eBay, it don't count. If I want to make money right now, can I do it in WoW? No. SL? Yes. Project Entropia? Yes. So, in a sense, can we measure "player earnings in real dollars" for those players who choose to play "the economic game" on top of the "VR stuff?"

5. Multiplicity: I'm going to go out on a limb here and suggest that it doesn't matter. I, as a human user have only 24 hours in my day. If I want to have 2 accounts and pay for them, and one of our measures is paid accounts... I count as 2 paid accounts. If I log in as "Character A" for 2 hours a day and "Character B" for 1 hour a month, "A" and "B" each go into the "time played per character" measure accordingly.

I can subscribe to an almost infinite number of MMOs and VWs, eh? I can count as a WoW sub and an SL resident. I can be a Puzzle Pirate and a New Pet. As long as I am going to be counted again and again and again in multiple worlds -- and as long as we get our metrics straight for how we're going to do our counting -- I don't think we can worry about multiples in one world. If a platform allows for multiple accounts, they count. Now... that's different than counting multiple "toons" or "avatars" per account. Having multiple toons or characters is essentially a feature of the system. I can see a game where you get to rotate between hundreds; that doesn't signify anything. But accounts is different. You want to log on today as "Fred" and tomorrow as "Ginger?" Fine. You count as two people... but only as half a person each time in terms of the appropriate usage stats. That should work out in the wash. If it doesn't, our numbers are bad/wrong. Aren't they? Because somebody who logs on twice as long as one person (maybe as two toons?) is providing just as much "stuff" to the system as someone who logs on for the same amount of time as two. Right?

Which is the old conundrum... Do I want one $100,000 customer or ten $10,000 customers?

Do I want one 10-hour-a-day player, or five 2-hour-a-day players? And which is more valuable?

One vocal troll or ten silent audience members? Who counts more?

6. Are you going to count re-activations the same as new subs? IE, if I join a service, play for 6 months, quit for 6 months, and come back... is my return counted the same as a brand new guy?

That's important, because the ability to retain and retract lost subscribers speaks to long-term success. Also, it's more expensive (generally) to attract new subs. A service that does well in getting back lost subs on top of getting new ones... that's good eating.

7. Are you going to count involuntary churn in your churn numbers; i.e., people that kicked off by the service for no-pay, bad behavior, etc., or just people that quit under their own steam?

8. Are you going to have any kind of lifecycle measurement to quantify subscriber levels or loyalty or participation? By curve or bucket? For example, some comment was made that some percentage of SL players spend 80 hours a week on the grid. Yowza. That's heavy use. I'm sure WoW has folks at that "tier" of usage, too. Do we want to quantify just "intensity" or any of the other measures, too?

9. Sector metrics. Specifically, are joiners of a particular system "growing the pie" or are they coming over from another platform? Much of the early growth in cellular was totally new to the medium. Now? It's jumpers. In MMOs/VWs, we've all made this assumption that WoW players are new to the MMO sector, and that may be largely true. But are they new to gaming? What about SL? Are these folks totally new to the whole idea of cooperative, technologically enabled "play?" If so, that might explain some of Raph's recent entries about the insularity of SL's blogs. This would also involve measuring "entrance/exit" (what did you play before and after this) and "multiple platform" stats (what else do you play).

That's what I've got in my head at the moment.

[IAmEric]

Hi Dmitri,

This discussion has been fascinating. It has taken many turns and spans across multiple blogs and multiple threads within each blog. Just to clarify one point, I agree 100% that getting at a meaningful number to represent the "population" of a virtual world is extremely important. I don't know how to link directly to a comment, but if you look at my initial foray into TN comments here:

http://terranova.blogs.com/terra_nova/2006/12/shirky_to_tn_re.html

(I guess just search for "IAmEric")

you'll see that we were trying to come up with a definition for a true "resident" of a virtual world (I had SL in mind). I thought that if enough smart people got together, we could come up with a satisfactory definition of a SL resident (unfortunately LL has already used the term "resident" to mean "registered account" which is NOT what I'm talking about).

The term "population" likely has various meanings for the various virtual worlds and it is likely to have no meaning at all for some. However, I also think having a meaningful number that is meant to represent the population of a virtual world would be useful.

Do you have any suggestions?

Perhaps, like Andy suggests, we should try to define more clearly what we are trying to get at by "population". What should the population number really represent?

My initial attempt was:

"A Second Life resident is someone who maintains a recurring and regular presence within Second Life. Their account must be at least 30 days old and they must have maintained an average of 3 hours in world per week since the creation of their account."

Then tateru offered this suggestion:

"I might have a looser definition of Resident than you. I'd say someone who spends time at least once every three months qualifies. Let's put a number on the hours. Say three hours, every three months. If I were running a bar, I'd call that a regular. I'd hope for a lot of those, rather than trying to rely on my die-hards."

Based on some feedback, I offered this refined definition:

"A Second Life resident is someone who maintains a recurring and regular presence within Second Life. Their account must be at least 3 months old, and they must have maintained an average of 3 hours in world during every 3 month period since their account was created excluding the initial 1 week period."

Tom offered this modification:

"A Second Life resident is someone who maintains a recurring and regular influence within Second Life. Their account must be at least 30 days old. In addition, they must have maintained an average of 3 hours in world per week since the creation of their account OR have performed an average of at least US$X per month in net financial transactions over the past 3 months."

I'm not particularly satisfied with any of these definitions, but its a start. I'm hoping someone with more clarity of thought than I have to come up with something more succinct that is agreeable to most reasonable people with an interest in the subject.

Once we have such a satisfactory definition, my proposal is to present our definition to LL and ask them to provide us with data, preferrably a time series, for the true population of SL as defined by us. They have no obligation to grant our request, but I don't see what harm it would do and from what I have seen of LL, they do not seem like unreasonable people and would likely grant the request.

What do you think? If each virtual world could come up with a coherent definition for "population" (it will likely be different for different virtual words), then that number should be roughly comparable across virtual worlds. It seems like it would be the easiest number to get at (aside from the meaningless number of registered accounts).

Best regards,
Eric

[Allen Sligar]

"I fell in love with my wife in my high-school cafeteria. That doesn't make the lunch lady a yenta."

Andy, your always good for a great line at the right time...thanks

"That's what I've got in my head at the moment."

All good stuff and a great post, you know you bring up some things I've missed myself, and some things that "are frankly not within our scope at the moment, and some things that would require deeper analysis than is possible by an external organization without proprietary data.

"You want "standard metrics" for MMOs and VWs, my point is that you need to decide the *goals* for those metrics."

True, IMO that consideration (goals) should be independant of the data set to maintain neutrality and integrity for comparative analysis purposes. Thats apples to apples.

This is not to say that deployable solutions are not available, much of this depends on your preferance of vehicle (quantitative/qualitative). Various slogans apply to this metaphor of course....Drivers Wanted, nobody rides for free, etc.

have a good new year...

Allen

[Ken Fox]

All of Dmitri's statistics could be collected without the help of the VW owner.

Duration: This requires account identity which is not recoverable without decoding the VW communication protocol or installing special software on the client. http://www.xfire.com/ is an example of an add-on friends list that would provide account identity.

Intensity: Average bytes per minute upload traffic. This number must be callibrated to specific VW communication protocols.

Economic activity A: Advertised feature.

Economic activity B: Advertised number. May be hard to figure out promotional discounts or special deals.

Multiplicity: Simultaneous connections per IP address. Most people running multiple accounts use broadband Internet and a hardware firewall. The firewall appears as a single IP address, but simultaneous connections can be distinguished based on the TCP connection id.

It's not necessary to have the VW owners' permission or cooperation to collect any of that data. Decoding the connection data may violate a TOS, but this is a third party not bound by TOS.

The Internet service providers can easily publish this information. It may take a government request, but I bet a consumer group could pressure the ISPs for the data. Lately the ISPs have been rolling over and providing data to anybody.

RIAA probably already knows the names and accounts of every LL user. ;)

[Ace Albion]

Well, from now on, all being well, I won't be paying money from my CC to use SL again. I haven't gone basic, I just have enough L$ income now to cover my costs when exchanged to USD in my SL accounts.

I don't know how you measure this type of activity... You could still say- this account pays us $72pa plus $40pm etc and not worry about where the USD to cover that came from. Except that L$ income was, in some cases (maybe most) provided by people who bought L$- either as the stipend as part of subscription or from that Linden account that prints money from time to time to sell on Lindex. So in effect, avatars Angie, Bob and Charles are subsidising my SL- they're paying my sub and my tier. And we're possibly double counting money being paid to LL as the currency gets exchanged and back through subs/lindex.

Add some more confusion- I'm the named owner of an island sim, but I only have 1/8th share I truly call my own. So personally, it's about $25pm alongside a downpayment of $150-ish. Except that I, on one account, paid LL $1250 and $195pm. That money is actually the funds from six different people/accounts, funneled through me, and provided again by transfers of inworld currency to USD against my account- the other islanders get their money again from people who buy their stuff etc. Counting me as one paying subscriber (actually I have three premium paid subscriber accounts now) just seems way too wrong. The content/service providers in SL are filters through which a bunch of consumers- *many of whom pay nothing direct to Linden Lab- I have No Payment Info On File customers* are in effect, invisibly paying for SL. They are also paying Anshe etc in paypal offsite to lease virtual beachfronts. Anshe is "users +1" filtering that through to LL as one tier payment for one island, per 12-16 people or whatever.

I guess it's like imagining that WoW or something like that had some "guild subscription" option, where Mighty Guild of Shiny pays whoever runs WoW $1000 a month and then makes its own arrangement with guild members as to how much they pay the guild treasurer/accountant. It's the same kind of idea. It's also all off the books, except for "Treasurer Elf Guy is user +1 and pays us $Xpm".

I don't know how you can compare that kind of business against "this girl = $15pm, that guy = $15pm" all lined up in a row.

There's another economic effect of all this- In world effort funding subscription. I paid up my most recent alt to subscription because I had enough L$ I could transfer to it and I didn't have to worry about cashing that money out and dealing with tax etc- some people work SL this way- buying (leasing) islands etc all from inworld graft. I think EVE has something similar with its "game time codes" that people buy for actual money and sell for in game money. So you can earn in game to pay. That's the RMT stuff people find hard to count or prefer to not count, but it's intrinsic to SL for sure. Those are my anecdotes anyway, just to show how I don't think SL is ever going to be apples in this orchard.

[Clay Shirky]

@Andy,

Right. Just like all books are the same. And all web sites. And all blogs. It's just words and pictures. So we can judge them based on one gross metric: popularity.

Andy, you are now putting words in my mouth. I have never said anything about demographics being the only metric -- there are lots of people working on economics or user experience or whatever, and I have never once proposed that those people are wasting their time.

I have proposed, and still believe, that that work would be better grounded if we had good demographics, and that the adoption rate, which is to say aggregate social judgment, matters, and it matters in a way that is comparable across platforms.

Think of social judgment on a two-axis chart: Try (denominated in integers), and Adopt (denominated in percentage points), where 'adopt' is a measurement of longer-term use.

There are services that many people try, and a high percentage of those adopt. We'll call those 'popular.'

There are services that few people try, but a high percentage of those adopt. We'll call those 'niches.'

There are services that few people try, and a small percentage of those adopt. We'll call those fizzles.

And then there are services that many people try, but a small percentage of those adopt. We'll call those hyped.

Your consistent claim has been that these kinds of comparisons are meaningless, but they are not, and the proof is that the press is reporting these figures as evidence that SL is in the first category.

And my complaint, consistently, has been that a) we don't know what category SL is in, and b) that the press, not Linden, has been the proximate source of that unknowing.

Your objections about the subtleties of the numbers would make sense of the press were making subtle assertions, but they are not. As I said in another thread, it cannot be the case that SL is both the future of the internet and not worth reporting right.

[Allen Sligar]

@Clay

I'm going to have to agree with your breakdown on adoption/try (I rather prefer "aquire").

This specifically:

"And then there are services that many people try, but a small percentage of those adopt. We'll call those hyped."

Is prcisely why SL is confusing, and may both be the "future" and not "worth reporting."

We can agree that SL is a valuable VW space even given all its fualts and that its gotten a lot of media, and that this media has mostly been numerically "challeneged".

Also look at the means of aquisition, DD of the UI to the client, no real barrier to multiple account creation, etc. Thats a no brainer on population inflation.

These terms are commonly used with some meaning:

Registered/Subscribed/Active/Concurrent.

However its pretty well understood that when using DD as a mode of aquisition for new users the conversion rate from registered (I downloaded the SL client) to Subscribed (I have a premium account I pay for) is dimsmal, really seriously dismal.

Is doesnt matter if its SL or Orcs n Gnomes Online, the conversion rates for digital download games are low. Because DD becomes a volume distrobution game at that point, your not competing on price point (because aquisition is free) your competing on features (to garner adoption)

Therfore SL may in fact be the future and not worth reporting, at least the conversion rate issue isnt news to most people who follow games.

Although maybe its worth reporting to a different target audiance....

I am not an expert on such things merely an observer.

[csven]

@Clay - "we don't know what category SL is in"

What are the choices?

[Clay Shirky]

@csven: the choices I'm proposing are popular, niche, fizzle, hyped. The breakdown is in the above post.

[csven]

@Clay - The problem I have with that breakdown is that I don't see those as categories in and of themselves. For me, adoption can mean different things depending on perspective. The assumption here seems to be that SL is *like* a videogame and *not like* a long-term use business application. I've often said I consider SL to be closer to a PLM, and it's not at all uncommon to hear of companies "adopting" a PLM platform, because for them the timeframe suits that term. I've never heard of anyone adopting a videogame. Not that it can't occur, but I don't think in those frames of reference in that way.

Years ago I chose an application. I've not used it seriously in over eight months. Have I adopted it? Your "repeat user" criteria makes that questionable, but the answer is a most definite "Yes", because I'll be returning to it... even if after a longer timeframe than most feel appropriate within a videogame frame of reference. I can do that pretty easily because it's open-ended, it doesn't rely on a host (which could go bust in the meantime) and it isn't a game with a pre-defined endpoint so that I can't continue using it in a way that's meaningful to me. Assuming that LL open sources it, Second Life is more like that application to me than it is to a videogame like WoW. Hence, trying to shoehorn SL "adoption" into the same frame of reference with a MMOG doesn't work for me.

I've been, by your definitions I'm fairly sure, a non-user of SL for the better part of a year. Yet I pay a monthly fee. Does that make me a "repeat user"? No. But I intend to return to using SL. Does that indicate some level of adoption? Yes. Do I intend to use SL in five or ten years whether it's a hosted service like today or an open source platform? It doesn't matter. The fact that this question is even able to be asked is, for me, a game changer.

Maybe the real issue isn't the word "Adopt", it's that you use the word "Service" which suggests to me a developer-centric/hosted-control mindset. However, if SL is open sourced, what then?

[csven]

@Clay - I thought I'd add this example. The application I "adopted" also went through a very heavy "hype" period in the mid-90's. It was, when I now think of it, not too dissimilar from SL. And a very large number of people reacted negatively after trying it. And they dumped it swearing to never return to it.

That application is still around and lo and behold, it's now becoming adopted by the same people who called it out as being nothing but over-hyped crap. The application didn't really change; they and their perspectives changed.

[Andy Havens]

@Clay: Your most recent comments directly above (about the placement of a service's popularity vs. adoption on the 2D grid), and here about how your motivation for all this is to keep companies from diving in and using SL as a platform to do things its clearly either not meant to or ready to do... come much closer to convincing me than anything else you've said.

If you'd started out your first article with a description of that grid and the categories: "popular, niche, fizzle, hyped," and your line about carrying ammo for programmers... and then making a statement to the effect:

"The recent over-reporting of SL's usage numbers, caused in part by Linden's shady term 'Resident,' and in part by the press' unwillingness to light up that shade, is leading many new users -- both individual and corporate -- to treat SL as a 'popular' service, when, in fact, it may still be in the 'niche' stage."

That statement, which (I think... I'm really, honestly not trying to put words in your mouth... just paraphrasing, 'cause that helps me process) seems to sum up what you've said in the last couple comments, is a lot easier for me to wash down than the, "Linden is intentionally snowing a gullible, chump press and a rube public that can't figure any of this out for themselves."

If you'd gone one step further, and added something like:

"Many of these new 'Residents' simply may not be sticking around. And this will be, in the long term, just as bad for Linden, Second Life and the VW community as it is for the readers of inflated statistics. Because any user or company who bases their expectations on gushing press tales of virtual crowds that haven't materialized is going to expect more from their Second Life than Linden can deliver. If you go into a park expecting Disney, but only get the County Fair... well, you're bound to leave disappointed."

That would have put it in terms that made the downside even more apparent in the context of your "popular, niche, fizzle, hyped" categories (I think).

I honestly hope I'm not badly characterizing your argument; truly, just restating for my own brain to get around. As a writer, that helps me to get closer to my own thoughts.

And I'm much closer to your POV than I was this morning on this. The "carrying ammo for programmers" and "grid" arguments are good ones. As Thomas can tell you, I embrace The Beginner's Mind; ie, I do admit room for my own instruction and change of opinion. But only after long, hard fights with smart people. ;-)

I'll tell you the two spots I'm still not on board with you, and you can decide if they're worth pounding any more on, or if this horse is dead.

First... your grid is 2D. I suggest a 3rd axis: "influence." There have been, throughout history, any number of truly popular services/platforms and other thingies (ooh, jargon) that have not ended up having much influence. People like vanilla. It's always there. It's popular. There are all kinds of popular pursuits, entertainments, etc. that deserve less attention because, well... they're the damn same as what came before. MySpace is hugely popular. It's clearly been hyped, too. It ain't niche and it ain't fizzled. But I would argue that its importance and its influence in how we work/play/engage with each other online (especially in the VW space) is, in the long-run, going to be rather low. We can get into that later, but I'm using it as an example. SL, on the other hand, even when it was (and still may be) niche, does many things that are and will be influential, I believe, to how the VW space will develop. Some of those influences are negative, some positive. But the "influence" axis helps provide some of the context that I keep braying about; it lets us address the fact that, for example, SL has been getting great press long before it had "popular" (if fake) numbers. It was niche/influential. There are many MMOs and social communication platforms out there that are niche/non-influential. Grouping SL in with them might be just as detrimental to getting the whole story right for the coders for whom you carry ammo as getting the numbers wrong.

Second, I'm still not sure that the "popular" number is what pushed the SL stories into the forefront of the press as much as your original angle asserts. As I've pointed out a couple times, there was a ton of press around SL back when even their bogus numbers were goofy-low. So the story, at that point, was still really a "this is niche, but really interesting" one. And the meat of much of the press hasn't really changed much. Yes, the numbers may be bigger and more inaccurate, and they may paint a "popular face" on what is still a niche product... but the description of the function and features is still very much the same as what we've been getting all along.

Now... those two points being made...

As a marketer, writer, poet and all-around loud-mouth, I understand that in order to get action in either the MSM or the blogosphere, we often have to abandon subtlety in favor of vim. If part of the strategy of your original articles was (is) to abandon subtleties such as "influence in context," and "the relative importance of the point I'm trying to make balanced against several others," in order to really bear down like a mutha wombat (no offense intended; I love me my wombats) on one single issue...

then we have nothing left to discuss. 'Cause I know that ya gots ta do what ya gots ta do.

With the caveat that non-subtle arguments are going to get picked to death by grognards like us on blogs like this until doomsyday.

Either way, I'm having fun. And, as I said, your last two comments have brought me at least 37% closer to your position on this. Good shew.

[Allen Sligar]

@Dimitri et. al.

Wondering if this thread has concluded what standard metrics are desired....I'd like to check them against the data points in our data base and include any I might be lacking. I'll happily provide you with a data dump, or summary stats of you need after closed beta ends (although the sample size wont be representative but an example data set, because its a beta sized population!).

If your interested the only requirement I have are these:

At least 5 but no more than 25 data points, include valid values for these data points, and make sure they are both granular enough and mostly usable across multiple data points.

If you can meet these requirements I can adjuest them for uniformity against our system and load them up (it only takes about 10 minutes).

Shoot me an email if you want to discuss this off the thread. ceo{at}gamemarketmetrics.net

Allen

[csven]

The one I'd like to see is one of the one's I listed above: The average amount paid to anyone for anything related to the virtual space per user per week across the world population.

[Allen Sligar]

@csven

There are assumptions in your metric, but I'll break it down as follows in easily digestible text:

Do items in the VW/Online Game you play cost real money? Y/N

What is the averege amount of real money you spend on VW/Online game items per week. (X.00-X.00 amount) (Possibly 0-200.00? can we assume consumption over 200.00/week is an outlier?)

This simple metric you've outlined brings issues and is dependant on other captured metrics. However this elicits some results:

1. # of people who participate in a VW/online game with RMT

2. Of those people in 1, who do participate in such spaces, how many spend money?

3. What is the breakdown by expendature by %(I'd be breaking this into 25.00 ranges or rather 8 valid values such as: 0-25.00, 26-50.00 etc.).

4. We would extrapolate these against other metrics we've already captured to get an "apples to apples" comparison.

As you can see simple metrics often demand other considerations, this is all very mundane low brow and utilitiarian for TN however to maintain data neutrality and integrity of a data set it's important not to make value laden (such as your population always participates in a VW economy, and understand what RMT is) assumptions.