Techdirt. Stories filed under "metrics"

There's No IP In Team: How Protectionism Is Holding Back Sports Metrics (And Everything Else)

Timothy Geigner — Thu, 28 Mar 2013 11:02:01 PDT

If there is a single place where the sports and the geek worlds collide, it is undoubtedly in statistics. It's long been said that baseball is a thinking man's game, in part because of the chess game that is built into its very skeleton, but also because of the role that math and numbers play in terms of making decisions on each team based on individual situations. By this time, only those that work really hard at staying away from baseball will fail to recognize names like Bill James or Billy Beane. The people now most responsible for constructing teams and their strategies are people with advanced degrees in fields like economics and statistics. What's interesting is how quickly advanced metrics, or sabermetrics, have exploded in use and depth in the past ten years after being almost universally derided by the major league clubs. Advanced stats are everywhere in baseball now, from the early focus on OPS (On-Base Plus Slugging) to WAR (Wins Above Replacement) to WRC+ (Weighted Runs Created) and so on. What's amazing is how far behind other sports appear to be in developing their own advanced statistical systems. Take basketball, for instance. It would be very easy to conclude that there has been nothing resembling the development of baseball statistics in professional basketball, otherwise we'd have heard about it and the knowledge of it would have spread as wide as it has in baseball, right?

Well, no, actually, and the reason why is a lesson in how collaboration, open development, and building off of the ideas of others provides the most advanced outcome. Such is Jason Schwartz's conclusion in his lead up to the Sloan Sports Analytics Conference, where at least some discussion of basketball metrics is occurring. That conference, now an ESPN sponsored event, grew out of what was once a simple Yahoo message board started in 2001 by basketball stats geeks. Early on, as was the case with baseball metrics, the forum was open for discussion, peer review, and the exchange of ideas. Unlike baseball, however, the NBA knew all about Moneyball by 2003 and teams were extremely interested in the potential of advanced metrics.

The NBA establishment quickly took notice. [Dean] Oliver, who published the seminal Basketball on Paper in 2003, seven months after Moneyball hit stores, was hired full time by the Seattle Supersonics in 2004. Another frequenter of the board, John Hollinger, was hired the following year by ESPN - and recently became a vice president of basketball operations for the Memphis Grizzlies. Hollinger's ESPN gig was filled by Pelton, who, after making his name at Basketball Prospectus, did a consulting stint with the Indiana Pacers' front office. Roland Beech, who created the popular website 82 games, was hired by the Dallas Mavericks in 2009 as director of basketball analytics. (His boss, Mark Cuban, is regularly one of the biggest names at the Sloan conference.)

So you're probably thinking, "Great! The teams took notice in the early stages, unlike what happened in baseball, meaning that the knowledge was embraced!", right? Well, that's true, but the result was the severe retardation of growth in basketball statistics. Why? Well, if you know anything about how patents and intellectual property often function today, you've probably already guessed.

As soon as each statistician joined an NBA squad, sharing in public became off-limits-and so, gradually, the think tank closed shop. What were the teams paying for, after all, if their new stat gurus were just posting their ideas online for the other 29 franchises to read? This has had a paradoxical result: Because NBA teams embraced advanced stats so quickly, progress on basketball analytics has actually slowed down. The top minds are now all working in silos, not only unable to collaborate but actually competing against each other.

This is, again, the exact opposite of what occurred in baseball. For baseball statistics, because teams were not impressed by the idea of advanced metrics, favoring instead old-timey scouts on the ground, the best minds were free to collaborate with one another, forming what are now some of the most prestigious sports stats think tanks in history, like Baseball Prospectus and FanGraphs.

Major League Baseball teams were hidebound enough to ignore Bill James and sabermetrics for a full quarter century-as a result, he and others hashed out ideas out in open, public forums. By the time MLB executives finally embraced advanced baseball statistics, the movement was fully formed.

If you want to draw the obvious analogy, baseball statistics were developed on an open source model, while basketball has mostly been proprietary. As Schwartz notes, it isn't necessarily a lack of knowledge that is the resulting problem, but rather the issue is that this knowledge is all segmented throughout individual teams and nobody has the collective manpower to use it to its full potential.

Many, including Oliver, believe the killer app is hiding in there somewhere. The challenge is that there's so much information, it's easy to get lost. "It's like saying you're going to Wal-Mart or Ikea to get something," offers Tommy Sheppard, the Washington Wizards vice president of basketball administration. "You better know what you want, or you're going to walk out with a ton of s***." That each franchise is working alone - and against each other - compounds the problem. Goldsberry describes it as 30 "micro-CIAs," all racing against each other to "procure actionable intelligence out of these haystacks of vast data."

Sound familiar? Now, here's where it gets really fun for the purposes of our analogy. The quality of team construction in baseball is leaps and bounds ahead of where it was 20 years ago, in massively large part because of the explosion of advanced statistics and the resulting understanding of the game. Think about that for a moment. Even as these teams compete with one another, because of this open source statistical model for knowledge of the game, every team is better off for it. The game has universally advanced. Basketball, however, under the proprietary model, has not. While there have been rule changes that have influenced how the game is played, player evaluation is still essentially the same game it was 20 years, or even 40 years ago -- and thus you still end up with teams that look good on paper based on the old stats, but fail to perform well as a team. Why? Well, perhaps because the best minds aren't collaborating to advance the game through knowledge, and thus they're measuring the wrong things (and optimizing for the wrong things as well).

Thinking of each league as a microcosm of society and industry, the implications for intellectual property in general, and patents in particular, are somewhat breathtaking.

Permalink | Comments | Email This Story

Should We Be Measuring Happiness As An Economic Measure?

Mike Masnick — Thu, 14 Feb 2013 00:16:32 PST

A lot of people have finally realized that traditional economic measures have all sorts of problems. Things like GDP mismeasure a ton of things, and by presenting an aggregate set of data, often obscure lots of issues. Also, things like GDP don't handle disruption very well. I've discussed in the past how you could argue that, purely on a GDP basis, something like Craigslist has been horrible. It effectively undercut newspaper classifieds, which was a multi-billion dollar business, and turned it into a much smaller business. If you measured such things purely by GDP, you'd say that it was bad. But, of course, Craigslist also created tons of value, enabling people to make transactions that couldn't have been made before, while also allowing other transactions to be made more efficiently and with less friction. Much of that will never show up in GDP, even if, intrinsically, most people recognize that something like Craigslist provided a lot more value to the world than it took away.

In trying to deal with that, we've started to see new forms of economic measurements pop up. One popular one is "happiness." There's even been some talk about using "Gross National Happiness" as a key economic measure. There's a great book from a couple of years ago by Nobel-prize winning economist Joseph Stiglitz, with Amartya Sen and Jean-Paul Fitoussi, called Mismeasuring Our Lives: Why GDP Doesn't Add Up. It was actually the result of a request from then French President Nicolas Sarkozy to explore how useful (or not) GDP was, including looking into alternate measurements, such as this idea of Gross National Happiness. If you haven't read the book, I highly recommend it.

Recently, the folks at Planet Money also did a report on the growing interest in measuring happiness, particularly as an official stat for American economic health. There appears to be growing interest in establishing a happiness index for the US, not unlike the unemployment index. Of course, you can think of the immediate problem. Just how do you measure happiness:

But once you get into the details, there's a lot of debate over the happiness data. One big divide: Should you ask people how they're feeling right now, or how they feel about their life in general?

You get different answers depending on what you ask. Which one is more important is a squishy, philosophical question.

The difference between asking about "right now" or "their life in general" can be massive. It shows up clearly in the data about how happy parents are vs. non-parents. There are tons of studies that suggest parents are miserable compared to non-parents. But nearly all of those studies are based on questions about "how happy are you now" type questions. Not surprisingly, the parent changing a diaper is probably going to report slightly less current happiness compared to the non-parent who's out at the bar with some friends, for example. But... it's not that simple. When other studies are done that ask parents and non-parents about how happy their overall lives are or how fulfilled their lives are, parents frequently report much higher feelings of fulfillment/happiness on a grand scale, while non-parents often report more regret. In other words: time frame makes a huge difference.

Of course, as the Planet Money report points out, just because something is difficult to measure, or involves highly subjective concepts, doesn't mean it can't be done. For example, unemployment data. You might think that this involves a nice, simple objective question, but when you look at the details, it's actually pretty subjective as well.

In the U.S, in order to be counted as unemployed, you have to be out of a job and looking for work. But what counts as looking for work? Checking Craigslist? Sending out three resumes a week? Five?

"It's actually kind of a hard question," says Justin Wolfers, an economist at the University of Michigan. "It's very subjective."

Yet every month, a single unemployment number is released.

So, you could see why a "Happiness Index" might be a compelling bit of economic data -- especially if you believe (as I do) that GDP is misleading. After all, if people are happier, isn't that a pretty important thing? Well, yes and no. Even as I find the topic interesting, I also worry a lot about the embrace of "Happiness" as an economic measure beyond the reasons laid out in the Planet Money report. Yes, it's difficult to calculate, but perhaps you can get past that so long as the calculation is done the same way over time. The real problem, for me, is that when you choose to make something a key economic number like that, you are guaranteed to start optimizing for it. That's what happens when you create metrics. Whether they're important or not, whether they're accurate or not, once you have a number, you naturally try to optimize for it.

It shouldn't be difficult, then, to quickly come up with scenarios for why a National Happiness Index could create significant problems as people optimize for it. First off, you encourage the kinds of short-term rewards that lead people to say they're happier, even if that creates massive costs down the road. Want to see governments leverage the present and put the costs on the future? Start using a happiness index. Second, if the focus is on maximizing present-day happiness, then you just focus on drugging the population. Yes, that's an extreme example, but hopefully it gets the point across. In economics, you need to measure the costs and benefits to things. You can "maximize happiness" in all sorts of ways if you ignore the costs to it. Put happy drugs in the water, and let everyone be thrilled. The Happiness index fails to take into account all of the consequences of doing something like that.

So while it's encouraging to see more of an exploration into alternative metrics, and getting beyond some of the older metrics that clearly "mismeasure" important aspects of our lives, we need to be careful to not just leap to the "next great thing" without realizing that it, too, likely has downsides.

Permalink | Comments | Email This Story

Of Clotheslines, Black Swans And Bad Measurements

Mike Masnick — Mon, 17 Sep 2012 14:31:00 PDT

Back in April, in writing about Union Square Ventures' Hacking Society event, I discussed the importance of measuring the unmeasurable, in noting that we all too often seem to be evaluating information-era economics using industrial-era metrics. That's a problem. Nick Grossman, who organized that Hacking Society session, has a great post discussing this same concept and highlighting Tim O'Reilly's discussion about this topic, in which he describes the clothesline paradox, which actually seems to come from a discussion in the early 1970s, and highlights how metrics can mislead. You can think of the clothesline paradox like this:

If you take down your clothes line and buy an electric clothes dryer the electric consumption of the nation rises slightly. If you go in the other direction and remove the electric clothes dryer and install a clothesline the consumption of electricity drops slightly, but there is no credit given anywhere on the charts and graphs to solar energy which is now drying the clothes.

In my mind, there are two "problems" associated with this, and while I think there is interest in attacking the first one, the second problem is often ignored. The first problem is that we notice that important information is measured with the wrong metrics. We see this all the time in the internet era. People talk about "the collapse" of the music industry, but miss the fact that more music has been produced, recorded and released in the last decade than in any previous decade. In fact, some of the evidence suggests more music was produced and recorded in the last decade than all other decades combined. Of course, that's an example of a metric that can be determined, but not all such metrics are that easy to pin down. For example, we talked about how Craigslist almost certainly helped contribute to the challenge that many newspapers are facing, because it undercut the cash cow that supported many of them: the classified advertising business. And if you used traditional metrics, you'd bizarrely and incorrectly suggest that Craigslist somehow "destroyed" value. But that's because no one takes into account all the value that Craigslist created, not for itself, but for its users. But how do you measure the fact that I can now find someone to take my old couch away for free? There's value in that transaction, but no one "measures" it. What about the fact that I can more efficiently rent out an apartment - without having to pay the local newspaper? Again, there's value, but it's not properly measured.

The second problem is a little trickier to understand. It's that when we have things that we can measure, we instinctively gravitate towards using those metrics, even if they're the wrong metrics! I was thinking about this as I read Paul Graham's excellent thoughts on "black swan farming," which is all about the counter-intuitive process involved in funding startups. There's a ton of tremendously thought-provoking lines in that piece, but I'm going to concentrate on one, which was really more of an aside, unrelated to the larger article (which you should go read), because it helped clarify my thinking on this point. Graham talks about not bothering to measure how many of the YCombinator companies he funds and trains later go on to raise more money after their initial fundraising efforts, noting:

I deliberately avoid calculating that number, because if you start measuring something you start optimizing it, and I know it's the wrong thing to optimize.

And here's where the problem of using the wrong metrics becomes compounded. Even if you know something is the wrong metric, just having the number almost forces you to optimize for it. So rather than looking at, say, what's best for the overall culture of music, we look at "revenue for the record labels" and decide we need to "fix" that. Or, we look at the patent system as a proxy number for "innovation" and then the focus becomes solely on increasing the number of patents we issue, rather than on actually maximizing innovation.

When you have the wrong metrics, not only do you have bad or incomplete information, but even when you know that it's almost impossible not to optimize for those metrics, because you don't have anything else to work towards.

There is a lot of new interest in quantifying all sorts of new data -- and one benefit of the information age is that it also helps to create new data that can be quantified. But not all quantified data is actually that useful, and unfortunately, we often get so focused on the fact that we have a number, we ignore the possibility that the number is not telling us anything useful.

I was recently reminded of Shelby Bonnie's opinion piece from three years ago about why we need to kill the CPM as a metric for advertising (for those who don't know, CPM -- or "cost per thousand" impressions -- is how most banner ads are sold). He noted, quite accurately, that even those with the best of intentions to get away from "CPM-based" advertising seem to end up there in the end anyway. Because we have that number. And it becomes what people optimize around, just because it's there.

All campaigns start with the best of intentions: “let’s do something creative, engaging, and unique!” But unless someone really senior from the agency or client side intervenes, the road for a campaign always leads to the media buyer and the dreaded spreadsheet, where the two most important columns are impressions and cost. Ironically, there’s usually some good stuff in campaigns, but they are thrown in for free as “value adds.” At some point, publishers decide that if all clients care about is impressions, then OK, we’ll give them impressions. The output is an industry that overproduces shallow, superficial, commoditized impressions. Why do we have so many bad sites that republish the same junky content–content that’s often made by machines or $1-per-post contractors? Why do sites intentionally try to get us to turn lots of pages with tons of top 10 lists, photo galleries, or single-paragraph summaries of someone else’s story?

The more I spend time thinking about these issues, the more I think these combined problems -- both not having the right data and then optimizing for the wrong data -- are the keys to many of the issues that we're regularly discussing around here. Figuring out ways to get beyond that, and to find the right data, and break our habits of relying on bad data are going to be increasingly important.

Permalink | Comments | Email This Story

A Question For SOPA Supporters: How Will You Gauge SOPA's Success?

Tim Cushing — Wed, 16 Nov 2011 11:15:55 PST

With all the discussion of what SOPA will or won't do in terms of breaking the internet or violating the first and fourth amendments, what I have yet to see discussed in depth is how supporters will measure the success of SOPA, if passed.

So, I'd like to take this chance to ask supporters of SOPA directly: how will you gauge the success of SOPA? What I'm interested in hearing is if any sort of metric has been defined.

I'll lay out my case briefly:

Like a lot of people, I'm of the belief that infringement can't be stopped. Even with this legislation in place, it's not difficult to imagine the same sort of whack-a-mole game that has plagued the content industries in the past will continue, only on a worldwide scale with the government's blessing. While this may increase the number of sites shut down or cut off from funding, I feel that it's overall impact will be minimal. I could be completely wrong about this, but I feel that those who still want to get something for free will still find a way, and find it much easier than those seeking to shut it down would imagine. The internet moves faster than 100+ year old industries and the government.

As collateral damage from this legislation mounts (and even the content industry's lawyer admits there will be some), there will be backlash from those affected. This has the possibility of encouraging more users to "stick it to the man," such as it were, increasing the amount of infringing activity.

If the intention of this legislation is to provide enforcement for copyright, my belief is that there should be some sort of metric or guideline to gauge its success. Without some sort of measurement in place, the very real possibility is that the enforcement efforts will continue to expand in scope and cause more and more collateral damage.

So, in all honesty, I'd like to open this thread to supporters of SOPA. I'd like to know how you'd measure the success of this legislation. Will it be based on the number of sites shuttered? Is it an increase in sales? If so, is there a certain percentage or dollar amount that would demonstrate the effectiveness of this bill? Is it something more vague or has someone out there run any sort of numbers on what a desirable outcome would be?

I'm not looking to get hung up on semantics ("piracy" vs. "infringement," "theft" vs. "sharing," etc.) or looking to rehash any arguments about infinite goods, piracy leading to sales, industry studies or anything that distracts from this question. I'm also not interested in hearing "we just want our rights and intellectual property respected/protected." This legislation goes far beyond anything that simplistic, especially something as unquantifiable as "respect." I'd like the discussion to stay focused on what SOPA proponents feel this legislation will accomplish.

How do you, as a supporter of SOPA, gauge SOPA's success?

Permalink | Comments | Email This Story

Measuring Success By Fan Passion, Not Billboard Chart Position

Mike Masnick — Fri, 16 Apr 2010 19:01:00 PDT

Kyle Bylin, over at Hypebot, has an excellent interview with Yan, the manager of the hip hop band Jedi Mind Trick, where he talks about how the band has embraced the internet to become much more successful. The interview is chock full of the type of thing we write about here. For example, on the band's response to "file sharing":

For JMT, it has been more of a blessing. Yeah, they've lost sales to illegal file sharing just like everyone else, but the efficiency of those file sharing networks has allowed their music to spread across the world in a way that the physical distribution network never achieved.

Those file-sharing networks are definitely part of the reason why JMT can play a show in front of 2,000 kids in Bogota, Colombia or to a similarly-sized crowd in Bucharest, Romania. They may have lost sales, but they have gained new throngs of fans by playing live in places that their music may have never reached without the Internet. With the improvements in digital distribution, the hope is that file sharers will become supporters.

This highlights a really important point that many of the critics to these ideas often miss. They seem to assume a zero sum world, where those who embrace the free distribution of their work are automatically "giving up" sales. But what JMT has discovered is that by accepting this, they've build a huge fanbase in places where it wouldn't have been possible before. As Yan notes later in the interview:

Over the 14 years of their career, the Internet enabled them to take what started as a passionate fan base in say Philly, NYC, Boston, and maybe a few other major cities like Los Angeles and grow that fan base across the world.

Without the Internet, they may have been relegated to a regional phenomenon or maybe a group that plays 10-15 shows a year in major markets. With the help of the Internet, they've grown from a regional niche into a worldwide niche with a worldwide fan base. They're 14 years into their career and there are still cities that they haven't played where there is a demand to see them play, so they're very fortunate in that respect.

Yan also shows how the band realized early on that the market had changed and (unlike some others) quickly figured out how to use that changing market to their own advantage:

The internet created an opportunity for them to have their music distributed across the world in a way that physical distribution hadn't achieved. They recognized that opportunity and seized it by developing their worldwide touring business. In the early years, they took low guarantees to go play for their fans; they were confident that they would draw a crowd and get invited back. Now, those tour dates are the anchor of their business. Between the creative accounting of labels and depleting sales, they knew that they had to develop their touring business if they hoped to carve out a sustainable career. Ironically, in this digital age, artists are looking to the analog experience of playing a show to a crowd of actual human beings as the cornerstone of their businesses.

Furthermore, Yan talks about the importance of connecting with fans, and notes how that's more important in the long run than how many albums are sold -- in part because they know that they will get support via other means:

The rise of the social networks has been a great asset for JMT as well. Those networks have created unprecedented access for fans to artists and vice versa. Our business is a customer service business; we care about how the fans feel about the music. We're always looking for new ways to interact with the fans, because the fans' reactions to what JMT is doing musically are a better barometer of their success than SoundScan numbers. For a long time now, we've chosen to measure their success in the fans' passion for the music rather than Billboard chart positions.

Not only that, but Yan also demonstrates how fans like supporting the indie bands and artists that they find. We've been hearing from some lately, who claim that no musician will be able to make money in the future because there will just be too much competition. But that assumes, incorrectly, that fans don't want to support bands -- they just need a good reason to buy:

It seems like human nature to root for the underdog. Fans of independent music are typically a different breed of music fan, because they generally have to work to discover you and they're actively seeking out music rather than waiting for it to be spoon-fed to them through traditional radio, TV, etc. outlets, so when they discover you they wear it like a badge of honor. JMT has fans that send them pictures of themselves with tattoos of JMT's logo or lyrics -- they're literally wearing the music as a permanent badge of honor. That type of passion isn't measured by a Billboard chart, but we're fine with that. We've built a business model that exists outside the gates of that hierarchy.

Great interview by Kyle, and it's great to see yet another band that has this all figured out. In fact, they've got it so figured out that they've set up an artist management business to help other artists do the same thing as well.

Permalink | Comments | Email This Story

Why We Need Better Metrics For Measuring User-Generated Content

Kevin Donovan — Mon, 5 Apr 2010 23:59:00 PDT

Much has been made about the iPad as a consumptive, rather than creative, device. Some, including law professor Tim Wu at a recent New America event, have voiced concern that it heralds the end of a golden era of user-generated content. But to truly understand the importance and impact of user-generated content - including on the traditional media that Clay Shirky has recently argued are fatally too complex to survive - we must have better measurement of the phenomenon. Without reliable data and sensible comparative metrics, it is impossible to say if we have even experienced a golden age of open creative possibility.

For example, nearly two years ago in response to Shirky, Nick Carr bristled at the idea that the Web was the necessary component for creative production, participation and sharing. According to Carr, the people he knew back before the Web were also creating - writing, photographing, drawing, constructing and volunteering. This is undoubtedly true, but because technology did not enable the inexpensive recording, archiving, sharing and finding of this creativity, it went largely unnoticed. Of course, cheaper technology almost certainly does enable more creative production, but how much is hard to say.

When Shirky notes that an amateur video of two children has garnered more views than American Idol, Dancing with the Stars, and the Superbowl combined, it is comparing apples and oranges. A minute video hardly competes with the Superbowl for eyeballs; certainly the Internet has opened opportunities to competitors to the Superbowl, but let's compare those. The problem is, we don't currently have the categories and metrics necessary to make sense of the rise (and potential fall) of creation. Some people are trying to create quantify the impact of blogs on the news cycle, but in regards to other media types, we seem to be ignoring the problem and living off anecdotes. So, how can we move ahead with better metrics for user-generated content and what should those metrics be?

Permalink | Comments | Email This Story