Of Clotheslines, Black Swans And Bad Measurements

from the we-measure-the-wrong-things-and-we-do-so-badly dept

Back in April, in writing about Union Square Ventures’ Hacking Society event, I discussed the importance of measuring the unmeasurable, in noting that we all too often seem to be evaluating information-era economics using industrial-era metrics. That’s a problem. Nick Grossman, who organized that Hacking Society session, has a great post discussing this same concept and highlighting Tim O’Reilly’s discussion about this topic, in which he describes the clothesline paradox, which actually seems to come from a discussion in the early 1970s, and highlights how metrics can mislead. You can think of the clothesline paradox like this:

If you take down your clothes line and buy an electric clothes dryer the electric consumption of the nation rises slightly. If you go in the other direction and remove the electric clothes dryer and install a clothesline the consumption of electricity drops slightly, but there is no credit given anywhere on the charts and graphs to solar energy which is now drying the clothes.

In my mind, there are two “problems” associated with this, and while I think there is interest in attacking the first one, the second problem is often ignored. The first problem is that we notice that important information is measured with the wrong metrics. We see this all the time in the internet era. People talk about “the collapse” of the music industry, but miss the fact that more music has been produced, recorded and released in the last decade than in any previous decade. In fact, some of the evidence suggests more music was produced and recorded in the last decade than all other decades combined. Of course, that’s an example of a metric that can be determined, but not all such metrics are that easy to pin down. For example, we talked about how Craigslist almost certainly helped contribute to the challenge that many newspapers are facing, because it undercut the cash cow that supported many of them: the classified advertising business. And if you used traditional metrics, you’d bizarrely and incorrectly suggest that Craigslist somehow “destroyed” value. But that’s because no one takes into account all the value that Craigslist created, not for itself, but for its users. But how do you measure the fact that I can now find someone to take my old couch away for free? There’s value in that transaction, but no one “measures” it. What about the fact that I can more efficiently rent out an apartment – without having to pay the local newspaper? Again, there’s value, but it’s not properly measured.

The second problem is a little trickier to understand. It’s that when we have things that we can measure, we instinctively gravitate towards using those metrics, even if they’re the wrong metrics! I was thinking about this as I read Paul Graham’s excellent thoughts on “black swan farming,” which is all about the counter-intuitive process involved in funding startups. There’s a ton of tremendously thought-provoking lines in that piece, but I’m going to concentrate on one, which was really more of an aside, unrelated to the larger article (which you should go read), because it helped clarify my thinking on this point. Graham talks about not bothering to measure how many of the YCombinator companies he funds and trains later go on to raise more money after their initial fundraising efforts, noting:

I deliberately avoid calculating that number, because if you start measuring something you start optimizing it, and I know it’s the wrong thing to optimize.

And here’s where the problem of using the wrong metrics becomes compounded. Even if you know something is the wrong metric, just having the number almost forces you to optimize for it. So rather than looking at, say, what’s best for the overall culture of music, we look at “revenue for the record labels” and decide we need to “fix” that. Or, we look at the patent system as a proxy number for “innovation” and then the focus becomes solely on increasing the number of patents we issue, rather than on actually maximizing innovation.

When you have the wrong metrics, not only do you have bad or incomplete information, but even when you know that it’s almost impossible not to optimize for those metrics, because you don’t have anything else to work towards.

There is a lot of new interest in quantifying all sorts of new data — and one benefit of the information age is that it also helps to create new data that can be quantified. But not all quantified data is actually that useful, and unfortunately, we often get so focused on the fact that we have a number, we ignore the possibility that the number is not telling us anything useful.

I was recently reminded of Shelby Bonnie’s opinion piece from three years ago about why we need to kill the CPM as a metric for advertising (for those who don’t know, CPM — or “cost per thousand” impressions — is how most banner ads are sold). He noted, quite accurately, that even those with the best of intentions to get away from “CPM-based” advertising seem to end up there in the end anyway. Because we have that number. And it becomes what people optimize around, just because it’s there.

All campaigns start with the best of intentions: “let’s do something creative, engaging, and unique!” But unless someone really senior from the agency or client side intervenes, the road for a campaign always leads to the media buyer and the dreaded spreadsheet, where the two most important columns are impressions and cost. Ironically, there’s usually some good stuff in campaigns, but they are thrown in for free as “value adds.” At some point, publishers decide that if all clients care about is impressions, then OK, we’ll give them impressions. The output is an industry that overproduces shallow, superficial, commoditized impressions. Why do we have so many bad sites that republish the same junky content–content that’s often made by machines or $1-per-post contractors? Why do sites intentionally try to get us to turn lots of pages with tons of top 10 lists, photo galleries, or single-paragraph summaries of someone else’s story?

The more I spend time thinking about these issues, the more I think these combined problems — both not having the right data and then optimizing for the wrong data — are the keys to many of the issues that we’re regularly discussing around here. Figuring out ways to get beyond that, and to find the right data, and break our habits of relying on bad data are going to be increasingly important.

Filed Under: , , , ,

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Of Clotheslines, Black Swans And Bad Measurements”

Subscribe: RSS Leave a comment
Anonymous Coward says:

Re: Sometimes Reality gives too Much.

The more educated you are, the more measures you know.

Finding and combining them in the right way is one of the highest levels of scoring in terms of demonstrating your knowledge in a field on universities!

It is the hardest lesson to learn and since reality often is too complex for optimizing or your measurements are too bad, you almost always have to settle for some sub-optimal measures.

Modellers dilemma: Too many variables makes the demand of accuracy on your measurements too high. Too few variables and your model will wilther away in the universe of: Just another mediocre correlation.

Bill W (profile) says:

For the world that isn’t run by lawyers, and we get to talk about that here a lot, it’s a world run by accountants. And if they find something they can count easily they will grab it, stuff it into a spreadsheet and you’ll pay hell trying to convince them that it’s meaningless. Meaning is for Engineers and Designers and such. Just like But, But, Piracy! It’s But, But, Numbers!

Anonymous Coward says:

Re: The problem with economics as a whole

“The entire field of economics tends to measure the wrong stuff.”

Thanks Suzanne. A simple statement like that seems to sum up why Mike always seems to be making odd claims, because economics tends to measure the wrong stuff.

Measuring marginal costs when the majority of the expensive is up front: WRONG.

Using a lack of sales as a justification for piracy not hurting anyone: WRONG.

Thinking that short term paint color marginal improvements on a product is true innovation: WRONG.

Thanks for stating in simple terms why Mike seems to be wrong very often. I also thank Mike for this big long post, as it clearly explains why he is wrong – because he looks in the wrong places for his data, and ignores the bigger picture.

Anonymous Anonymous Coward says:


Imagine trying to build a building, but instead of feet and inches (or meters and centimeters) you try to use gallons and tablespoons (or liters and milliliters). Could you do it? Maybe! The question is how many wrong turns will it take to get it right?

We measure unemployment by counting how many folks applied for unemployment in the last six(?) weeks. To me, unemployment is how many folks can work vs how many folks are working.

Now those two examples do not equate to what Mike is talking about directly. But they are examples of how political forces (not just government) work to obfuscate real issues.

Think Hollywood Accounting!

Anonymous Coward says:

Re: Imagine

“Imagine trying to build a building, but instead of feet and inches (or meters and centimeters) you try to use gallons and tablespoons (or liters and milliliters). Could you do it? Maybe! The question is how many wrong turns will it take to get it right?”

Imagine someone telling you that the foundation and structure wasn’t important, but how you delivered the keys to the clients was absolutely critical. That would be Mike focusing in on marginal costs while dealing with large up front cost business structures. You could make a business case out of it, perhaps, but you would be truly missing most of the deal.

The eejit (profile) says:

Re: Re: Imagine

Remember how capitalism is supposed to work?

For the most part, we’re not arguing against that: we’re arguing against unsustainable business models. Selling digital content and only digital content? Not sustainable. Selling digital content as part of a larger bundle? Sure. Selling t-shirts, watches and general merchandise surrounding the content? Sure.

Just because a total cost exists does NOT mean that we must buy at that price or else. What it means is that, for your product to be viable, people must buy at that price. That doesn’t apply to technically infinite goods (such as .mp3s and .avis). To try and make the world conform to you is akin to The Folly of Cnut.

Anonymous Coward says:

Re: Re: Re: Imagine

“Selling digital content as part of a larger bundle? Sure. Selling t-shirts, watches and general merchandise surrounding the content? Sure.”

It’s a fail. If people don’t want the “stuff”, your content is worthless because you are whoring it out trying to hock stuff.

It’s just not very good business sense.

Anonymous Coward says:

Re: Re: Re:3 Imagine

Yes, we agree. If you give your music away for free, as some sort of bait for other things, you are pretty much doing the “give it away and pray” model. You are hoping like hell that someone buys the over priced “thing” to pay for the music.

So I am glad you agree. Can you explain this to Mike and Marcus now?

Tim Griffiths (profile) says:

One example of this issue is the 95% piracy rate where people take the number of unauthorised uses of a game or app and match it up to the number of people who brought it. Yet that utterly ignores the fact that a pirate is going to be counted towards vastly more uses of apps and games than most people can buy. Which weights the numbers and can make a minority of the over all market the majority of a use for any given product.

It’s a case where the issue of piracy is a market issue not a product issue and by using metrics produced at the product level you are distorting your reality of the over all market. In the case of iOS we can show this in how piracy happens on jailbroken phones and that is a metric that is measured (10-20% when I last looked) yet app devs talk about a 90-95% piracy rate. How you deal with a market is vastly different when you are looking at most 20% of people who might be pirates then 95% of people.

Ninja (profile) says:

That. I’ve learned so far that sometimes the direct measurements aren’t what you are supposed to be looking at or trying to optimize but rather a composite indirect measurement that is formed by other direct ones. And sometimes you have to add a factor to your indirect that simply can’t be expressed in a numeric value. That’s pretty challenging.

Your music industry example is right on spot. You simply MUST add shows, merchandise and other forms of making money that actually have nothing to do with selling CDs or mp3. Heck, I’d say you need to include crowdsourcing efforts, small groups presenting on bars and many other sources of revenue before stating anything about the music business. The RIAA is but a small part of the pie and the fact that this slice is shrinking steadily doesn’t mean the pie isn’t growing fast.

montgoss (profile) says:

Software development

I’m a software engineer. All I could think about while reading this article is management’s obsession with SLOC (Source Lines of Code). They seem to base everything on this number, even though it is practically meaningless. But that’s the number that is easiest to measure! So they do!
It kills me a little every time someone starts talking about SLOC…

Add Your Comment

Your email address will not be published.

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...