Can We Come Up With A Better Way To Measure Innovation?

from the the-market? dept

For quite some time, we've pointed out that patents are not a proxy for how much innovation a company does. In fact, research has proven this pretty conclusively. And yet, because they're easily countable, the press and politicians love to use the number of patents as a proxy for how much innovation is happening. That leads to silly articles from folks who should know better, making statements like "Clearly, the global recession seriously hampered innovation in the United States" because fewer patent applications have been filed.

But, that leads to a separate issue. If you aren't using patents as a proxy, then how do you measure innovation? Tim O'Reilly is asking for suggestions on a measure innovation metric:
How might we construct a metric that would reflect the transformative power of the web (no patents), Google (nowhere near as many as their innovations), Facebook (ditto), Amazon (ditto, despite the 1-click flap), Craigslist, Wikipedia, not to mention free software such as Linux, Apache, MySQL and friends, as well the upwelling of innovation in media, maker culture, robotics... you name it: all the areas where small companies create new value and don't have time, money or inclination to divert effort from innovation to patents?
The problem is that I'm not sure there is any single metric that really works here -- especially when it comes to disruptive innovations. You could go with revenue, but one of the features of truly disruptive innovations is that they sometimes shrink direct markets (while greatly increasing the size of indirect markets). So that might not be very useful either. You could go with user adoption -- but that may be fleeting or possibly gamable.

Even with older successful technologies, I'm not sure these kinds of metrics would most accurately highlight how much their innovation meant. Telcos made lots of money, but much of the real innovative value of the telephone was what other businesses they eventually enabled. How do you calculate that?

The only real metric I can think of -- though I'm not sure how one would measure this accurately -- is how much you would have to pay customers to get them to stop using a certain innovation. If you went around and surveyed people, and figured out how much it would cost to get them to, say, stop using search engines or email or mobile phones or automobiles, you might be able to get a sense of the "value" of certain innovations. From there, as a baseline, you could potentially monitor the delta over time. Thus, as the iPod grows in "value," the value of a portable CD player would decrease. As mobile phone cameras got better, the value of portable cameras would decrease, etc. It would be a lot of work, but could give you a much better general sense of innovation and how it changes over time than any patents.


Reader Comments (rss)

(Flattened / Threaded)

  1.  
    icon
    Hulser (profile), Mar 29th, 2010 @ 10:00am

    Patent inflation

    Another way to look at it is that patents actually could be good way to measure innovation if there wasn't a "patent inflation" caused by giving out patents for so many non-innovative ideas. The problem as I see it is that the standards for what's innovative has become so low, the count of how many patents a company has is all but meaningless.

    Yet another aspect to this topic is that the quest for an accurate measure of innovation may be a red herring, diverting attention away from the more important measures of success. We're coming out of a period of time of massive innovation. Doesn't it make sense that there might be a bit of a lull in real new ideas for a while and the better measure of a company's worth isn't how innovative they are -- no matter what the metric is -- but instead, how effective the company is in using existing ideas?

     

    reply to this | link to this | view in thread ]

  2.  
    identicon
    Melanie, Mar 29th, 2010 @ 10:09am

    Innovation over time

    Such a way of measuring innovation will always capture a decline in 'value' for innovations that spark lots of competition. As more and more smartphones compete with the iPhone (or other innovations compete with smartphones), people would demand a lot less to stop using any one of them. They could just take the bribe and switch to a competitor. Does that mean that the original innovation is worth less? How do you decide at what point in time to measure? After widespread adoption but before significant competition?

     

    reply to this | link to this | view in thread ]

  3.  
    icon
    Hosermage (profile), Mar 29th, 2010 @ 10:25am

    innovation depreciation?

    I don't see how you can depreciate innovation as it gets old. I can see how an innovation may not reach its full potential unless you give it enough time to evolve, but depreciate older innovations? I disagree. Just because iPod came along, it shouldn't discount CD-players when it was popular. In fact, I think the value of innovation should be accrued over time, so something like the wheel will have tremendous value since we've been using it for so long and its value will only stop growing until we stop using/producing it.

     

    reply to this | link to this | view in thread ]

  4.  
    identicon
    Hoeppner, Mar 29th, 2010 @ 10:30am

    I can't think of a proper number that would let me stop using search engines. Would be even worse if it included "search this site".

     

    reply to this | link to this | view in thread ]

  5.  
    icon
    :Lobo Santo (profile), Mar 29th, 2010 @ 10:38am

    Disinterest Value

    Your proposed metric needs a time dimension.
    For example: "How much money would convince you to stop using your iPhone for a week?" versus "How much money would convince you to stop using your iPhone forever?"

    Basic Micro Economics refers to the use/satisfaction you get out of something as 'utility.' To the best of my knowledge, there is not yet a good metric for utility.

    A metric for innovation... that's a tough one.

     

    reply to this | link to this | view in thread ]

  6.  
    icon
    mike42 (profile), Mar 29th, 2010 @ 11:02am

    Hardware/Software update sales

    How about hardware/software update/upgrade sales? If the new stuff isn't innovative, you won't buy it. It's got to have some feature that you're willing to spend money on, or at least spend time to download, for it to be worth while. You could probably cut out new sales, as that may be people just trying out the platform. In any case, it would be much more reasonable than counting patents.

     

    reply to this | link to this | view in thread ]

  7.  
    icon
    fogbugzd (profile), Mar 29th, 2010 @ 11:38am

    It's easy

    There is a short-term metric and a long-term metric that come to mind.

    Long term:
    1 divided by the number of patents filed.
    Basically, if more patents are filed now, then it will be harder to innovate in the future. Also, as noted in the article, the most innovative companies seem to file the fewest patents. If more patents are being filed, it means that companies are probably spending too much time filing patents and not enough time innovating.

    Short term:
    1 divided by the number of patent lawsuits filed.
    As noted many times before, companies that can't innovate litigate. A rise in the number of lawsuits filed indicates the number of companies that have given up on their own innovation and who want to stifle innovation by others.

     

    reply to this | link to this | view in thread ]

  8.  
    icon
    Steven (profile), Mar 29th, 2010 @ 11:42am

    General metric.

    I can't think of a good way to measure the innovation of a given company, but a general metric I think would be something along the lines of:

    value created
    __________________________

    people * resources used

    So in basic terms more innovation will either use fewer resources and/or produce more/better things. I added people in there but maybe people would just be part of the resources bit? Not sure.
    I think it has to be a bit general because innovation is tricky. For example, the Wii was an innovative game system (probably not very interesting from a value perspective) but the hardware from that combined with some open source software created inexpensive interactive whiteboards that make a wide range of other industries more productive. So in that case we have innovation, but how many innovation points do you give to the Wii vs the software?

     

    reply to this | link to this | view in thread ]

  9.  
    icon
    gojomo (profile), Mar 29th, 2010 @ 12:14pm

    Innovation == volatility or changes in expectations?

    At some level, innovation means effective changes in how results are achieved. So let's say there is a futures or prediction market in future results. The volatility, or changes in predicted levels, over time could each be considered indicators of the kind and volume of innovation.

    For example, consider measured life expectancy, and a prediction market in future life expectancy. Rises in both the volatility and level of the futures market give some reading about how much innovation people believe is in progress and likely. Conditional contracts -- (life expectancy for a diagnosis of condition X in 2020 | single-payer health-care) -- might offer best-guess estimates of the innovation expected under certain paths of action.

     

    reply to this | link to this | view in thread ]

  10.  
    identicon
    Anonymous Coward, Mar 29th, 2010 @ 1:11pm

    Re: It's easy

    Your short-term measure does not work for the vast majority of companies. There are only around 2,800 patent lawsuits filed per year. There are an estimated 13 million companies in the U.S. What that means is that your short term measure will be meaningless for 12,997,200 companies.

    As for your other measure, that too seems to have weaknesses. Patents filed is not inversely proportional to innovation. Mike claimed (I could not read the paper the posts linked through to, it seems to have been removed) that patents are UNRELATED to innovation. If that is true, your inverse proportionality would provide a meaningless number.

     

    reply to this | link to this | view in thread ]

  11.  
    identicon
    Anonymous Coward, Mar 29th, 2010 @ 4:04pm

    how about how many times masnick uses your name as an example of good business models?

     

    reply to this | link to this | view in thread ]

  12.  
    identicon
    angry dudu, Mar 29th, 2010 @ 4:40pm

    Punk?

    Punky? Punk!

     

    reply to this | link to this | view in thread ]

  13.  
    identicon
    Anonymous Coward, Mar 29th, 2010 @ 5:19pm

    I don't know if there is a good way to measure innovation but the traditional method has always been a function of how much more efficient/effective a new method is at doing something when compared to the old method.

    For example, say I Invented the printing press. What was the old method of making copies? It was by hand. How long did that take? Say it took an average of 20 minutes per page in order to get 99 percent accuracy. Lets assume the first printing press, in comparison, can do 10 pages a minute with 95 percent accuracy (presses often mess up pages, you must throw pages away and fix the problem, they have all sorts of issues).

    How much more efficient is the new system than the old? Well, you could see how many pages each does at 100 percent accuracy. At 100 percent accuracy the printing press does 9.5 pages per minute. Humans, on the other hand, do less than 1/20 pages per minute at 100 percent accuracy. That's a 190 fold difference. So we have just innovated by 19,000 percent.

    Of course this doesn't measure the fact that when a printing press makes a mistake, it usually messed up the entire page whereby we can easily identify the mistake and discard it hence preventing us from ever replicating the same mistake. Some human mistakes, on the other hand, are far more likely to make it through filtration, and there is a chance that such mistakes could be copied and create a domino effect of mistakes where you end up with a document that is much different than the original. Sure, if you keep on copying a page and then copying that copy, etc... you wind up with a distorted image even with a printing press, but another innovation is the fact that, with the printing press, there is hardly ever a need to copy copies of copies, unlike the case when humans used to write stuff down. Of course all of these effects can also be separately measured and quantified but I don't want to go through the thought process required to do so and demonstrate it. One can also quantify the cost in labor for making copies vs the cost of operating and maintaining a printing press and the compare the cost per page both in time and money.

    Now lets say someone invents the Internet. We can, again, compare the Internet as an information distribution platform to the latest printing press (not the above slow one). Say the latest printing press made 200 pages per minute and it would take an hour to deliver it to everyone in a neighborhood and the cost in labor was $600. The Internet, on the other hand, can deliver it all in 1 minute to everyone and the cost of labor is $15 (someone needs to type the reports up and post them. Lets forget the cost of the Internet vs the cost of the printing press). Again, we can easily compare the cost per person being delivered a service and quantify it and compare the service time's and hence, in a sense, quantify innovation and compare which method of doing something is more efficient and by how much (ie: percentage wise).

     

    reply to this | link to this | view in thread ]

  14.  
    identicon
    Anonymous Coward, Mar 29th, 2010 @ 8:19pm

    Re:

    (To continue)

    So I suppose, from an economics perspective, the best way to "measure" innovation is to determine how much additional consumer + producer surplus is created by an innovation. So to measure how much a country innovated during a period of time one would have to add the consumer and producer surplus created by each individual innovation during that time to get the total additional consumer and producer surplus created by all innovations during that time.

     

    reply to this | link to this | view in thread ]

  15.  
    icon
    Simon (profile), Mar 30th, 2010 @ 2:54am

    Why bother measuring innovation?

    Innovation isn't a goal in itself, it is a tactic; a means to an end.
    In football / soccer, you don't win games on the amount of passes you made down the wings (a tactic), you win games based on the number of goals you scored (a simplistic reading but I'm sure you get my drift). The thing that should be measured is the effect of the innovation on your goals - did our sales increase? What's our market share now?

     

    reply to this | link to this | view in thread ]

  16.  
    identicon
    Nick Kinports, Apr 5th, 2010 @ 3:25pm

    How about The Innovation Power Score?

    Seems to solve the issue you are presenting: http://www.maddockdouglas.com/innovation-power-score

     

    reply to this | link to this | view in thread ]

  17.  
    identicon
    John Gabrick, Apr 9th, 2010 @ 4:33am

    Great Metric

    I've been doing this for over a decade, and I have to say that this is one of the best ways I've seen. All of the current methods: revenue, cost savings, number of patents, etc. have too many other dependent variables to make them a true indicator of value. In this case, the metric is simply time and money. When people dig into their wallet and pay for something that is the true value--you don't have to look any farther than eBay to see how time and money really set value.

    www.stepbystepinnovation.com

     

    reply to this | link to this | view in thread ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Save me a cookie
  • Note: A CRLF will be replaced by a break tag (<br>), all other allowable HTML will remain intact
  • Allowed HTML Tags: <b> <i> <a> <em> <br> <strong> <blockquote> <hr> <tt>
Follow Techdirt
A word from our sponsors...
Essential Reading
Techdirt Reading List
Techdirt Insider Chat
A word from our sponsors...
Recent Stories
A word from our sponsors...

Close

Email This