Techdirt. Stories filed under "methodology"

Feds Say We Need Stronger IP Laws Because Grocery Stores Employ Lots Of People

Mike Masnick — Thu, 7 Jun 2012 10:40:00 PDT

You may recall that, back in April, we criticized a Department of Commerce/US Patent and Trademark Office "study" that claimed it was trying to "better understand" intellectual property by adding up all the jobs in "IP-intensive" industries. We had significant concerns about the report, in particular the reasoning behind the methodology of how it chose "IP-intensive" jobs (and why it thought that had anything to do with stronger IP laws).

The folks over at KEI have put the problems of the report into plain view by digging into the details of the methodology, and realizing that the vast, vast majority of these "IP-intensive" industries are actually coming from the trademark side of things. Now, that's fine, but trademark is an entirely different issue than copyrights and patents. And, even if we dig into the trademark side of things some big questions are raised about the methodology. For example, in counting up all these jobs, what "IP intensive" industry employs the most people? Grocery stores.

According to the report, the number one IP intensive industry in terms of employment is “grocery stores,” with 2.5 million jobs. The six industries with more than a million jobs are the following:

1. Grocery stores, 2.5 million
2. Depository credit intermediation, 1.7 million
3. Computer systems and designs,. 1.6 million
4. Insurance carriers, 1.4 million
5. Management and technical consultants, 1.2 million
6. Clothing stores, 1.1 million
These are just a few of the industries the Department of Commerce calls IP-Intensive: Oil and gas extraction, Residential building construction, Grain and oilseed milling, Dairy product manufacturing, Lessors of real estate, Gambling industries, Household and institutional furniture, Pulp, paper, and paperboard mills, Sporting goods and musical instrument stores, Travel arrangement and reservation, etc.

Now, we can argue over just how much grocery stores rely on IP protection, but I think most normal people would agree that even if we abolished IP laws, grocery stores would likely still exist. Would they employ a different number of people? Possibly, though I'm not sure the difference would be significant. The simple fact is that even if grocery stores do benefit from trademark law, the number of jobs at grocery stores that exist because of trademark laws is minimal. And yet all of them are counted towards this total number -- which is regularly being touted by the entertainment industry as proof as to how important stricter IP laws are.

But, you might be saying, you know what's missing from the list above? Yes, that's right: the entertainment industry. In fact, as KEI points out, the only sector in that list above that can "honestly" be described as "IP-intensive" is "computer systems and designs," and if you look, those were some of the folks arguing most vociferously against copyright law expansions like SOPA and for patent reform that reduced the impact of patents. To use those jobs as evidence of the need for a stronger IP regime is downright misleading.

KEI points out that when you look at the number of jobs in the industries that people normally think of as relying on IP, the numbers are much smaller... and even then are potentially misleading in terms of the need and reliance on actual IP laws:

Industries like these overwhelm the statistics on jobs for the more legitimate choices, such as Sound recording industries (just 36.4 thousand jobs), or Software publishers (259.8 thousand), making it seem as though the IP-Intensive industries are truly enormous employers.

[....]

It was also pointed out that while the Software publishing sector was a high wage sector, it was a relatively small employer of the professionals in its key occupations. For example, less than 5 percent of computer programmers work for Software publishers. What do the other 95+ percent of computer programmers do? Quite a few build applications and services than use various open source free software platforms, which are both inexpensive and easy to customize.

And, where is the growth for employment? For the computer, mathematical science occupations, the rate of growth is three times higher outside of the Software Publishers Sector than inside the sector. So what can USPTO or the Economics and Statistics Administration tell us about the relationship between IP and employment? Not much, other than grocery stores, insurance companies and oil companies use a lot of trademarks, and not many people work in the sound recording industry.

The report itself explicitly states that it is not designed to be used to support any particular policy. Yet since it's come out, and despite this ridiculous methodology, it is regularly cited by the MPAA, the RIAA and supporters of things like SOPA and ACTA as "proof" that we need these laws. In fact, Jamie Love from KEI notes that right after the event where they discussed these problems with the methodology of the report, and the fact that all these grocery store employees have been unwillingly drafted to claim they need more IP laws to protect the small number of jobs under the RIAA umbrella, he went to a meeting where a US trade official used the report to claim that it showed we needed to adopt ACTA:

After we left the TACD plenary where the IP and employment report was discussed, the TACD IP Policy Committee met with four US IP/Trade agencies, and three Directorates of the European Union. The meeting began with a US trade official telling us about this new employment study that showed the need for ACTA. No kidding. And, we will certainly continue to hear about this report, with very little understanding of how it was put together.

Before the report came out, the White House had reached out to me, and said that since they knew I was interested in these issues, they could set up an interview with the "economic experts" who put together the report. I tried to take them up on the offer. The response was so ridiculous and so embarrassing for the Department of Commerce that I have actually emailed them back asking them to make sure they really wanted me to run a story based on what was said. I'll have a follow up story soon, either with that response or (hopefully) with a more substantive discussion, should they reconsider.

Permalink | Comments | Email This Story

US ITC Uses Ridiculous Methodology To Claim 'Piracy' In China Costs US Firms $48 Billion In 2009

Mike Masnick — Thu, 19 May 2011 10:15:07 PDT

I'm always a bit wary of news reports about "studies" claiming huge dollar value "losses" due to infringement, as when you look at the details, the methodology is almost always suspect. Of course, this generally applies to industry-driven studies. So, I was a bit more interested to find out the details of a newly released study from the US International Trade Commission, done at the behest of the Senate, to determine the "cost" of intellectual property infringement in China. I was surprised that Reuters, of all publications, would have such a vague description of the report, and not discuss the methodology at all. After all, Reuters employs Felix Salmon, who is one a very small number of folks who has spent quite some time debunking the methodology of similar studies.

In this case, the ITC is claiming losses of a rather astounding $48 billion. Having seen similar studies over the years, both good and bad, my first reaction was that this didn't pass the laugh test (at all) and sounded like the typical exaggerations from industry. So, I looked at the actual ITC report (pdf and embedded below) and it turns out it's even worse than I expected. Rather than taking any sort of actual objective study, the ITC simply asked 5,000 companies for what they thought their "losses" to Chinese infringement were. Not only that, but the ITC tried to choose the firms who were most likely impacted by this -- which means those who have the highest incentives to lie or exaggerate, because they want to have greater protectionism against Chinese competition.

Seriously, this methodology is just dopey. It's like asking horse and buggy makers how much in "losses" they would suffer if the automobile market were allowed to move forward -- and then basing regulatory policy on what they had to say. What's most frustrating about this is that folks at the ITC know this. Just last year, it held hearings on this topic for this very report in which it was told, repeatedly, by experts that such methodologies were woefully inaccurate. Given that, it's somewhat incomprehensible that the ITC would still use such an obviously wrong and biased methodology to support its claims.

It's both disappointing and troubling that the ITC would use such a methodology (and that the press would parrot the numbers back as fact, without bothering to look at or even mention the methodology). The real problem is that this clearly bogus study will now likely have a tremendous impact on US policy towards China.

Permalink | Comments | Email This Story

Does Being More Vocal In Video Game Violence Debate Mean You Have The Better Argument?

Mike Masnick — Mon, 25 Apr 2011 09:09:23 PDT

A few folks sent over the news of some really bizarre research done by Brad Bushman and Craig Anderson on the question of whether or not violent video games harm teens. First of all, the research is already somewhat suspect, in that Anderson has a long history of claims about how violent video games must harm children based on questionable data. This new report is based on such questionable and loop methodology, you almost wonder why they even bothered.

What they did was take the amici briefs from the Supreme Court case concerning California's anti-violent video game law, and run some numbers on who wrote the briefs and how many "published" studies they had. And that was how they determined which one was more credible. I'm not kidding. Quantity over quality:

The researchers analyzed the credentials of the 115 people who signed the Gruel brief, who believe video violence is harmful, and the 82 signers of the Millett brief, who believe video violence is not harmful. (The briefs are named after the lead attorneys for each side.)

The data for the study came from the PsycINFO database, which provides more than 3 million references to the psychological literature from the 1800s to the present, including peer-reviewed journal articles, book chapters or essays, and books.

For each of the signers of the two briefs, the researchers calculated how many articles and books they published on issues relating to violence and aggression in general and on media violence specifically.

The results showed that 60 percent of the Gruel brief signers (who believe video game violence is harmful) have published at least one scientific study on aggression or violence in general, compared to only 17 percent of the Millett brief signers.

Moreover, when the researchers looked specifically at the subject of media violence, 37 percent of Gruel brief signers have published at least one study in that area, compared to just 13 percent of the Millett brief signers.

And they claim that this is "a very objective approach." It's also a profoundly meaningless approach. In case you didn't follow it, there were a ton of amici briefs filed by various parties in this case. This study picked just two of the briefs. The first one (pdf) filed by California State Senator Leland Yee (whom, I believe, may have written the legislation in question), the California Chapter of the American Academy of Pediatrics and the California Psychological Association. That brief supports California's position in the case. The second one (pdf) is a brief from "social scientists, medical scientists and media effects scholars," which is the one that supports the other side, saying that the law isn't constitutional. You can read the two briefs that I linked to above, and you can judge the relative merits of both.

But that's not what Bushman and Anderson did. They simply took the signers of each brief and measured how many of them have published studies on this specific question. Of course, that's a meaningless and arbitrary number, especially when presented in percentages. Based on this methodology, it would mean that if only one person signed the amicus brief, but had published research, then that one would clearly be the most credible, since 100% of the signers would have published. Obviously, that makes no sense.

Now, Bushman and Anderson -- clearly expecting the quantity over quality issue to make for easy mockery of such a ridiculous study -- also added a second element to try to show "quality" as well:

In a further analysis, Bushman and Anderson examined where the signers of both briefs have published their research. The best academic journals have the highest standards and the most rigorous peer review, so only the best research should be published there, Bushman said.

The researchers used a well-established formula, called the impact factor, to determine the top-tier journals, and then calculated how many signers had published in these journals.

Results showed that signers of the Gruel brief had published over 48 times more studies in top-tier journals than did those who signed the Millett brief.

But, again, this attempt at showing "quality" is really a "quantity" study in disguise. It's not looking at the actual credibility of any of the studies, but trying to create an aggregate (but meaningless) number. And, again, the entire basis of this result is a meaningless dataset. I'm really wondering who would possibly read this and think that the results are credible.

Oh yeah, and one final point. Guess what two academics signed that first brief? You guessed it: Craig Anderson and Brad Bushman. Talk about researcher objectivity huh? They create a bogus methodology to try to "prove" that the brief they signed is more credible than someone else's brief? Honestly, when they present methodology like this, it serves mostly to raise questions about their methodology on any other study as well. They've made it clear that they're not researching the truth. They're starting with an established position and trying to figure out ways to present evidence to support that. That's not science.

Permalink | Comments | Email This Story