Garbage In, Garbage Out On Studies Concerning Which Countries 'Lead' In Education

Mike Masnick — Tue, 25 Sep 2012 03:15:59 PDT

We've been discussing how bad metrics can lead not only to bad conclusions, but also to a tendency to optimize for those bad metrics. That seems to apply to some of the recent news concerning "lagging" by the US in the OECD's education metrics. However, as Greg Ferenstein highlights in a piece over at TechCrunch, these stats may be meaningless. Not only has the US pretty much always lagged since these stats were first calculated, there's little indication that the stats being measured are representative of anything actually important.

Ferenstein's article is well worth reading, but it highlights a few key points. First of all, what the OECD is measuring might not be particularly meaningful:

However, the report implies that education translates into gainful market skills, an assumption not found in the research. For instance, while Chinese students, on average, have twice the number of instructional hours as Americans, both countries have identical scores on tests of scientific reasoning.

Second, whenever you're talking about aggregate numbers, a lot of important nuance can be lost in the mess. For example, there is evidence to suggest that while the US may not be good at educating everyone, it does seem to do quite a good job with taking its best students and preparing them for the world.

Most importantly, the innovators at the helm of an economy come from the top quarter of students. While the United States has a dismal track-record of inequality, we treat our brightest minds quite well. The “average test scores are mostly irrelevant as a measure of economic potential,” write Hal Salzman & Lindsay Lowell in the prestigious journal, Nature, “To produce leading-edge technology, one could argue that it is the numbers of high-performing students that is most important in the global economy.”

The United States, they find, has among the highest percentage of top-performing students in the world.

Now, there are reasonable policy arguments to be made about this, and about what it means for everyone who is not included in those "brightest minds." You can reasonably argue that we should be seeking ways to move more of the "non-brightest minds" students into the "brightest minds" category. But we shouldn't take the aggregate data of ill-performing education metrics and from it assume that the entire system is broken. When you look for an across-the-board solution to metrics like that, you very often end up wiping out the good stuff (such as how we handle top performers) in the process -- potentially making the whole thing worse.

Ferenstein also points out a second point on the "economic impact" of our education system: historically, we've tended to fix that economically by attracting the best and brightest from other countries to come to the US for higher education and for work as well. This is something that we're doing less and less of these days, due to ridiculous and reactionary laws on immigration and civil liberties.

None of this is to suggest that we shouldn't figure out ways to improve our education system. It does remain pretty clear that good education is important to other aspects of the economy. But we need to be quite careful in understanding what we're really measuring and what it's tied to, rather than just accepting that if one report says we're "lagging," it must really be an actual problem.

Permalink | Comments | Email This Story

Would You Rather Be 'Right' Or Realistic?

Mike Masnick — Tue, 28 Feb 2012 08:20:24 PST

We recently wrote about the excellent comic from Matthew Inman's The Oatmeal to highlight how companies turn would-be buyers into infringers by not making the content available. Here's a snippet. In response, some folks sent over columnist Andy Ihnatko's response, in which he suggests the comic is actually representative of the sense of entitlement that people feel towards such content. Ihnatko's writeup is slightly amusing as he tries to mock those consumers for actually having an opinion on how they consume content and concludes with this basic statement:

The world does not OWE you Season 1 of “Game Of Thrones” in the form you want it at the moment you want it at the price you want to pay for it. If it’s not available under 100% your terms, you have the free-and-clear option of not having it.

I sometimes wonder if this simple, grown-up fact gets ignored during all of these discussions about digital distribution.

I was going to write a rather long response to why this is kinda silly, but Marco Arment did a better job than I ever would in explaining the difference between being "right" and being "pragmatic." You really should read the whole thing, because it involves a rather detailed example involving the physical layout of a restroom. I'm going to skip over that part and highlight the summary point, but it's worth reading the full thing:

We often try to fight problems by yelling at them instead of accepting the reality of what people do, from controversial national legislation to passive-aggressive office signs. Such efforts usually fail, often with a lot of collateral damage, much like Prohibition and the ongoing “war” on “drugs”....

[....]

Relying solely on yelling about what’s right isn’t a pragmatic approach for the media industry to take. And it’s not working. It’s unrealistic and naive to expect everyone to do the “right” thing when the alternative is so much easier, faster, cheaper, and better for so many of them.

This point could be seen as the central theme of many of the 40,000 plus posts that have been made on this blog: dealing with reality is always going to be a more effective way to go about things than taking some "moral" stand on how things "should" be. And, if you can deal with the realities and it actually solves the whole moral "I'm right!" part at the same time, what good is it to not deal with realities?

Taking the point even further, there's a simple fact of today's world, which is that consumers have power. Ihnatko's entire point seems to assume that this consumer power is "entitlement." I tend to think of it as consumers making their will known -- and that tends to lead to better products that should make everyone better off. What Ihnatko ignores is that a market is not determined by just one side. It's the interplay between buyers and sellers, and if the buyers aren't happy, they express that to the sellers in certain ways -- and infringement is one of those ways. It's a market signalling method. I'd argue that it's just as much an "entitlement" mentality by the "sellers" to pretend that only they get to decide what the consumer should be able to get, without listening to what the consumer wants.

Permalink | Comments | Email This Story

Techdirt. Stories filed under "practicality"

Garbage In, Garbage Out On Studies Concerning Which Countries 'Lead' In Education

Would You Rather Be 'Right' Or Realistic?