Garbage In, Garbage Out On Studies Concerning Which Countries 'Lead' In Education

from the does-it-matter dept

We've been discussing how bad metrics can lead not only to bad conclusions, but also to a tendency to optimize for those bad metrics. That seems to apply to some of the recent news concerning "lagging" by the US in the OECD's education metrics. However, as Greg Ferenstein highlights in a piece over at TechCrunch, these stats may be meaningless. Not only has the US pretty much always lagged since these stats were first calculated, there's little indication that the stats being measured are representative of anything actually important.

Ferenstein's article is well worth reading, but it highlights a few key points. First of all, what the OECD is measuring might not be particularly meaningful:
However, the report implies that education translates into gainful market skills, an assumption not found in the research. For instance, while Chinese students, on average, have twice the number of instructional hours as Americans, both countries have identical scores on tests of scientific reasoning.
Second, whenever you're talking about aggregate numbers, a lot of important nuance can be lost in the mess. For example, there is evidence to suggest that while the US may not be good at educating everyone, it does seem to do quite a good job with taking its best students and preparing them for the world.

Most importantly, the innovators at the helm of an economy come from the top quarter of students. While the United States has a dismal track-record of inequality, we treat our brightest minds quite well. The “average test scores are mostly irrelevant as a measure of economic potential,” write Hal Salzman & Lindsay Lowell in the prestigious journal, Nature, “To produce leading-edge technology, one could argue that it is the numbers of high-performing students that is most important in the global economy.”

The United States, they find, has among the highest percentage of top-performing students in the world.

Now, there are reasonable policy arguments to be made about this, and about what it means for everyone who is not included in those "brightest minds." You can reasonably argue that we should be seeking ways to move more of the "non-brightest minds" students into the "brightest minds" category. But we shouldn't take the aggregate data of ill-performing education metrics and from it assume that the entire system is broken. When you look for an across-the-board solution to metrics like that, you very often end up wiping out the good stuff (such as how we handle top performers) in the process -- potentially making the whole thing worse.

Ferenstein also points out a second point on the "economic impact" of our education system: historically, we've tended to fix that economically by attracting the best and brightest from other countries to come to the US for higher education and for work as well. This is something that we're doing less and less of these days, due to ridiculous and reactionary laws on immigration and civil liberties.

None of this is to suggest that we shouldn't figure out ways to improve our education system. It does remain pretty clear that good education is important to other aspects of the economy. But we need to be quite careful in understanding what we're really measuring and what it's tied to, rather than just accepting that if one report says we're "lagging," it must really be an actual problem.

Reader Comments

Subscribe: RSS

View by: Time | Thread

  1. identicon
    Anonymous Coward, 25 Sep 2012 @ 4:27am

    Waiting for the Google funded "Education is Rising". ;)

    reply to this | link to this | view in thread ]

  2. identicon
    Anonymous Coward, 25 Sep 2012 @ 4:47am


    keep waiting

    reply to this | link to this | view in thread ]

  3. icon
    Forest_GS (profile), 25 Sep 2012 @ 5:50am

    What's important is inviting immigrants over here to create businesses. That will create more jobs.

    reply to this | link to this | view in thread ]

  4. identicon
    abc gum, 25 Sep 2012 @ 6:00am

    Do you think the quality of public eduction is deteriorating?
    Do you think merit based pay is a good idea?

    If you answered yes to either of these questions then you might have OECD.
    OECD is a preventable condition which should not go untreated. Ask your education professionals about correcting this problem - before it's too late.

    reply to this | link to this | view in thread ]

  5. identicon
    PRMan, 25 Sep 2012 @ 6:42am

    We used to talk about this...

    At my last job, we had a lot of Asians and we used to talk about this quite a bit. At first, they argued that Asian education is superior to American. But I asked them, "If that's the case, then why is everything cool invented in America?" They really didn't have an answer for that.

    As time went on and we continued to discuss it, what amazed them is what they weren't taught, and that is problem-solving. Asian education tends to be rote memorization and recitation. But Americans learn to problem solve and to do creative works and sports, things that Asian nations don't teach at all. And Americans are also taught to be different, while Asians are typically taught not to stand out.

    So, you could say that instead of trying to turn all our people into smart little clones, we allow our creative people and social people to develop into their more natural skill sets, leading to an overall healthier society, which together accomplish great goals, and finance and market those goals into realities.

    reply to this | link to this | view in thread ]

  6. icon
    Jay (profile), 25 Sep 2012 @ 7:10am

    Re: We used to talk about this...

    That's becoming less important in the US as our priorities have shifted over the years. The US isn't spending the money on education nor is it spending money to make better informed citizens. Compare what was occurring in public education in the 60s to what has happened with education now. The liberal arts are needed yet get little attention for vocational skills.

    reply to this | link to this | view in thread ]

  7. icon
    Seegras (profile), 25 Sep 2012 @ 7:22am

    What should the not-so-bright do?

    Well, it's pretty self-explanatory if you look at the demographics. The brightest drive innovation, and the not-so-bright become bureaucrats and lawyers trying to stifle innovation.

    Or maybe they get into professional sports, as it seems the US universities are mostly about.

    reply to this | link to this | view in thread ]

  8. identicon
    abc gum, 25 Sep 2012 @ 7:28am

    Re: Re: We used to talk about this...

    "The US isn't spending the money on education nor is it spending money to make better informed citizens"

    Because some people believe it is easier to rule the less educated. And it may actually be true, however, the end result may not be what they expect.

    reply to this | link to this | view in thread ]

  9. identicon
    abc gum, 25 Sep 2012 @ 7:29am

    Re: What should the not-so-bright do?

    Or - it may be a bit more complicated than that.

    reply to this | link to this | view in thread ]

  10. icon
    Vic B (profile), 25 Sep 2012 @ 9:22am

    Really? What job was that? How long ago was that? What "cool" stuff are you talking about...your iPhone? It is unquestionable that America's smarts has been and still is driven in great part by foreign talents, often from India, China or former Eastern European countries because their education systems emphasize sciences. Look at the name of the brightest PhD science students from our top universities if you need any validation.
    What these foreign students and countries don't have is the cultural understanding for making the "cool" things you crave to buy although that is slowly changing as you can now find in Asia (Korea, Japan,..) high end toys that are not available in the US. However, it is still easier for Asia to put their scientific and manufacturing minds to work on US or European ideas than it is to figure out what people from another culture on the other side of the world might want to buy. Thankfully the world isn't as homogenous as Americans think it is...

    reply to this | link to this | view in thread ]

  11. identicon
    Anonymous Coward, 25 Sep 2012 @ 10:03am


    So you're going to argue against American education by pointing at the people that come from other countries to learn at American universities...? Stop and think about that for a minute, please.

    reply to this | link to this | view in thread ]

  12. icon
    Vic B (profile), 25 Sep 2012 @ 10:21am

    Re: Re:

    American universities still have unmatched financial resources at their disposal to pursue R&D at the graduate level and that's enough of an incentive to come study in the US.
    If the US makes it difficult for these students to establish themselves here and/or their country universities figure out a way to match US funding then, no they won't come to the US. This is in fact happening and the reason why prestigious US and European universities are opening campuses abroad.

    reply to this | link to this | view in thread ]

  13. icon
    RonKaminsky (profile), 26 Sep 2012 @ 10:49am


    Thanks for giving us a prime example of what the inequalities in our education system generate!

    The results of the education system of the US, and its political system, fit each other ideally. The first-past-the-post system thrives on a relatively uneducated, easily manipulated majority of voters, yet also needs to produce enough highly educated voters to prevent the economy from becoming too uncompetitive.

    reply to this | link to this | view in thread ]

Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Use markdown for basic formatting. HTML is no longer supported.
  Save me a cookie
Follow Techdirt
Techdirt Gear
Shop Now: Techdirt Logo Gear
Report this ad  |  Hide Techdirt ads
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Report this ad  |  Hide Techdirt ads
Recent Stories
Report this ad  |  Hide Techdirt ads


Email This

This feature is only available to registered users. Register or sign in to use it.