Wikia's Search Strategy Heavy On Buzzwords And FUD

from the like-a-broken-record dept

Wikia, Jimmy Wales' for-profit venture, has been talking for awhile about taking on Google in the search space. The company believes it can do better by augmenting traditional algorithm-based search with wiki-like collaboration and human editors. So far, the company doesn't have anything to show for its efforts, but it recently announced the purchase of the open source web crawler Grub from LookSmart (remember them?). As part of the company's PR efforts, Wales has tried to make the case that existing search tools are "broken" and that another party needs to come along and fix it. This same line, that Google is broken for whatever reasons, gets repeated by every fledgling search startup out there. While Google has its share of problems (spam, etc.), it's unlikely that most users would see things as being so bad. In the end, neither FUD nor buzzwords, like "wiki", "open source" or "semantic web" will be enough to dethrone Google if the underlying product isn't clearly superior.
Hide this

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: search, wales, wiki
Companies: google, wikia

Reader Comments

Subscribe: RSS

View by: Time | Thread

  1. identicon
    Trerro, 30 Jul 2007 @ 9:12pm

    There's definitely a niche for this

    The discussion on this post seems to be a debate on whether or not Google is, in fact, 'broken.' Broken's probably a bad word to use, since Google is obviously not in a useless state, but there's many things people don't like about it, particularly the number of ways of raising Pagerank without raising actual popularity or usefulness of your site. I hardly think this makes Google 'broken', but it's certainly flawed. Of course, everything's flawed, and there's always room for improvement. The question for any potential Google competition is very obvious, but much harder to answer - how will you do it better?

    Google was successful on a very simple, but important, idea - have an algorithm that has a way of determining a site's value without just scanning for key words in meta tags or in the page itself. Initially this was simply link count, though of course that's been refined over time. Still, the Pagerank system alone doesn't work well in some cases - especially when you're researching something controversial... the more supported position is of course, always going to get linked to more. In cases where it's the WAY more supported position, the less popular side won't even appear on the first page.

    DMOZ was mentioned, and while that's a great directory, very few things are going to appear there unless they're already easily searched for, or well-known in their community - and that's if you have GOOD editor on the topic. Assuming the topic in question has a good editor, this often makes it a great place to quickly find some relevant sites, and it DOES do a great job of filtering out the crap that gets good search results but has no real value, but it's hard for a relatively unknown site to ever get known through something like DMOZ. It's a great supplement to a search engine, but it really can't replace one. It's also based heavily on each category having (usually one) editor. Even assuming the editor is very professional and mostly unbiased, you run into another issue - what happens when he simply gets sick of updating his section? An entire category can go years without updates, because not only do people have to realize it's not getting updated, but someone with an interest in fixing that problem - and enough knowledge to do it - needs to appear.

    The wiki approach, in theory anyway, makes it much easier to keep stuff up to date, and the idea of a discussion page for all the search results allows you to quickly scan for dissenting opinions on the results - very good when you're trying to fully understand all sides of an issue - especially when the discussion page may have a link you want that the search results don't.

    The problem is that for that approach to work, you need a good system of checks and balances, and you need VERY fast responses to vandalism (Wikipedia gets hit often enough, but a search engine as popular as Google would get hit CONSTANTLY.) Making page edits take a few days to actually commit would probably help (especially if you have section editors DMOZ to check periodically for crap.)

    Ideally, I think you need a mixture of the wiki approach and the DMOZ approach - let anyone edit it, but make sure someone's there to actually review the edits - ideally more than one someone. Also make sure that there's a way to make sure all of these chief editors are actually still active (simply checking login frequency should work) and have some sort of way of removing a heavily biased section leader.
    Of course, none of this works if new but relevant pages aren't discovered... easy enough in a topic where there's a strong community and the editors are probably active in said community (though this creates another issue... if the community itself is controversial, you need to ensure links to sites AGAINST it can appear too...), but much harder for a more academic topic, or simply a relatively obscure one... so you're still going to need crawler bots finding stuff, and then people manually sifting through it.
    This raises another problem though... manually sifting through bot results is not exactly interesting, and unlike wikipedia, this would be a for profit company - people might be more than happy to submit known links on a favorite topic, but VERY few people are going to do raw data processing for free. You'd need a pretty big staff to fully support the effort.

    Search engines are one of those things where almost anyone can think of an improvement. Actually getting it implemented is another story, however. I do think it's only a matter of time before a viable Google competitor appears. Wales has a chance, but he's going to need more than just wiki mechanics, and it's almost certainly going to take a lot of trial and error to get right.

Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here

Subscribe to the Techdirt Daily newsletter

Comment Options:

  • Use markdown. Use plain text.
  • Make this the First Word or Last Word. No thanks. (get credits or sign in to see balance)    
  • Remember name/email/url (set a cookie)

Follow Techdirt
Sponsored Promotion
Public Money, Public Code - Sign The Open Letter at
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Recent Stories

This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it

Email This

This feature is only available to registered users. Register or sign in to use it.