Techdirt Has Again Been Removed From Bing And (Mostly) DuckDuckGo
from the not-this-again dept
Welp, here we go again. Last month I wrote about how Techdirt had been deleted from both Bing and DuckDuckGo. Over on the discussion at HackerNews, DDG’s CEO and founder, Gabriel Weinberg, jumped in to the conversation to note that this wasn’t intentional (which we never suspected it was). The resulting conversation on HackerNews is actually pretty interesting, as it appears there was some level of misunderstanding among many users about how much DuckDuckGo relies on Bing for its underlying web search.
Either way, a few hours later DuckDuckGo added back… a single link(!) to Techdirt’s front page, which we mentioned in an update. The next day, I heard from a couple people who said they had reached out to people at Microsoft, and I was told that this sometimes happen, and that the Bing team will eventually fix it (though it might happen faster if something gets public attention). Either way, about a day after I had written about Techdirt being erased, we were back in both Bing and DuckDuckGo and I considered it a one-off bug that had been fixed.
But… it’s back. I happened to just check on Bing and saw that we’re gone again (though now there’s also a big obnoxious box trying to get me to chat):
But, this time it’s weird, because it says there are 2,030 results (should be a lot more!) and then says “some results have been removed,” but it shows no results at all. If you click on the “2” at the bottom, it just takes you right back to this exact same view.
As for DDG, it still displays the one single link to our homepage and nothing else:
While that may be better than nothing, it’s pretty close to nothing. We do still get a fair bit of traffic from people searching for particular stories and now neither Bing nor DDG will send people to those stories. I did some searches on our most popular articles, like the Elon speedrun and the “you’re wrong about 230” and… all the results send people to other sites talking about our article.
Which isn’t really great.
Meanwhile, Google returns 94,900 results which is much closer to our total number of pages.
I’d really like for there to be real competition for Google out there in the search market, but it shouldn’t require me having to nag a trillion dollar company in Redmond every few weeks to put me back into their index.
Filed Under: bing, search index, search results, techdirt
Companies: duckduckgo, microsoft, techdirt
Comments on “Techdirt Has Again Been Removed From Bing And (Mostly) DuckDuckGo”
Legit question, is there a reason you search “site:techdirt.com”? When I search “techdirt” on Bing, I at least get a couple hits with 38,700 results (I still get 0 results on anything past page 1). Granted it’s not an index of multiple news articles, but it’s certainly not returning 0 results.
Re:
“site:techdirt.com” will return results from this site only. Which is the homepage, apparently. This is a url filter that all major search engine seem to support.
“techdirt” will return a variety of results. Of these, Bing and DDG only return the techdirt.com homepage from this site. The other results are references to “techdirt”, such as a Patreon or Twitter page with this name.
Re:
Using “site” shows me specifically what Techdirt pages are indexed rather than other results that just mention Techdirt.
Bing, the AI powered chatbot?
lol, the new clippy!
just as goof!
And yet there are still those that say bing and duck duck go are just as good if not better than google. smh
Re:
Still better then Google tho.
Re: Re:
In what way(s)?
Microsoft at its finest
Google: We’ll give you the best performance, search results (tailored towards our cryptic standards ultimately based on selff-serving ad views,) and you’ll like it, because we own this market based on legacy. We nagged users incessantly to use our browser and it worked.
Microsoft: Hey, we’re late, again! Give us a try with these extraneous, not-fully-thought-out features that are fringe and none of our customers actually asked for. We are holding steady based on a userbase of legacy, and there’s not much you can do about it, cuz you’re lazy and you know it. Our Search experience isn’t a priority. We hope to gain marketshare with forcing users on our platform to use us whether they want to or not. We’ll nag them incessantly. (Seriously have you seen the settings menu on Edge?)
DuckDuckGo: Hey, um, privacy, please? You don’t need to manipulate to get a userbase. We’ll hop on Bing a bit as it seems like a saf(er) long term bet. Here’s a clean site and a few apps to help your privacy.
Quoting Ian Fleming
“Once is happenstance. Twice is coincidence. Three times is enemy action.” -Ian Fleming
Yeah, I’ve noticed Bing seems a bit trigger happy about domain unexplained removals (for example https://www.bing.com/search?q=site%3Aretrocrush.tv has been broken since at least late 2021).
timed out?
Maybe Bing removes older data from their index after a certain time, while their crawler is too slow to re-index Techdirt before then?
Maybe because of how many links Techdirt has it’s getting tagged as a linkfarm? Do linkfarms still exist?
Re:
Yes. And those known as web databases are still helpful when properly curated.
Not /everyone/ likes people making suggestions to them based on the opinions of the one suggesting. A good number use sites and web apps that index without algorithms.
This comment has been flagged by the community. Click here to show it.
After all the stories about how Conservatives aren’t being censored.
Just
Desserts
Re: Re:
wat
Every time something like this happens to Techdirt, Mike has consistently said that it is extremely likely not the result of some conspiracy, but instead inadvertent human or machine error. He’s not turning around and saying “the conspiracy is real, but its only targeted at us!”
Re:
“After all the stories about how Conservatives aren’t being censored.”
They’re not.
Put up or shutup.
What conservatives call censorship is when they are not allowed to blast megaphones in the ears of those who are not at all interested in what is being blasted.
Re:
You comment seems to endorse censorship of anyone not being a “conservative”. Seems to me you are kind of stupid to think like that.
This comment has been flagged by the community. Click here to show it.
Re:
lol bye straight trash
Re: Re:
You can’t hide us forever, Hyman. We know you’re downvoting the truth.
The world is moving onto a new paradigm shift that transcends the limitations of straight people. You’re not needed anymore. You’re like an asshole on the elbow of humanity, briefly amusing but long since overstayed your welcome.
Re:
“After all the stories about how Conservatives aren’t being censored”
All you guys have to do is present an example of that actually happening because they’re conservatives and not the usual “oh, yeah, if you look into that user they were bigoted monsters who lied about someone they didn’t like”.
We’re still waiting for those examples. It’s hard to get people on your side when you’re lying about the issues you say you’re concerned about, unless you are actually admitting that “hateful bigot” and “conservative” are synonyms.
Try Kagi
A search on Kagi seems to pull back exactly what you’d expect: the Elon speedrun and you’re wrong about section 230
Just saying, there are some non-trillion dollar companies who seem to be building viable competitors to Google search.
Re:
Your Kagi links all give me a “403 Forbidden” error.
Re: Re:
I was getting the 403 yesterday, but it’s “working” now. Except all I get is a login page. If I click “create trial account”, it sends me to a “signup” page that redirects back to the same “signin” page I just left.
According to the “Why do we need an email address?” link, this is a paid service that only works when logged in. Under “Why should I trust you?”, they say “We do not log search queries. Queries you type are never associated with your account. The simple reason is we don’t have any reason to do so, as it would only be a liability for us.”
That seems pretty tone-deaf to me. Search-engine queries are some of the most privacy-sensitive data there is, and they’re asking me to give them those queries in a way trivially linkable to myself, on only the promise they’ll protect my privacy. Well, they’re an American company, and are subject to the same secret orders as any other (remember Lavabit?). Sometimes companies change management or ownership. The FTC’s history is full of companies hit with consent decrees—but no actual financial “liability”—on the basis of broken privacy promises.
There are zero-knowledge cryptosystems that could, in theory, be used to prove subscribership, without revealing one’s identity. It might be difficult to turn that into an easy-to-use interface, and I don’t think most people have the mathematical knowledge to trust in such a thing. But to basically answer “why should we trust you?” with “we have no reason to act in an untrustworthy way” is unsatisfactory. That’s like Google saying they have a policy to “not be evil”.
Re: Paying for a search engine? No thanks.
Sorry, what Kagi is promising doesn’t seem like anything for which I’m willing to pay. No dice.
Re: Re: Kagi
Besides, isn’t Kagi based around a meta-search for Google and Bing? I tried it out, did a site: search for one of our sites, and the results were identical to what those two brought up. I know this since Google brings up framesets that we haven’t used since the 2000s in the top 10, and guess what? So does Kagi. Plus a bunch of weird results that only Bing was finding.
Still in Beta?
Bing, works just as well as Windows!!
Both techdirt and torrentfreak are classified by some filtering providers as torrent/piracy
When I travel I can circumvent that by using my private VPN
If you travel and are a regular reader of techdirt or tottentfrrak you might want to consider having a VPN on your home network so that if a library or coffee shop blocks it you can circumvent filtering with your VPN
You are not committing any crime doing this, at least in Canada, Mexico, or the United States.
There is no law in any of Canada’s 13 provinces, Mexico’s 31 states, americas 50 states, or at the federal level in any of these countries that makes it a crime to bypass filtering or firewalls
Re:
You can stop harping on the fact that VPNs are legal. Everyone around here is aware of it.
This comment has been flagged by the community. Click here to show it.
So what?
try deleting some articles
I’ve heard somewhere that if you delete your old articles it’ll improve your search rankings.
Or something…
/s
Re: Not how it works
Unfortunately, that’s just one of many myths in my industry of SEO. Deleting old content does not make a site “fresh” and since Bing is fully capable of indexing tens of millions of pages from any one site, with content going back 20 years on some sites, the issue is something else entirely.
Re: Re:
I believe they’re referring to a recent news story that CNET tried doing exactly that.
Google censors too
I saw a documentary about Ukrainian artists and tried to search for one of them. Google “could not find any results for this search” but bing could.
I have had an instance in the past where my vpn software stopped Google forwarding a search to the Australian intelligence services to put me on a watch list. Nord vpn did not even know that.
Did Techdirt get Goolniked?
Re:
Thomas Goolnik!
Thomas Goolnik!
Thomas Goolnik! Thomas Goolnik!
Thomas Goolnik Goolniiiiiik!
Re:
Did the “Inventor” of email discover DMCA takedowns?
Why did it happen?
We’re you able to get ahold of anyone to explain why the site was delisted?
Is it really just an “oops” or is there a person setting the search results to not return the site?
And of course, there’s the usual statement: if it’s happening to a site as large as yours, who else is this happening to?
If nothing else, Bing has always prided itself on returning different results than Google will. Honestly though I’ve always assumed (but I do not know for sure) it’s because Bing favours older SEO practices such as meta keywords.
Maybe Bing just doesn’t like the sitemap generated by Jetpack?
Yandex shows at least 2k hits.
Cause RU is really really all about open information.
Baidu only returns 2 links to a jobs.techdirt subsite.
So… the Chinese search engine is better than Bing.
Re: even more search engines
Sogou (another popular Chinese search engine) returns plenty of results, but mostly for older articles (2006-2010)
Daum (Korean search engine) does ok, returning results from 2016-2020
Naver (another Korean search engine) actually returns good results from a wide range of dates, though for some reason you have to add
&where=webto the end of the url.Personally I like Daum better when searching in Korean but it looks like Naver gets a win here.
Interesting...
Since I’m not a tinfoil hat wearer, I’ll continue to use Google, which works as it should.
I asked the Bing chatbot why it was not showing any techdirt.com results
It replied: “That’s a good question. It seems that Bing has removed or de-indexed the entire techdirt.com domain from its search results, which affects other search engines that rely on Bing, such as DuckDuckGo and Ecosia. The reason for this is not clear, but it could be related to some of the topics that Techdirt covers, such as technology, innovation, policy, and civil liberties. “
Re:
That’s pretty much the same response as the last time this happened…
https://www.techdirt.com/2023/07/27/techdirt-has-been-deleted-from-bing-and-duckduckgo
Predictable
Hi Mike, you probably know that when I first alerted you to this in January 2023 I had been running site: searches for half a year. We have a site the same age as Techdirt where usually between 10 and 55 of the thousands of pages we actually have show up on Bing, and this has happened since last year. (Of those 55, 40 per cent of the results on the pages after the first are repeats.) The day you wrote your last article on this topic, we “recovered” to 1,700—but soon we were back to single digits. That 1,700 day was pretty good as it meant we were getting close to The New York Times, which Bing shows has 5,000 pages. Talk about the big time. (For a laugh, try running site:microsoft.com. Google and Mojeek have more pages than Bing’s 195,000.)
I’ve gone through all the hoops that Microsoft people and their support helpers suggest for nought, and it’s clear to me that something inside Bing is very dead—and it wouldn’t be the first time I’ve observed a big tech site fail in a big way. WorldWideWebSize.com shows Bing’s index has collapsed, too, at least when Bing doesn’t lie about the number of results it has for any given search. Alltheweb and Inktomi probably had bigger indices than Bing does today.
You mention competition for Google. There is Mojeek, which shows c. 47,000 results for site:techdirt.com. It’s the biggest index I know in the occident outside of Google. It was because of Bing’s failure that I began hunting, and I haven’t been disappointed with Mojeek’s site: search results. We use them for our site’s search function now.
Re:
I just tried “site:techdirt.com” on Mojeek… and got 0 results.
Then it suggested “site:techdirt.com techdirt” and I get… 47,000 responses.
Just for the hell of it I tried “site:techdirt.com techdirt” on DDG and Bing… and then I start getting results again as well. So… that’s all very weird.
Re: Re: Mojeek
By design, Mojeek requires a keyword. I guess Bing now does, too, but neglected to tell anyone?