DuckDuckGo ‘Restores’ Pirate Sites and Points to Bing as Culprit

Last week we noticed that several high-profile "pirate" sites were unfindable in DuckDuckGo's search results. It wasn't clear why these domains had been 'removed' but after some back and forth, Bing is now mentioned as the culprit. DuckDuckGo is working hard to mitigate the problem, which is more widespread than we initially reported.

duckduckIt’s not a secret that Google and Bing are tweaking their search results to accommodate copyright holders.

However, it was a surprise to see that pirate and related sites also started to disappear from DuckDuckGo, as we reported last week.

An initial review revealed that the official domains for sites such as The Pirate Bay, Fmovies, and YouTube-DL were no longer showing up. This led us to conclude that they had been removed for some reason or another.

Not Removed?
DuckDuckGo didn’t immediately respond to our findings but after two days DuckDuckGo founder and CEO Gabriel Weinberg reacted on Twitter.

According to Weinberg, the search engine never removed anything. Instead, the problems were attributed to the “site:” search operator we used as an example in our article. Apparently, that is broken.

“We are not ‘purging’ YouTube-dl or The Pirate Bay and they both have actually been continuously available in our results if you search for them by name (which most people do). Our site: operator (which hardly anyone uses) is having issues which we are looking into,” Weinberg wrote.

tftweet
These comments suggest that our coverage was wrong. However, that’s not the full picture, far from it.

When we wrote our article, we didn’t only use the “site:” operator. We also searched for keywords directly, without the domain names showing up. This means that the sites were unfindable for another reason.

DuckDuckGo Restoral
Coinciding with the tweets from DuckDuckGo’s CEO, several of the domains we listed in our article started to reappear in the search engine. The Pirate Bay homepage reappeared, and the same applies to YouTube-DL and Fmovies, even when we use the ‘broken’ “site:” operator.

These restorals supported the suggestion that this was merely a temporary technical issue. However, it wasn’t hard to find other examples of domains that were still unfindable.

On Sunday, we updated our article to add that YTMP3.cc and Prostylex were not appearing in search results; a few hours later these reappeared as well.

prostylex
While we were amazed by this magical superpower that allows us to revive websites by simply mentioning them, someone else was pulling the strings. Behind the scenes, DuckDuckGo was working hard to restore sites that were mentioned in the media.

It’s Bing
At this point, it became clear that the search engine wasn’t at all happy with what was happening. They never actively removed any of these sites. Instead, a third-party data provider ‘removed’ the results for them.

Like many other smaller search engines, DuckDuckGo uses hundreds of data sources, including Bing. After some back and forths, DuckDuckGo’s spokesperson informed us that Microsoft’s search engine was the culprit.

“Yes, this is related to using data from Bing,” DuckDuckGo’s Senior Communications Manager Allison Goodman said.

Goodman also asked us to share a list of additional sites that were affected, so they could look into these. And indeed, a few hours after we sent over more affected domains such as 1fichier.com, 2embed.ru, and torrentgalaxy.com, these were restored as well.

Broader Issue
It’s understandable that DuckDuckGo wasn’t happy with the coverage. However, the problem was real. And since it’s emanating from Bing, other smaller search engines that rely on that data may be affected as well.

“Since these occurrences originated on Bing, they were passed down to our results, as well as other Bing syndication partners,” Goodman clarifies.

For DuckDuckGo, it may be tricky to resolve the issue permanently as long as it relies on Bing. Aside from the potential legal implications of actively restoring pirate sites, there are dozens, if not hundreds of other domains that are still unfindable.

We don’t intend to keep hammering on this but, at the time of writing, the immensely popular YouTube ripper 320ytmp3.com is not listed in search results, and the same applies to the streaming service gimy.app.

Inaccurate DMCA Removals?
While looking into these issues, we noticed that Bing also affects DuckDuckGo in other ways. From what we can see, the DMCA removals also spill over, including the inaccurate ones.

For example, some news articles from TorrentFreak are not available in Bing, presumably due to takedown requests. That includes this news report about a leaked Game of Thrones episode.

A few weeks ago Warner Bros. asked Google to remove this article. Google refused to do so, but it looks like Bing complied as the article is unfindable there. When we search for the title or even the URL, it’s not there.

The same applies to DuckDuckGo, where the article is not listed. Instead, the top hit is a copy of our reporting. In Google’s search results our news article is the first result, as expected.

All in all, we want to emphasize that these issues are not caused by DuckDuckGo, which has been trying hard to mitigate the problem. However, the issues do exist and it is clearly more than a broken search functionality.