The BitComet Incident, December 2005

BitComet, one of the dominant BitTorrent clients at the time, came out in support of the mainline DHT standard in early June of 2005. It set the tempo for other clients and showed that DHT in general was a really good idea. Yet BitComet was also going to provide the first major client dispute later that year with the release of v0.60.

At the core, the same old argument of wanting absolute control over peers (and thus the private flag) made into a big deal by those desperate to prevent any loss of control.

In short, BitComet decided to implement a system where if the tracker on a private-flagged torrent went offline for a certain period of time, the client would enable DHT, and when the tracker came back up, DHT would be disabled again. This common sense approach provoked outrage based on deliberate misrepresentations, but it generated enough noise that BitComet withdrew the 0.60 client, eventually replacing it with v0.61 and the now ‘standard’ DHT functionality.

More DHT Push-back

This was only the first major push-back against DHT by those desperate to remain in control, and perhaps the most overt. It’s become common for private trackers to ‘demand’ that DHT be disabled in the client and for people to become disorientated as a result, when often those making the claims were the most confused. There have even been claims made that certain clients leak peer info to DHT, a myth we debunked back in 2009.

There is one thing that does ‘encourage’ people to disable DHT though. Especially in 2005 (but still occasionally now) network hardware had problems with DHT. The problem was caused by the significant use of UDP, rather than the more traditional TCP. Routers with limited ram would have it filled by the UDP traffic and lock up or go very slowly.

Thanks to Gigabit LAN and Wireless-N and AC, however, this is less of an issue now than it once was, even with multiple computers running DHT connected. It’s now mostly the province of ISP-provided hardware, which tends to hover on the low-end of the equipment spectrum for cost purposes.

The Pirate Bay Boost, September 2009

Despite all its benefits, DHT adoption was slow until 2009 when it got a huge boost from one of the biggest torrent sites around, The Pirate Bay. In November 2009, they announced they were going to be closing their tracker, and moving towards magnet links, to make the site lighter and quicker.

Magnets were not new, they were part of the DHT spec, but hadn’t gained much use until TPB’s change. Suddenly, lots of users were trying to work out how to get magnets to work, both with clients and browsers, and because of the requirement for DHT in order to make magnets work, DHT itself started becoming more popular. Of course, that meant that a lot of the old rumours about DHT also resurfaced, some of which still persist to this day.

Search at last, March 2011

As mentioned earlier, search was one of Gardner’s main motivations for creating DHT, but it wasn’t until 2011 that a DHT search engine emerged. BTDigg started at the end of March 2011, and was an invaluable resource for many.

The ability to tap into the DHT network as a whole with a decent interface meant it was not only the biggest search engine (since it had pretty much any torrent that ran on a DHT-enabled client without the private flag set) but also the most like mainstream search engines, in that it actively searched and indexed entries, rather than waiting for them to be manually added like traditional indexes.

“The main reason to make BTDigg,” their spokesperson ‘NE’ told TorrentFreak, “was an absence of independent search engines in the BitTorrent network. It was the same situation as the Web in pre-Google era, the era of Web Catalogues.”

Alas, the site closed its web front end earlier this year, remaining available via both TOR and I2P, a situation they put down to their host withdrawing one of their servers, with no reason given. In the meantime, successors like Strike carry on their work

DHT today, 2015

Which brings us to now, ten years after DHT provided the last functional change to the BitTorrent protocol (utp was more of a technological one). DHT has enabled Bittorrent to remain the top dog, and while a dozen-or-so competing P2P systems jockeyed for dominance in the six years after Napster’s launch, nothing has even come close to challenging BitTorrent.

“I researched the question ‘Why is BitTorrent still popular?’ and concluded that the main reason is because the BitTorrent protocol inherits good features from previous protocols (like eMule & Kazaa) and eliminates their weak sides. So the appearance of BitTorrent was logical evolution of previous people’s efforts in file sharing,” NE told us earlier this year.

DHT was clearly the last step, enabling an underlying central mesh network to support the decentralized nature of the individual torrent swarms. It gave it the resilience and longevity to last.

Ten years on though, is there anything that the two architects of DHT would do differently if they could go back to 2005? Loewenstern doesn’t seem to think so.

“It’s quite humbling to have academics who really know their stuff go over your work with a fine toothed comb and find all the problems. There are a couple of BEPs (Bittorrent Enhancement Proposal) for addressing some design decisions but as [BitTorrent creator] Bram Cohen says, ‘the DHT works,’ so I’d rather go make something else cool.”

Gardner agrees saying, “Nothing major to be honest, after 10 years most things have been ironed out. I mean everything needs rewriting once it is complete, that’s the way software works.”

Ten years on and going strong, DHT is clearly ready for the next ten. With over 25 Million users between the two networks, it’s not going anywhere.