There are places in the digital world that conventional search engines cannot reach. The search engines pull information from the part of the internet known as the “surface web,” but beneath it lies an immeasurable space called the “deep web” that is not indexed. The deep web is also home to the murky territory called the “dark web.”
Search engines use robots (called crawlers or spiders – in essence, applications that run automated tasks) to fetch data from websites they are allowed to access. They do this by following hyperlinks, which are useless when content is private (authentication), restricted (subscription), generated dynamically (temporary), or intended for a limited audience.
Behind the visible internet lies a huge world of online databases, intranet systems, internal company sites, content reserved for members or subscribers, information that is password protected or meant for specific users, and encrypted networks. All this forms the deep web, and the larger public can reach much of it through a direct link.
The unindexed part of the internet is also home to anonymity networks, such as Tor and I2P. At this level, all traffic is encrypted and IP addresses for services and users are theoretically impossible to determine, conditions that attract a strong criminal element and have earned this space the nickname “dark web,” which many users believe is just for illegal activities.
In the case of Tor, thousands of computers form an encrypted network that sits on top of the normal internet and is available only through special tools and protocols not supported by regular web browsers. A connection through this network passes through three nodes that encrypt it at each step and know only where to relay it next, unaware of the full route.
While the dark web harbors websites that provide a wide array of goods outside the law, not all activity is illegal in this part of the web. Anonymity does not benefit just criminals. Activists, whistleblowers, and publications that are censored in some countries also turn to the dark web to make themselves heard.
For a regular user, this isolated world may appear hostile, since accessing it requires specialized software or intermediary services and websites here follow a naming convention that is not on the human-readable side. Instead of the legible domains on the surface web, those on the dark web are typically a string of random characters that end with the .onion extension.
Many operators of hidden services avoid attention and keep resources available for a smaller number of reasons and don’t advertise their websites to search engines operating on the dark web. For these reasons, without the exact address you want to load, reaching certain websites can be difficult.
Just like in the real word, all that you see is not all there is. The deeper you go, the darker it gets. The surface web is enough for most of us, and we access the deep web every day when we log into a service or pull statistics from a database. The dark web is where good and bad coexist under the rule of anonymity.