Smarter Encryption DuckDuckGo's feature to automatically redirect to HTTPS

The use of HTTPS in "theory" came evento encrypt the content between a user's computer and the server that contains the web page in order to avoid MITM attacks that this case would be "impossible". DuckDuckGo does not escape from this and that is whyand I create a function called Smarter Encryption, designed to automatically send HTTPS requests to HTTP sites if the site supports HTTPS and if it is on the list of sites that can be updated from DuckDuckGo.

At Smarter Encryption there is a long list of websites that contain encrypted versions (HTTPS) of its websites, which DuckDuckGo uses to ensure that you only interact with these encrypted versions. The search engine automatically generates this list as you continuously browse the web.

Besides that raises two main scenarios in which it allows to improve privacy:

1. First, many websites offer an encrypted version (HTTPS) and an unencrypted version (HTTP) of their website, but for various reasons they do not automatically redirect traffic from their encrypted version. DuckDuckGo Smarter Encryption supports this scenario;
2. Another would be if even if a website offers HTTPS and the browsing user accesses one of its web addresses and this first attempt is not encrypted yet causing the browsing behavior to leak.

This is particularly often seen on social media., where many news links are displayed as unencrypted links, exposing the details of what you read in this first HTTP request. DuckDuckGo Smarter Encryption also supports this scenario where it forces access to HTTPS.

This is how Smarter Encryption works:

Clicking or searching for an unsecured domain (http). The http domain it will look on your local list to see if it can be updated immediately. Otherwise it will be converted to SHA-1 hash

The first four characters of this hash  are sent to the anonymous DuckDuckGo service, smarter_encryption.js, which ensures that logs never contain IP addresses or other personal information.

So, like the anonymous requests on DuckDuckGo Search, the publisher (in theory) can't know things about the people making these requests.

However, DuckDuckGo has added another layer of privacy protection to this anonymous service by asking the device to send only the first four characters of the hash domain, so that the service cannot in any way indicate the exact domain that is visited.

Anonymous service returns all hash domains from the full Smarter Encryption list corresponding to the first four characters of the sent hash. Here the device checks the returned hash domains to see if the hash of the domain I know is visiting exactly matches one of the returned hash domains. If so, it gets updated!

The company has created a list of more than 10 million sites that it continues to update. Due to this large size, the list cannot be fully stored in apps or extensions installed on devices. Instead, the publisher stores the busiest sites locally on devices and retains the rest of the list on its servers.

This feature is not restricted to DuckDuckGo users since the code used for Smarter Encryption now It is open source and available on GitHub under the Apache 2.0 license.

Pinterest has taken the plunge and uses Smarter Encryption for its external links. The platform says that after integrating the DuckDuckGo feature, "about 80 percent of outbound traffic now goes over HTTPS, an increase of more than 30 percent."

Regarding the Smarter Encryption code can be consulted in the following link.

For those interested in trying Smarter Encryption, they can download the browser from the Android or iOS app stores. While for Chrome it is offered in the form of a plugin.

The links are these.

• Android
• iOS
• Chrome