A multi-billion-dollar megacorp, Google, apparently needs help to secure its OS.
A startup on a shoestring budget is working to clean up the Android security mess, and has even demonstrated results where other "secure" Android phones have failed, raising questions about Google's willingness to address the widespread vulnerabilities that exist in the world's most popular mobile operating system.

"Copperhead is probably the most exciting thing happening in the world of Android security today," Chris Soghoian, principal technologist with the Speech, Privacy, and Technology Project at the American Civil Liberties Union, tells Ars. "But the enigma with Copperhead is why do they even exist? Why is it that a company as large as Google and with as much money as Google and with such a respected security team—why is it there's anything left for Copperhead to do?"

Copperhead OS, a two-man team based in Toronto, ships a hardened version of Android that aims to integrate Grsecurity and PaX into their distribution. Their OS also includes numerous security enhancements, including a port of OpenBSD’s malloc implementation, compiler hardening, enhanced SELinux policies, and function pointer protection in libc. Unfortunately for security nuts, Copperhead currently only supports Nexus devices.

Google's Android security team have accepted many of Copperhead's patches into their upstream Android Open Source Project (AOSP) code base. But a majority of Copperhead's security enhancements are not likely ever to reach beyond the its small but growing user base, because of performance trade-offs or compatibility issues.

Dan Guido, CEO of Trail of Bits, has also puzzled over the vulnerability gap between the stock Android OS and Copperhead, and points out that the same could not be said for Apple's iOS.

"If I had to imagine a world where there's a Copperhead for iOS, I don't even know what I'd change," he tells Ars. "The Apple team almost always picked the more secure path to go and has found a way to overcome all these performance and user experience issues."

A billion people around the world rely on Android to secure their digital lives. This number is only going to grow. How did we get here, and can Copperhead—or even Google—put out the garbage fire?

A deal with the devil

Google did a deal with the devil for market share, says Soghoian, who has described the current parlous state of Android security as a human rights issue. By giving Original Equipment Manufacturers (OEMs) and wireless carriers control over the end-user experience, Google allowed handset manufacturers to find ways to differentiate their products, and wireless carriers to disable features they thought would threaten their business model.

As a result, Google's power over OEMs—such as Samsung or Motorola, who manufacture and sell Android handsets—consists solely of the Android license and access to the Google Play Store. The AOSP code base is licensed with Apache 2.0, and the kernel uses GPL2, which means there's nothing stopping OEMs from deploying stock Android under a different name. But doing so would also mean losing access to the Play Store. This gives Google significant leverage over OEMs, but by no means absolute control—a competitor willing to forgo the Android trademark and offer customers access to their own app store, as Amazon has done, can walk away from the negotiating table with little to no consequence.

But Soghoian thinks Google isn't trying very hard. The company could, he points out, demand that OEMs implement default full-disk encryption as part of the Android and Play Service licence terms. The company currently requires FDE when the hardware supports it, but extending that requirement to lower-end Android manufacturers might scare off a non-trivial fraction of OEMs—and that would hurt Google's bottom line as an advertising company.

"The important thing to remember," he says, "is that if Google goes nuclear and cuts an OEM from the Google Play store, and Gmail, and Google Maps, and YouTube, Google isn't just hurting that OEM and its customers, it's also hurting itself."

"Every phone that doesn't have YouTube and Google Mail and search is a phone that isn't making money for Google," he adds.

Copperhead, for their part, are not in the business of surveilling users in order to display targeted advertising and so are free to optimise Android for security. Their first challenge was to find a handset to support that offered regular security updates—no small ask.

Just the Nexus, thanks

Most OEMs, for instance Motorola, do not ship the monthly security updates available from the AOSP. The business model for handset manufacturers ends with the sale to a consumer—at which point there are no financial incentives to maintain the devices for the next three years or so.
Copperhead chooses to focus on optimising security for what they believe are the most secure handsets currently available: the Nexus devices whose software, if not hardware, Google controls directly, and which receive prompt monthly security updates.

"What we're doing is starting with the Nexus; a pretty good starting point," Copperhead's Daniel Micay explains. "And we're significantly improving the security of the operating system. We're making a lot of under-the-hood changes and exploit mitigation to make it harder to exploit the vulnerabilities that are there."

Micay's goal is to port the Grsecurity and PaX patches to the Android Linux kernel, which would dramatically improve the security of all Android handsets, but this goal has been stymied by hardware woes—some of which not even Google appears capable of resolving, at least not on its own.

Grsecurity for Android

Grsecurity and its subset PaX harden the Linux kernel by making whole classes of vulnerabilities difficult, if not impossible, to exploit. Micay got his start as the Arch Linux maintainer of the Grsecurity and PaX patches for that distribution, and embraces the same security vision as Brad Spengler, who founded the Grsecurity project, and who has famously clashed with Linus Torvalds over the years for the latter's reluctance to ship a more secure kernel.

But Copperhead's efforts to implement the Grsecurity patches for Android ran into a brick wall: Nexus devices, and indeed all newer mid- to high-end handsets, use the ARM64 architecture. While parts of Grsecurity have been ported to ARM32, little work has been done on ARM64, Micay says, leaving only a small subset of non-architecture-specific code for him to deploy. Porting Grsecurity to ARM64 is not a trivial undertaking—Micay estimates months of work for an experienced engineer, and Spengler and his team are not inclined to help without getting paid.

"Work on Grsecurity is still done in our free time," Spengler says. "We don't have any personal need for ARM64 support, so it's not a priority for us in our limited time. I do have a development board on order, however."

"We would have to research how KERNEXEC/UDEREF functionality could be implemented best on ARM64," he adds. "Given our experience with that research and implementation on ARM, I'm not inclined to do more free work for the full-time funded upstream or multi-billion dollar corporations to rip off."

Given the dramatic security benefits porting Grsecurity to Android would bring, and the relatively low cost of such work, Soghoian wonders why no one is paying Spengler and Micay to do so.

Is Android critical infrastructure?

"In an ideal world, the US Department of Homeland Security would write a check to Spengler for $5 million and keep him busy," Soghoian says, pointing to the Core Infrastructure Initiative (CII), founded after Heartbleed to take better care of critical open source security software like OpenSSL.
Because the Grsecurity project has the potential to positively benefit every Linux user on the planet—including servers, desktops, and more than a billion Android users—Soghoian argues that the project is the kind of thing the CII should be funding.

"The White House announced they were going to put more money into the open source community a few months ago," he says. "It's a totally realistic scenario."

Soghoian also criticises Google for failing to step up to the plate. "Google could pay for the development of Grsecurity using the money found between the cushions of their sofa," he insists. "This is not a big-ticket item in the grand scheme of Google's budget."

But even if ARM64 support were immediately available for the Android kernel, it would still be a year or two before Copperhead—or even Google—could deploy those patched kernels.

Kernel freeze

Linux device drivers have been the operating system's Achilles heel since day one, and the Android platform is no exception. Android phones ship with kernels frozen to ensure driver compatibility—which usually means that a new Android device comes with a kernel that's already a year or two old.
"It's like if you have a printer and the last printer driver made was for Windows 95, you can never upgrade your computer to a newer version," Soghoian explains. "Android is bigger than just Google, and when Google's partners drag their feet it undermines the security of the entire ecosystem."

As an Android device ages, the kernel may get backported security patches, depending on the OEM’s willingness to push updates, but the handset will miss out on the latest security advances, since upgrading the kernel would break hardware compatibility with the drivers.

This ties Copperhead's hands. Given limited resources, Micay says he's focusing on implementing new security improvements to Android, rather than backporting a limited, non-architecture-specific subset of Grsecurity to the older kernels currently running on Nexus devices.

"Nexus devices are stuck on Linux kernel 3.10, which is not supported by PaX and Grsecurity,” he says. “I've chosen to focus on long-term progress so there's no value in porting stuff back to 3.10 since future devices will use different newer kernels."

Google is playing up the security enhancements in Android 7.0, dubbed “Nougat”, which will ship later this year. How significant is the new release in terms of security?

“N” to the rescue?

The Android security team did not respond to requests for comment over the last couple of weeks. On July 27, they published a blog post touting the integration of a subset of Grsecurity patches in Android N.

Micay dismisses this announcement, saying that Google has in reality implemented less than one percent of Grsecurity into Android.

"Android N is making more progress on kernel exploit mitigations than past releases of Android, but it's basic stuff and doesn't change the fact that the kernel is a very soft target," he says. “They're taking baby steps forward for the kernel's security. Security elsewhere in Android is moving much faster though (the mediaserver hardening, multiprocess WebView, SELinux policies, hidepid=2, etc.)”

Google finally responded to our request for comment on Monday, August 1. In a brief statement, Adrian Ludwig, Google's director of Android security, wrote: “Copperhead has been a valuable contributor to Android Open Source project. We appreciate their contributions, and hope that they continue to work on research and development that improves security of the entire Android ecosystem.”

Can Copperhead succeed where others failed?

The marketplace is littered with dead and dying Android security startups. Silent Circle's Blackphone is going nowhere fast, and nor is backdoor-loving Blackberry’s Priv. CyanogenMod OS, although by no means optimised for security, has also found competing with stock Android a far-from-profitable venture.

Absent an unexpected cheque in the mail from Google or the US DHS, how will Copperhead fund its cutting-edge work on securing Android?

The startup currently sells Nexus devices with Copperhead OS preinstalled, and Micay says they are in talks with a number of potential enterprise clients and resellers who would benefit from hardened Android devices customised to suit their users.
"There are no doubt many organisations globally who want full control over the software stacks on their devices, who do not want the cloud services that are bound to the dominant mobile device operating systems, yet they want modern devices," says David Mirza Ahmad of Subgraph, a Copperhead-like effort to secure desktop Linux. "Copperhead could be the answer."

Copperhead ships with F-Droid installed by default, but without Google Play.

Nexus owners comfortable with re-flashing their own devices can, of course, download Copperhead OS and install it themselves. The company also accepts donations and offers a Patreon subscription.

For his part, Micay is in this for the long haul, whether Copperhead is financially successful or not: "Even if I have to get a full-time job I'll still going to be doing this because it matters."

J.M. Porup is a freelance cybersecurity reporter who lives in Toronto. When he dies his epitaph will simply read "assume breach." You can find him on Twitter at @toholdaquill.