On Friday Akamai published a blog post[1] indicating they were not vulnerable
to the Heartbleed attack. They patched OpenSSL to put the private key in a
separate part of memory, surrounded by guard pages. Akamai shared the patch
with the OpenSSL developers[2]. In their blog post they stated: "In our initial
assessment, we did not believe that customer private keys could have been
leaked as a result of this vulnerability;" and imply that this is the result of
their "custom secure allocation scheme."

Reception on Twitter was very positive.

The problem: Akamai's implementation doesn't work and isn't secure.

Security is hard because you can't always tell if what you did works. In this
case, someone with security experience can read their patch in a short amount
of time and figure out that this patch won't protect your private keys.

The patch only enables the 'secure' allocator when parsing RSA keys and only
stores the 'p' 'q' and 'd' fields securely.

First question is then, are 'p', 'q' and 'd' the only sensitive parameters?
NO. (The Chinese Remainder Theorem parameters are included in OpenSSL private
keys by default and they are enough to recover the private key).

Second question: Does OpenSSL ever copy the private key, or parts, after the
key has been read?
YES. 

Third question: Does OpenSSL allocate temporary variables for intermediates
when performing operations with a private key.  Can they be used to recover the
private key?
YES and YES.

Each of these questions leads to one conclusion: Akamai's published 'secure'
allocator will NOT protect your private keys.

Akamai Customers: This patch does not, on its own, protect against private key
disclosure through Heartbleed. This means your certificates on Akamai servers
need to be rotated, and anything sent before then is vulnerable to Heartbleed
compromise. If you send customer passwords to Akamai, you should ask your
customers to change their passwords again. They'll enjoy that.  Even with early
warning, Akamai decided against rotating your certificates.

Akamai: Having the private keys inaccessible is a good defense in depth move.
For this patch to work you have to make sure all sensitive values are stored in
the secure area, not just check that the area looks inaccessible. You can't do
that by keeping the private key in the same process. A review by a security
engineer would have prevented a false sense of security. A version where the
private key and the calculations are in a separate process would be more
secure. If you decide to write that version, I'll gladly see if I can break
that too. 

Perhaps Akamai is not actually running this version in production, but another
'super secure' allocator. In either case they should not be sending out
non-functional, bug ridden patches to the OpenSSL community, while claiming
they protected Akamai against the Heartbleed attack. Andy Ellis, CSO of Akamai,
said on Twitter that the 'secure' allocator was written 13 years ago.  I'm
happy to provide the results of my 15 minute security review, since it is so
overdue. (To be fair, I found the issue in minutes, but confirmation took
longer.)

Saying something is secure means nothing. When competent offense people say
they tried A, B, and C and those failed, then maybe it is secure against A, B,
and C. This tells us nothing about attacks X, Y, Z and 12. Checking for the
attacks that you can think of only provides security up to the limits your
offensive knowledge.

Kind Regards,
    Willem Pinckaers - Lekkertech.


If you want to test each aspect for yourself:

First fix the code that Akamai submitted to OpenSSL. It has at least the
following problems:
1/ No actual guard pages (fixed in their second submission)
2/ The second submission gives back a pointer to the mprotected guard page area
and crashes on use. (cmm_arena += pgsize just after the 2nd mprotect to be able
to actually use it.)
3/ secure_malloc_init is never called, so the 'secure' allocator is never even
enabled. (I called it using (1024*1024, 4096, 1024) as arguments).
4/ Given the above problems I wonder how Akamai manages to run this in
production.
5/ Start the server, do one SSL request to it. Dump all memory, search for
p/q/d and CRT parameters. You will find one copy of p,q, and d in the secure
area. Outside of the 'secure' area you will find p,q, and the CRT parameters.
6/ Enjoy reading the code and spotting the integer overflows.

Still feeling curious?  Here are some additional questions:
If this code was running on the Akamai networks for years, why did they have a
copy laying around that didn't call mprotect? 
If the second version is running on the Akamai servers, why does it return a
pointer to a memory area that is mprotected, to be inaccessible (leading to a
direct crash)?
Why does the code not actually call secure_malloc_init?
Why does the code not check for integer overflows in calloc?
Why does the code not check the return value from mprotect?

[1] https://blogs.akamai.com/2014/04/heartbleed-update.html
[2] http://thread.gmane.org/gmane.comp.encryption.openssl.user/51243