Final Assignment

Jeffrey Ryan

OpenSSL Analysis
Abstract
OpenSSL is very popular SSL/TLS toolkit and cryptography library used by millions of websites
on the Internet [1,2]. Since its inception in 1998 it has been plagued with design flaws and lack
of code maintenance. A couple of alternatives to OpenSSL have emerged since the Heartbleed
vulnerability in April 2014 – LibreSSL by the OpenBSD development team and BoringSSL by
Google. This paper will first describe some of the design flaws found by the LibreSSL team and
others. Some of OpenSSL’s deployment and maintenance shortcomings are then elaborated upon
with supporting examples. Secure software design principles that LibreSSL developers and even
the OpenSSL developers are following in the Heartbleed aftermath will be explained. The paper
concludes with suggested testing, verification, and validation techniques to be applied to the
OpenSSL and/or LibreSSL codebases.
Design Flaws
OpenSSL has a number of design flaws that were just too much to overcome so a team of
OpenBSD developers, led by Theo de Raadt, forked from OpenSSL v1.0.1g to create an
OpenSSL alternative called LibreSSL. Some of the design flaws noted include [3, 4]:
 Difficult to read code
 Failure to check function return codes
 Custom implementation of a memory management abstraction layer
 Custom implementations of standard C functions like printf ()
 Poor random number generator design
OpenSSL source files generally lack code comments, have bad indentations, contain nested
if/else statements several layers deep, and are scattered with nested preprocessor directives
(#ifdef/#ifndef) up to 17 layers deep [5]. Improving code readability by itself doesn’t reduce
vulnerabilities but it does lower the barrier of entry for other developers to review and redesign
code to make it more secure [3].
Failure to check return codes from functions like malloc () could result in the usage of a NULL
pointer. An attempt to dereference a NULL pointer can result in several things, the worst being a
system crash due to a segmentation fault.
Not every OS that OpenSSL can run on contains the malloc () function or other memory
allocation functions. OpenSSL used its own memory management functions to address this issue,
like CRYPTO_malloc () and CRYPTO_realloc (). Problems with this include not freeing
memory after use, using memory after free, memory leaks, and the inability of static code
analyzers to see any memory management issues because of the abstraction. Buffer overflow
vulnerabilities were essentially impossible to detect [3] and garbage collectors failed to see the
memory leaks.
The developers of OpenSSL added a layer of abstraction to some standard C functions, most
notably the printf family of functions – printf (), vprintf (), snprintf (), and vsnprintf (). All of
these would have BIO_ prepended to the function name, i.e. BIO_printf (). A case can be made
for creating wrapper functions but in the case of OpenSSL, the reasoning is unclear. Performance
took a hit when printing to STDOUT, sometimes taking almost 5X longer [6] than just using the
built-in printf () function.
The OpenSSL random number generator can return a random number even if the entropy pool is
small, raising the risk of predicting what the random number will be, therefore allowing an
attacker to predict private encryption keys [3].
Deployment and Maintenance Shortcomings
OpenSSL developers tried to make it compatible with all platforms. This lofty goal was a major
contributor to the substandard code readability described earlier. It would also introduce so many
compiler and configuration options that deployment for any given platform became difficult.
Sounds redundant but a TLS best practice is to use secure cipher suites – not older unsecure
cipher suites even if still supported by the library [9]. OpenSSL does not use standard cipher
suite names according to their own documentation [7]. For example, the NSA recommends
minimum levels of security when utilizing TLS in RFC 6460 – Suite B Profile for Transport
Layer Security (TLS) [8]. A server configured with 128 bits of security must be able to use the
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 cipher suite. OpenSSL condensed
this cipher suite name to ECDHE-ECDSA-AES128-GCM-SHA256, still long but it gets the
point across. Every other cipher suite name implemented in OpenSSL has been condensed,
sometimes even leaving out the authentication used, in which case RSA authentication is to be
assumed. If an organization is attempting to deploy OpenSSL, they need to be aware of the non-
standard cipher suite names and assumptions used.
LibreSSL developers removed approximately 90,000 lines of C code in just the first week after
the fork, code that would not have to be maintained going forward. It is also important to keep in
mind that smaller codebases reduce the chance of bugs, a key design principle described later.
Obsolete operating system code like VMS, DOS, Win16, and EBCDIC, an encoding scheme
used by IBM mainframes back in the 1960s, were removed. Antiquated features, old CPU-
specific code, and old compiler-specific code were also removed [3].
OpenSSL recently made significant changes to the API with the 1.1.0 release. Many data
structures were hidden and access functions provided for modification of their values. On the
surface this is considered good practice but there needs to be a transition plan [11] for users of
older OpenSSL versions. Significant code rework now needs to be performed on existing
projects using older versions of SSL if they wish to migrate to v1.1.0. This code rework in turn
could introduce new bugs and vulnerabilities.
Bugs would be reported but often take many years to be fixed, if ever. The graph in Figure 1 [10]
shows the number of open tickets vs total number of tickets closed. Well maintained code would
have the number of open tickets fluctuate around a relatively low and manageable number but
you can see it just keeps growing at a steady rate right until when Heartbleed discovered. The
user community became frustrated with the apparent lack of interest from OpenSSL developers
to resolve tickets leading to less community involvement.
 Figure 1 - OpenSSL RT Ticket Tracking System. Blue line = total # of closed tickets, Red line = # of open tickets

More secure code comes at a price though, some of it paid for in performance. LibreSSL is at
best comparable to and often slower than OpenSSL [3], as the cipher suite tests in Figure 2 show.
The performance hit is mainly due to increased memory sanitization and disabled optimizations.

Figure 2
Key Principles of Secure Software System Design
The LibreSSL development team strives to follow several key design principles and best
practices.
One of the first design principles the LibreSSL team tried to adhere to is the Principle of
Economy of Mechanism [12] – security mechanisms should be as simple as possible. If the
implementation and security mechanisms are complex, the possibility of security vulnerabilities
is higher. They eliminated hundreds of files and tens of thousands of lines of code to get the
codebase as small and simple as possible. Smaller, simpler code also makes the testing and
verification process easier.

Figure 3 - LibreSSL vs OpenSSL # of files, lines, words, and characters [21]

Both the OpenSSL and LibreSSL teams follow the Principle of Open Design [12] – the security
of a mechanism should not depend on the secrecy of its design or implementation. The source
code for both SSL/TLS cryptographic libraries is freely available for inspection by anyone who
has the interest. Unfortunately, the OpenSSL code suffers from code readability issues described
earlier.
LibreSSL tries to adhere to the Principle of Fail-Safe Defaults [12] – unless a subject is given
explicit access to an object, it should be denied access to that object. OpenSSL enables SSLv2
and SSLv3 by default unless you explicitly configure it with the -no-ssl2 or -no-ssl3 options [13].
Support for SSLv2, which is susceptible to the DROWN [14] attack, was never available in
LibreSSL. Support for SSLv3, which contains a protocol vulnerability known as POODLE [15],
was disabled in LibreSSL v2.2 and completely removed in v2.3. OpenSSL has many
configuration options that must be specified to disable features. Proper adherence to this
principle would have the user specify configuration options of features they wish to be enabled,
not disabled.
A design principle that is more difficult to follow is the Principle of Psychological Acceptability
[12] – security mechanisms should not make the resource more difficult to access than if the
security mechanisms were not present. LibreSSL and OpenSSL are mostly invisible to end users,
any user that has a need for secure communications between their web browser and a web server.
This principle can also apply to developers, although not matching the definition exactly.
OpenSSL can be difficult to use in applications, especially the TLS protocol. LibreSSL created a
new library called libtls that simplifies usage of the library. It even has the ability to warn
developers if they attempt to use unsecure older versions of TLS or some of its unsecure cipher
suites.
The OpenSSL Foundation didn’t have much funding until the Core Infrastructure Initiative (CII)
was created shortly after Heartbleed in 2014 [19]. It is made up of various technology companies
providing funding to open source projects, OpenSSL being the first benefactor. The funding
provided the means to hire two full time developers for designing and implementing best
practices, handling bug reports in a timely manner, and performing more thorough code reviews
of volunteer submissions. Areas of the OpenSSL library getting an overhaul are the TLS state
machine, memory management, the interface to crypto functions, and I/O layers (think
BIO_printf ()).
A best practice the OpenSSL team could do better with is consolidating code that performs
similar tasks into one location like a library or header file. Trying to maintain this code in more
than one place multiplies the amount of work required if changes are necessary. And to reiterate,
more code means more possibility of bugs and vulnerabilities.
The OpenSSL team has developed and published a coding style document [20]. It describes
indentation, braces placement, naming conventions, and commenting guidelines.
Testing and Verification
Testing and verification of the OpenSSL and LibreSSL libraries is important for such widely
used software libraries. Static and dynamic code analysis, automated verification including
regression testing, and performance testing should all be performed at the appropriate time
depending on the type of testing or analysis.
The LibreSSL team uses the Coverity scanner for static analysis of their code. It has found
several issues in OpenSSL, and more underlying issues once its memory abstraction layer was
removed. The Coverity scanner has found dead code, memory leaks, and logic errors among
others [17].
Several code IDEs can be setup to perform static code analysis automatically. Visual Studio
2013 has a static code analyzer built in that can be run manually or automatically at build time.
CLion is a C/C++ IDE that can run on Linux and contains on-the-fly code analysis [22]. Eclipse
is another IDE that can run on Linux and has several plug-in modules for static code analysis.
Dynamic analysis can be performed with Valgrind [18]. Valgrind is a framework of tools for
memory error detection, thread error detection, cache profiling, and heap profiling. A downside
is it can slow down execution of the binary by up to 100x but since it does not need to be run in
production environments, this is normally acceptable. The Memcheck tool does exactly what it
sounds like – detecting memory management problems. Like Coverity, Valgrind was unable to
catch memory errors until the memory management abstraction layer was removed. Helgrind can
be used to locate data races in multithreaded applications. DRD is similar to Helgrind but
specializes in C and C++ code. Finally, there a couple of profilers, Cachegrind and Massif, that
can be used to profile the cache and heap to improve performance.
Due to the longevity of OpenSSL and LibreSSL, regression testing is important to ensure
existing code still functions as designed after a code modification, whether it is a feature
addition, code refactoring, or bug fix. Valgrind has a regression test option using a make file
option.
Performance testing is yet another test that Valgrind can do with a make file option.
Conclusion
Some of the design flaws of OpenSSL were described, leading to the creation of LibreSSL, an
OpenSSL alternative. Several deployment and maintenance shortcomings were also provided
culminating in a graph showing how performance is affected when using the more secure
LibreSSL library. A handful of key secure software design principles and best practices were
portrayed for both LibreSSL and OpenSSL development going forward. Formal testing and
verification methods are outlined including manual and automated code analysis, regression
testing, and performance testing.
References
1. Welcome to OpenSSL! Retrieved from https://www.openssl.org/
2. OpenSSL Usage Statistics. Retrieved from https://trends.builtwith.com/Server/OpenSSL
3. Khan, F (October 21, 2015) LibreSSL: The Secure OpenSSL Alternative. Retrieved from
http://resources.infosecinstitute.com/libressl-the-secure-openssl-alternative/
4. Brodkin, J (April 22, 2014) OpenSSL code beyond repair, claims creator of “LibreSSL” fork.
Retrieved from https://arstechnica.com/information-technology/2014/04/openssl-code-beyond-
repair-claims-creator-of-libressl-fork/
5. Beck, B (May 17, 2014) LibreSSL – The first 30 days and the Future. Retrieved from
https://www.openbsd.org/papers/bsdcan14-libressl/mgp00012.html
6. Deanie (September 25, 2014) What are the benefits to using BIO_printf() instead of printf()?.
Retrieved from http://stackoverflow.com/questions/26048277/what-are-the-benefits-to-using-
bio-printf-instead-of-printf
7. OpenSSL Documentation. Retrieved from
https://www.openssl.org/docs/man1.0.2/apps/ciphers.html
8. Salter et al. (January 2012) RFC 6460 – Suite B Profile for Transport Layer Security (TLS).
Retrieved from https://tools.ietf.org/html/rfc6460
9. Ristic, I. (March 31, 2017) SSL and TLS Deployment Best Practices. Retrieved from
https://github.com/ssllabs/research/wiki/SSL-and-TLS-Deployment-Best-Practices
10. Salz, R. (October 12, 2016) Fact to Face: Goodbye RT, Hello GitHub. Retrieved from
https://www.openssl.org/blog/blog/2016/10/12/f2f-rt-github/
11. Sing, J (December 30, 2016) OpenSSL 1.1 API migration path (or lack thereof…). Retrieved
from https://www.mail-archive.com/tech@openbsd.org/msg36437.html
12. Bishop, M. (January 10, 2003) Design Principles for Security Mechanisms. Retrieved from
http://www.informit.com/articles/article.aspx?p=30487&seqNum=2
13. OpenSSL - Compilation and Installation. Retrieved from
https://wiki.openssl.org/index.php/Compilation_and_Installation
14. Wikipedia - DROWN Attack. Retrieved from https://en.wikipedia.org/wiki/DROWN_attack
15. US-CERT (October 17, 2014) SSL 3.0 Protocol Vulnerability and POODLE Attack.
Retrieved from https://www.us-cert.gov/ncas/alerts/TA14-290A
16. Penzin, V (October 10, 2015) Linux: LibreSSL. Retrieved from http://penzin.net/libressl.html
17. FreeBSD – LibreSSL/ History. Retrieved from https://wiki.freebsd.org/LibreSSL/History
18. Valgrind. Retrieved from http://valgrind.org/
19. Mimoso, M (April 29, 2015) OpenSSL Past, Present and Future. Retrieved from
https://threatpost.com/openssl-past-present-and-future/112485/
20. OpenSSL Coding Style (January 12, 2015). Retrieved from
https://www.openssl.org/policies/codingstyle.html
21. Ducklin, P. (July 13, 2014) LibreSSL ships first portable version, now up to 48% less huge!
Retrieved from https://nakedsecurity.sophos.com/2014/07/13/libressl-ships-first-portable-
version-now-up-to-48-less-huge/
22. CLion A cross-platform IDE for C and C++. Retrieved from
https://www.jetbrains.com/clion/