How To Fix POODLE (And Why You’re Probably Still Vulnerable)

The internet has been in an uproar over the past few days as a result of Google’s announcement of the POODLE vulnerability, which effectively breaks SSLv3 completely. We’ve scanned every single site that has passed verification with Tinfoil Security (that is, signed up and verified ownership) using our free testing tool, and sent emails to all those customers that have vulnerable sites. Our anonymized statistics of websites that have been tested show that 68%—that is, more than two-thirds—of the internet is vulnerable as I write this.

There has been a lot of misinformation flying around the internet over the past few days, and a lot of overly technical (or not technical enough) “how to” guides that are actually incorrect. This isn’t because the internet intends to mislead, but simply because cryptography of any sort, but especially TLS/SSL, is hard. We don’t claim to be experts, but we try really hard to be well-versed. With that in mind, this is our guide on how to protect your servers and what mistakes you’ve probably (unintentionally) made.

How Everyone Is Messing Up Cipher Disabling

The simplest thing to think is “OK, so it looks like if I disable any cipher suite that has CBC in the name, I’ll be fine.” This is wrong, and everyone is doing it, because it is what everyone is recommending.

There are a number of cipher suites, specifically AES128-SHA, AES256-SHA, and similar, that do not contain CBC in the OpenSSL cipher suite name. In reality, these cipher suites are named TLS_RSA_WITH_AES_128_CBC_SHA and TLS_RSA_WITH_AES_256_CBC_SHA, respectively. These cipher suites do work in CBC mode, no matter what OpenSSL chooses to call them!

We’ve disclosed this exact vulnerability to many companies at this point, including some large, extremely engineering-focused companies I guarantee you've heard of, all of whom have mitigated it now by disabling these last few ciphers. Given that we continue to see this exact mistake across most of the websites we’re testing, you should make sure you check it on yours too.

Your best option

Disable SSLv3 completely. Do not support it as a protocol at all, and reject any clients which require it. Unfortunately, this proves hard to do for many companies, as they have legacy clients whom they need to support that only support SSLv3 (specifically, IE6 on Windows XP and older Android clients, usually).

Our recommendation is simple: if, in any way, you can completely disable support for older clients that require SSLv3 and disable support for that protocol entirely, please do it.

Your next best option

If you must maintain support for SSLv3, your next best option is to enable the TLS_FALLBACK_SCSV cipher suite value. This cipher is a patch submitted to OpenSSL by Google (the same guys who found the exploit in the first place). This cipher solves the issue of retrying failed connections, thus preventing attackers from forcing browsers to use SSLv3. Additionally, it may also help prevent future exploits, as it prevents downgrading from TLS 1.2 to 1.1 or 1.0 as well.

Our recommendation: Make sure that you support the TLS_FALLBACK_SCSV cipher suite value. If you use OpenSSL 1.0.1, make sure you upgrade to 1.0.1j. Similarly, if you use 1.0.0, make sure to upgrade to 1.0.0o, and if you use OpenSSL 0.9.8, upgrade to 0.9.8zc.

Your last resort

If you cannot disable SSLv3, and if you cannot upgrade OpenSSL to support TLS_FALLBACK_SCSV for some reason, you have one additional option left: carefully disable all cipher suites for SSLv3 that run in CBC (cipher-block chaining) mode. Really, you should do this anyway, since TLS_FALLBACK_SCSV doesn’t actually resolve POODLE for anybody using SSLv3, it just prevents any newer clients from downgrading to SSLv3 and thus becoming vulnerable, limiting the number of clients that are affected.

This means that if you have to use SSLv3, your only real option left is to use some form of an RC4 cipher. This also isn’t perfect (there is a known attack on it by way of Royal Holloway, but it requires 224 connections), but is better than using any of these CBC ciphers which can be broken much faster.

If your configuration doesn't allow you to disable these CBC ciphers only for SSLv3 (for example, if you run nginx), you should consider using something like HAProxy to terminate SSL traffic in front of it and disabling / redirecting around it there. This, however, is hard and prone to error so you can go with the next recommended option: make sure your preferred cipher is set to RC4, rather than any CBC cipher.

Our recommendation: The following cipher suites that SSLv3 is able to use must be disabled in order to effectively prevent POODLE:

IDEA-CBC-SHA, EXP-DES-CBC-SHA, DES-CBC-SHA, DES-CBC3-SHA, EXP-DH-DSS-DES-CBC-SHA, DH-DSS-DES-CBC-SHA, DH-DSS-DES-CBC3-SHA, EXP-DH-RSA-DES-CBC-SHA, DH-RSA-DES-CBC-SHA, DH-RSA-DES-CBC3-SHA, EXP-DHE-DSS-DES-CBC-SHA, DHE-DSS-CBC-SHA, DHE-DSS-DES-CBC3-SHA, EXP-DHE-RSA-DES-CBC-SHA, DHE-RSA-DES-CBC-SHA, DHE-RSA-DES-CBC3-SHA, EXP-ADH-DES-CBC-SHA, ADH-DES-CBC-SHA, ADH-DES-CBC3-SHA, EXP-RC2-CBC-MD5, IDEA-CBC-SHA, EXP-DES-CBC-SHA, DES-CBC-SHA, DES-CBC3-SHA, EXP-DHE-DSS-DES-CBC-SHA, DHE-DSS-CBC-SHA, DHE-DSS-DES-CBC3-SHA, EXP-DHE-RSA-DES-CBC-SHA, DHE-RSA-DES-CBC-SHA, DHE-RSA-DES-CBC3-SHA, ADH-DES-CBC-SHA, ADH-DES-CBC3-SHA, AES128-SHA, AES256-SHA, DH-DSS-AES128-SHA, DH-DSS-AES256-SHA, DH-RSA-AES128-SHA, DH-RSA-AES256-SHA, DHE-DSS-AES128-SHA, DHE-DSS-AES256-SHA, DHE-RSA-AES128-SHA, DHE-RSA-AES256-SHA, ADH-AES128-SHA, ADH-AES256-SHA

Check, and check again.

We’ve been updating our tester constantly and improving its detection and results. We’ve added even more information regarding supported ciphers and the like. When you think you’ve fixed your server against POODLE, you should run another check.

We will continue to keep our eyes on this and attempt to update this post with more ways companies are making common mistakes in mitigating this attack, but if you follow the above directions, you should be OK. Scan yourself for even more vulnerabilities using the button below.

Feel free to chat with us or email us at any time.


Michael "Borski" Borohovski

Michael Borohovski is cofounder and CTO at Tinfoil Security. He got his start in security when he was just 13 years old, and has been programming for longer than he can remember. When he's not busy breaking software or building it, he also loves singing, juggling, and magic tricks. Yes, magic tricks.

Tags: browser security SSL poodle 0day


Securing Your Microservices via Bi-Directional TLS

In the beginning, there was the monolithic webapp. Then, as the application grew and became harder to scale, pieces began to split off into separate services. Microservices have become an increasingly popular architecture choice for separating concerns while speeding up development and deployment. However, security remains a critical but rarely talked about piece. Microservices significantly increase your attack surface as these services send messages back and forth across a network rather than just processes on one machine. Security for microservices, or any service/network oriented architecture, comprises two components: the Transport Layer and Application Layer.

Transport Layer

Hardening your transport layer is crucial, especially inside shared environments like AWS or Rackspace where you cannot be exactly sure where your network traffic is going or who might be listening in. Transport Layer Security (TLS), sometimes incorrectly still referred to as SSL (the predecessor to TLS), remains one of the cornerstones of encrypting and authenticating connections. Even if your services don’t communicate with HTTP(S) or RESTful APIs you can still wrap your network sockets with TLS.

It is usually prudent to secure all network traffic with TLS, despite the fact that engineers often seem to have misgivings about doing so. If you’re worried about performance loss from TLS, a load balancer can provide specialized hardware to efficiently terminate client TLS connections while also holding a persistent TLS connection open to the backend service. This persistent backend connection reduces the overhead of handshaking a new TLS connection with each request.
Terminating TLS with a load balancer(http://aws.amazon.com/blogs/aws/elastic-load-balancer-ssl-support-options/)

An often overlooked feature of TLS is authentication. While TLS can make guarantees about encrypting data as it moves across a network, it also provides a mechanism to enforce that the client and server don’t have a man-in-the-middle listening in. For public-facing services you must always rely on a public (paid) certificate authority. If you’re lucky enough to control both the server and every single client, you can roll your own certificate authority to sign your certificates.

During a typical TLS handshake the client and server exchange pleasantries and carefully begin setting up a secure tunnel. During this process the client should check that the certificate presented by the server is signed by a trusted authority (or chain of authorities). In addition, many TLS libraries allow the client to authenticate that the common name of the certificate matches the hostname it is trying to connect to. Both checks allow the client to assert that the server is actually who the client believes it is and communication is not being intercepted.

Application Layer

Besides transport security, services need to authenticate who is making the calls and ensure that they are authorized to do so. Conveniently TLS provides a mechanism to do this as well: not only can the client authenticate the server’s certificate is cryptographically valid, the server can similarly authenticate the client. During the handshake the server requests a certificate from the client, which it can provide. Mirroring the client, the server checks the validity of the certificate against a trusted certificate authority. However, the server can then extract out the details of the client from the certificate, such as the common name, and rather than checking against a hostname can use application-layer logic to authenticate the client is authenticated and authorized to do what they are trying to do. This bi-directional TLS authentication allows both sides of the connection to assert they are connecting with the other party they expect.

Bi-directional TLS is not often used, likely due to the pain points of creating and managing many certificates and the associated revocation lists. However, managing a set of allowed certificates is very similar to managing a set of allowed API keys. One way is to manage a specific set of revoked certificates, acting as a blacklist. However, if client certificates are treated as API keys you can instead manage allowed clients via a known whitelist. You gain cryptographic assurance that your client is who they say they are, while also ensuring your communication is encrypted.

Conclusion

Bi-directional TLS may not fit in every situation, but it’s a useful tool to have in one’s toolbox and may help leverage technology you’re already using. Tinfoil’s scanners are authenticated via bi-directional TLS, in addition to other network layer and application layer authentication methods. Just as you don’t want your application to have a single point of failure, you don’t want to depend on a single security method either.


Ben Sedat

Ben Sedat is the Engineering Wizard of Tinfoil Security. He's a bit of a blend between a traditional software engineer (builder) and security engineer (breaker). He spends a lot of time thinking about security: both detection as well as creating solutions for the security issues that exist in software and the internet. He also plays lots of video games. Lots.

Tags: SSL TLS


Stop Paying For SSL Certificates You Don't Need

Securing your stack with a SSL/TLS doesn't have to be costly or time consuming. There are two sides to TLS: Public Key Infrastructure (PKI) and Public Key/Asymmetric Cryptography. Hopefully your public network data is already encrypted with SSL/TLS (leveraging Public Key Cryptography), but you can also leverage a PKI to easily gain additional security wins for your internal network traffic.

SSL/TLS is most frequently used for encrypting data on the wire. You want to prevent the person next to you at Starbucks from being able to see your password as you submit a login form on the WiFi. It is used to verify the identity of a web server -- peer verification. You want to know that data going to and from www.tinfoilsecurity.com actually is coming from our servers, and not from someone else's. Early on in the TLS handshake your browser (and any underlying library like OpenSSL) does most of the heavy lifting for you: checking that the certificate's CommonName or SubjectAltNames matches the hostname and verifying the chain of trust. However, a tragically rarely used feature of SSL/TLS is that the client also has the opportunity to present a certificate during the handshake and the server can use it to verify the client's identity as well.

Why doesn't every browser present a certificate to every SSL website identifying the visitor as me? It certainly would make logins simple, but the rub comes with trust -- any certificate could identify the user as Ben Sedat. Public Key Infrastructure tries to solve this, and if both the website and client trust an authority to sign the certificates and keep track of them then you can establish a trusted link and make it a lot harder to forge fake things into the system.

Websites do this by requesting a certificate from a set of known Certificate Authorities. Browsers and operating systems keep lists of valid CAs and alert you when the certificate's chain of trust isn't valid. These CAs charge for their services, usually anywhere from $10 to $100 a year. Both you and your website's visitor trust the CA and everything works well, at least until it doesn't. What isn't widely publicized is that you can easily set up your own CA.

This CA isn't going to be trusted by your website visitors, but your infrastructure can trust it as long as you keep it safe. All too often we see companies and services disabling SSL entirely. They think that they need an expensive certificate, signed by a well-known CA. In the case of a website with clients that aren't under your control, that may be necessary. However, if you have more direct control over the client (like if you're your own API client) you can establish who you want to trust. Many libraries, like the Ruby hooks around OpenSSL and SSL connections allow you to specify which certificate authorities you want to trust in addition to the system defaults.

There isn't a perfect security system but you can easily add hurdles to delay and frustrate an attacker. SSL encryption of internet traffic is one such hurdle, and enforcing client/server identities on top creates one more hoop an attacker has to jump through to be successful. A client certificate isn't the be-all-end-all either -- it can be combined with usernames/passwords, two-factor authentication on another device, or all sorts of other methods to harden a system even more.

Go Forth and Make Your Own CA

OpenSSL is the de-facto standard command line utility for dealing with certificates. At Tinfoil we also really like XCA (http://sourceforge.net/projects/xca/); it's open-source and provides a (mostly) usable UI over the somewhat arcane OpenSSL invocations. Creating a CA and some certificates is easy in XCA, and once you're done you can have pieces of your application authenticating each other and keeping the communication secure.

  1. Creating a certificate authority
    First secure your XCA database (and private keys) with good password(s). Keep these safe, and don't put it anywhere digital! This is a critical layer of security for your CA.

  2. Generate your CA's private key
    Choose a nice long key here. There's some debate over key sizes regarding performance but it's worth it to go big here unless you're constrained by an embedded device. 1024-bit is considered too small, with 2048-bit being better currently but eventually breakable. 4096-bit may take a while to generate the key but it's worth it. It's also worth noting that RSA keys are more performant while encrypting, while DSA is faster for signing requests. ECDSA uses elliptic curve, which allows for smaller keys but is more computationally intensive.

    Creating a CA key

    The equivalent openssl incantations (for newer OpenSSL versions it may be easier to use the genpkey command):
    RSA: openssl genrsa -des3 -out blogCA-key.pem <bytes>
    DSA: openssl dsaparam -out blogCA-params.pem
             openssl gendsa -des3 -out blogCA-key.pem blogCA-params.pem
    ECDSA: openssl ecparam -out blogCA-params.pem
                 openssl ecparam -genkey -in blogCA-params.pem -out blogCA-key.pem
                 openssl ec -des3 -in blogCA-key.pem -out blogCA-key.pem

  3. Generate your CA's public certificate
    Choose the CA template and Apply All of the extensions. This flags your new certificate as a Certificate Authority, which some clients care about when evaluating the chain of trust. Also give it a commonName, essentially an identifier for your certificate, and choose an end date fairly far in the future.

    Creating a CA Certificate
    CA Certificate Options

    Via the command line:
    openssl req -new -x509 -days 730 -key blogCA-key.pem -out blogCA-cert.pem

  4. Generate a server certificate
    You can generate the private key, request, and certificate for a server application all within XCA. For bonus points, you can generate the private key and request on your server/clients directly, transfer just the request over, sign it with the CA and transfer the certificate back to the server. This prevents the private key from ever going on your network, although secure tunnels like SSH can make this process safer.

    Creating a server key
    Creating a server certificate request

    Create the private keys via CLI:
    RSA: openssl req  -newkey rsa:<bytes> -keyout server1-key.pem -out server1.csr
    DSA: openssl req -newkey dsa:blogCA-cert.pem -keyout server1-key.pem -out server1.csr
    ECDSA: openssl req -newkey ec:blogCA-cert.pem -keyout server1-key.pem -out server1.csr

    Signing the request via CLI:

    openssl x509 -CA blogCA-cert.pem -CAkey blogCA-key.pem -CAcreateserial \
    -days 365 -req -in server1.csr -out server1-cert.pem

    You can repeat this process for your servers, and/or your clients. Now you have all the credential pieces to have pieces of your infrastructure reveal themselves and verify each other during communication.

  5. Validating the trust
    Now it's time to use the certificates in your infrastructure. Standard peer verification from the client's perspective checks against the requested hostname, but in (cloud) infrastructure that could be a frequently changing target. One option is to use a load balancer behind a CNAME. Another option when you are your own client is to check the common or subject names in the provided certificate and validate against those and your certificate authority.

    There can be some gotchas in the minefield of secure SSL communication, especially as we're using the commonName as a service identifier rather than just matching the hostname. Some examples below perform peer verification from the client's perspective with that in mind. Feel free to send along any equivalent ones in other languages and I'd be happy to credit and include them.

    Sample ruby client code, with sockets:
    require 'socket'
    require 'openssl'
    
    context = OpenSSL::SSL::SSLContext.new
    context.verify_mode = OpenSSL::SSL::VERIFY_FAIL_IF_NO_PEER_CERT
    
    store = OpenSSL::X509::Store.new
    store.add_file 'blogCA-cert.pem'
    
    context.cert_store = store
    context.verify_callback = proc do |preverify, ssl_context|
      raise OpenSSL::SSL::SSLError.new unless preverify && ssl_context.error == 0
    end
    socket = TCPSocket.new('service.example.com', 8080)
    
    ssl_connection = OpenSSL::SSL::SSLSocket.new(socket, context)
    ssl_connection.connect
    ssl_connection.post_connection_check('server1') # Check the CommonName or AltSubjectNames
    # SSL Connection Ready for Use Now
    Sample ruby client with Net/Http:
    require 'net/http'
    require 'openssl'
    
    store = OpenSSL::X509::Store.new
    store.add_file 'blogCA-cert.pem'
    
    Net::HTTP.start('service.example.com', 8080, :use_ssl => true, :verify_mode  => OpenSSL::SSL::VERIFY_NONE) do |http|
        # Our own verification, as net/http uses hostname validation. Sadly, the flag for this (https://github.com/ruby/ruby/blob/trunk/lib/net/http.rb#L658) is unchecked
        raise 'Possible MITM' unless OpenSSL::SSL.verify_certificate_identity(http.peer_cert, 'server1')
        raise 'Bad trust chain' unless store.verify(http.peer_cert)
        # SSL Connection Ready for Use Now
    end

It's worth taking a moment to go over some basic CA security. Like a real CA, you need to keep your CA's passphrase and private key secret. I recommend taking a page from Bitcoin wallet private key protection and put the private key on some cold storage, like a VM or flash drive that you can keep entirely off the internet.

Expiration dates are the other gotcha of this type of security. Generating new keys and certificates should be easy now, so you can use a shorter certificate lifespan before you have to rotate them. If you use a calendaring solution it's worth it to put in a reminder a few weeks before the certificate expires so you remember to renew. Don't get caught unawares.

That's the basics of creating and using your own CA! For extra credit, you can post your CA's certificate revocation list someplace and check against that as well. Certificate Authorities are fairly easy to set up and provide extra hurdles of identification if someone has intruded on your network and wants to man-in-the-middle your network traffic.


Ben Sedat

Ben Sedat is the Engineering Wizard of Tinfoil Security. He's a bit of a blend between a traditional software engineer (builder) and security engineer (breaker). He spends a lot of time thinking about security: both detection as well as creating solutions for the security issues that exist in software and the internet. He also plays lots of video games. Lots.

Tags: security SSL guide


Tinfoil Security Blog

Tinfoil Security provides the simplest security solution. With Tinfoil Security, your site is routinely monitored and checked for vulnerabilities using a scanner that's constantly updated. Using the same techniques as malicious hackers, we systematically test all the access points, instantly notifying you when there's a threat and giving you step-by-step instructions, tailored to your software stack, to eliminate it. You have a lot to manage; let us manage your website's security.