Just Behave Already: Property Testing

What is property testing? In short, it can be described as a method of testing output of a program against the expected behavior, or properties, of a piece of code. Why should you care? The same reason we here at Tinfoil Security care: good testing goes beyond ensuring your code is functional. It can be crucial line of defense when it comes to the security of your applications, and property testing is a uniquely powerful tool in accomplishing these ends. But before we dive deeper, lets review more traditional testing.

it "finds the biggest element in the list" do
  assert 5 = biggest([5])
  assert 6 = biggest([6, 5])
  assert 100 =
    |> Enum.to_list()
    |> biggest()

The test above is fairly straightforward. It attempts to check that the biggest function will in fact return the biggest element of a provided list. It falls short in a few noticeable ways: What if the list is empty? What if it isn't sorted? What if there are duplicate integers?

Traditional testing very often focuses on specific examples and is dependent on the assumptions of the programmer. The function above may have been created with only positive integers in mind and it may not occur to the writer to test for cases involving negatives.

This example is a simple one, but it demonstrates a major drawback of traditional testing: it reveals the presence of expected bugs, rather than the absence of unexpected bugs. How would we pursue the latter? Enter property testing.

What is a Property?

Property testing is reliant on describing the properties of a function or piece of code. Very simply, these properties are general rules on how a program should behave. For the example function above, we might define the following:

  • "biggest returns the largest element of a list."

We might describe the properties of other well-known algorithms as such:

  • "sort returns a list with every element in ascending order."
  • "append returns a list with a length equal to the sum of the lengths of both lists passed to it."
  • "append returns a list with every element of of the first list, followed by every element of the second list."

Once we have defined the general properties of a program we can move beyond specific examples. From there we can use generators to test the output of our code against these properties using a variety of generated inputs.

How to Describe a Property?

This is easier said than done, however. Describing the properties of a program can be difficult, but there are a few general strategies, as described by Fred Herbert's excellent book on Property Testing in Erlang:

Modeling: Modeling involves reimplementing your algorithm with a simpler (though likely less efficient) one. Our biggest function for example could have it's output compared with an algorithm that uses sort to arrange a list in ascending order, then returns the final element. Sort is far less time-efficient, O(n log n) compared to the O(n) of our biggest function, but since it retains the same properties of biggest we can use it as a model to test our results against.

Equivalent Statements: Equivalent statements are used to reframe the property into a simpler one. For instance, we could say that the element returned by biggest is larger than or equal to any of the remaining elements of the input list. This simplified property is not quite the same but fundamentally equivalent to the one we had defined above.    

Symmetry: Some functions have natural inverses. The process of encrypting and decrypting data, for example, can be described by the following properties:

  • The input encrypted, then decrypted will return the original input.
  • The input decrypted, then encrypted will return the original input.

Oracles: Oracles involve using a reference implementation to compare your output against and, as such, are perhaps the best way to test the properties of your code. Oracles are most often used when porting existing code from language to another or when replacing a working implementation with an improved one.


Implementing property tests is not easy. It relies not only on describing the properties you wish to test against, but also on constructing generators to create the large, varied sets of randomized input to feed into your code. A single property may be tested hundreds of times, and generators will often create increasingly complicated inputs across these test iterations, or "generations" as they are called.

One can imagine that this randomly generated input could quickly become too unwieldy for the developer to make sense of. The failing case may contain large amounts of data irrelevant to what the actual cause of the failure. To help narrow things down to the true cause a property testing framework will often attempt to reduce, or "shrink", the failing case down to a minimal reproducible state. This usually involves shrinking integers down to zero, strings to "", and lists to [].

Fortunately, there are a variety of language-specific property testing libraries currently available. StreamData, for example, is an Elixir property testing library - and candidate to be included into Elixir proper - that provides built in generators for primitive data-types as well as tools to create custom ones. Generators can even be used to generate symbolic function calls, allowing the possibility to fuzz and test transitions on a state machine.


As a final note, it should be mentioned that while property testing is a powerful tool, it is not a perfect solution. Describing the properties of a piece of code can be difficult, as can coming up with tests for those properties. Furthermore, these tests are reliant on well-made generators to come up with the varied and unexpected input, which in itself can be a difficult and time consuming task.

It is also important to note that more traditional testing should not be entirely eschewed for property tests. The real strength of property testing is in using generated input to automate all the tedious work of thinking up unusual edge cases and writing individual tests for them, and it is at its best when used with unit tests that check for unique edge cases or document unusual behavior.

At Tinfoil Security, we understand that thorough and effective testing is an essential part of creating of efficient and secure technology. If you have any questions or would like to let us know how property testing has helped in your projects, feel free to reach out at contact@tinfoilsecurity.com.

Peter Ludlum

Peter is a Software Engineering Intern at Tinfoil Security. A recent graduate of App Academy, he enjoys nothing more than bringing beautiful (and functional) web pages to life. When he isn't coding, Peter is usually lost in a book or strumming out a new tune on the ukulele.

Tags: security guide DevSec DevOps

What the hack happened in 2017?

Cybersecurity got a lot of public attention this year mostly thanks to some very notable breaches that impacted not just individuals and corporations, but also governments (subsequently affecting infrastructure and defense)!

In 2017, we saw an array of ransomware, malware, data breaches and leaks galore. Some of these you will have no doubt heard about, but there were many that flew under the mainstream radar. Here is a list highlighting some of the most notable cybersecurity attacks from this year, in our “12 days of hacking, what my blackhat gave to me” … come on, tis’ the season for holiday puns!

1) Equifax: the credit reporting agency Equifax had a major data breach that allowed hackers to obtain upwards of 145 million of their customers’ personal data, including social security numbers, dates of birth and home addresses.

2) WannaCry: a ransomware cryptoworm encrypted data on approximately 230,000 computers running on Microsoft Windows in over 150 countries, demanding ransom payments in the Bitcoin cryptocurrency. The US and UK governments have stated that North Korea was to blame for these attacks.

3) US National Security Agency (NSA): a virtual disk image owned by a division of the NSA, containing over 100 gigabytes of data from an Army intelligence project, codenamed "Red Disk," was leaked revealing a surveillance program pointed at domestic interests named “Ragtime” that targeted Americans.

4) US Central Intelligence Agency (CIA): WikiLeaks gained access to sensitive “classified” and “top secret” documents, detailing the agency’s intentions to hack into iPhones and Android devices, as well as smart TVs.

5) Cloudbleed: Tavis Ormandy from Google’s Project Zero discovered and reported a security problem with Cloudflare’s edge servers. He was seeing corrupted web pages being returned by some HTTP requests run through Cloudflare, resulting in customer data leaks from Uber, 1Password, Tinder, and online dating site OKCupid, among others.

6) Sabre Systems: a reservation software company, had over 32,0000 of their hotels and other lodging establishments hacked for access to their payment and customer data tied to bookings processed through a reservations system. Other major companies that had data stolen by the breach include Google, Hard Rock Hotels, and Loews.

7) School’s Out: A hacker identified as Rasputin breached and leaked data from several universities and government agencies, including NYU, Oxford, Cambridge, and the US National Oceanic and Atmospheric Administration using SQL injection.

8) NotPetya/Petya/Nyetya/Goldeneye: A computer virus referred to as NotPetya targeted Ukrainian businesses using compromised tax software affecting major global businesses, such as FedEx, but hitting Ukrainian infrastructure particularly hard. It disrupted utilities like power companies, airports, public transit, and the central bank; just the latest in a series of cyber assaults against the country.

9) Bad Rabbit: Most of us know pop-ups should not be trusted, but a major ransomware campaign, referred to as Bad Rabbit, managed to sneak into computers through an Adobe Flash installer. It was placed on compromised news and media websites. This ransomware mostly affected Russia, but experts saw also saw cases in Ukraine, Turkey and Germany.

10) Virgin Airlines: After Alaska Airlines acquired Virgin America, an unknown hacker broke into Virgin America's corporate network compromising 3,120 employees and contractors logins, while 110 additional employees may have had personal information stolen.

11) Uber: It came to light this year that hackers infiltrated Uber and were able to steal data from 58 million of Uber’s customers. Uber responded by forking out $10 million to pay off the hackers and try to cover it all up.

12) GOP Voters: Almost 200 million voter records were exposed online after a GOP data firm misconfigured a security setting in its Amazon cloud storage service. This became an alarming pattern for breaches stemming from insecure Amazon servers where data is stored. Other companies whose data was also exposed and leaked on Amazon servers include Verizon, Accenture, Viacom, and the U.S. Department of Defense.

Tinfoil Security genuinely believes there are more good folks out there vs. bad, so we have hope and look forward to bringing joy to the world of cybersecurity in 2018 and beyond!

Neda Blocho

With a background in running the world's top accelerator program out of Stanford University and a tour as a seed stage investor in Silicon Valley, Neda has seen first hand the great need for solving issues around cyber security! Neda makes sure the world knows how much better and safer their DevOps lives can be by partnering with Tinfoil.

Strutshock: Apache Struts 2 (RCE CVE-2017-5638) in Plain English

If you’ve paid attention to the news recently, you’ve certainly heard about the massive breach at Equifax, now estimated to have put 145 million of people’s sensitive personal data out into the world. You’ve probably also heard it was due to unpatched software. So what was the issue, why does it matter, and what can you do? Let’s start with the basics.

In vulnerable versions of Struts 2, there is a bug in how errors are handled when the server receives a request containing a Content-Type, Content-Length, or Content-Disposition header. With an unexpected or malformed request, you would expect the server to send back a 400 error, or otherwise ignore the request. However, with a vulnerable version of struts2, the Jakarta Multipart Parser will parse that header string and, if it contains executable OGNL code, execute it. What, then, is OGNL and why would it matter if it parses and executes some code? OGNL, or Object Graph Navigation Language, is a Java library which, on the development side, provides convenient tools for getting and setting values on Java objects. For instance, connecting the forms on your web page to the struts representation of that data on the server. Being able to inject commands through OGNL means an attacker can then substitute values on the server such that it will run whatever malicious code the attacker wishes to send.

Short version: malicious Content-Type/Length/etc. header -> Jakarta Multipart parser error -> OGNL receives the header as executable code -> server now runs malicious code as if it were part of the original program. So if we have a site we think is vulnerable, we could set up a script that will send one of these headers that spawns a bash shell and takes a command from the user. Have a look at the video for a quick demo.

As you can see, this gives us quite a bit of power over the system. We can traverse through directories, create files, view environment variables (which in turn gives us access to the database), and even cause the site to crash by deleting necessary files.

The Danger

If a server is running a vulnerable version of Struts 2, an attacker who knows how to exploit this might as well have your server running locally on their computer. They can access the source code of your site. If your site has access to a database, so do they. If they want to tell your server to sleep, putting you out of business for a while, they can. Even if your server has limited privileges, or the site runs on a virtual machine, the attacker might be able to learn something about where your sensitive data actually lives that could be leveraged to launch an attack where it will cause the most harm.

The Solution

You probably hear it all the time, but keep your software up to date. Test your production code and make sure you’re able to roll out the most up to date versions of the libraries and technologies you rely upon as quickly as you can once they are available. The fix to this vulnerability is available in Struts 2 versions or 2.3.32 and later, both of which have been available since March. For note, the Equifax breach happened at the end of July, well after the tools were available to fix the vulnerability in question. Applying the patch is simpler than trying to roll your own header validation and far quicker.

If you don’t know if you’re vulnerable, or if you think you’ve set up an effective solution, you can check using our free service here. If you’re already using our web scanner product, you’re already being checked for the Strutshock vulnerability in your scans, along with many other common vulnerabilities. Thanks for reading, and stay secure.

Alex Bullen

Alex is Tinfoil Security's Top-Shelf Programmer (and fetcher of things from high shelves). A former psychology wonk and recent App Academy grad, Alex endeavors to treat every challenge as an opportunity to improve his code-fu. When not busily building blocks of precisely put code, you can find him reading fantasy novels or practicing kung fu.

Tags: security vulnerabilities

Creating Testable, Modularized Chrome Extensions

When creating software, it’s very important to be able to verify the correctness of your code, and writing a Chrome extension is no exception. For an automated testing suite to work, we require two things. First, the code must be modularized in a way that allows for simple unit tests to be written. If this isn’t the case, writing tests can become a very tedious and messy process. Second, the testing suite needs to be able to access and interact with the application as a whole (integration tests). At first glance, it appears Google Chrome’s extensions satisfy neither of these requirements. However, with a bit of creativity, we can find solutions to both.

How Chrome Makes Testing Hard

We’ll begin by looking at the first requirement: The code must be modularized in a way that allows for simple unit tests to be written. Chrome extensions are broken into three parts. The popup (the window that shows when you click the extension icon), the background script (any extension logic that runs in the background), and the content script (any extension logic that runs as a part of the current web page). Each of these must be exactly one file.

This means if your extension has a lot of complex functionality, you end up with a massive background script and popup script. I took a look at the Google Keep extension and found their popup script is 8,500 lines, and their background script is 26,000 lines! Trying to find a bug in a 26,000-line Javascript file sounds like a great way to ruin your Monday. As expected, this also makes unit tests challenging.

We’ll discuss a few solutions soon, including how Google likely manages their 26,000-line background script, but for now let’s move on to the second requirement: The testing suite needs to be able to access and interact with the application as a whole. The good news is that Google provides ChromeDriver, a way of programmatically interacting with Chrome. The bad news is that they seem to have forgotten about programmatically interacting with extensions.

As of today, there isn’t a way to click on your extension’s icon and open the popup using ChromeDriver. The workaround is to navigate to http://(your extension id)/popup.html, and view your popup there. This workaround is flawed, however. It renders the popup in a differently sized window than your normal popup, which makes asserting on UI behavior challenging or impossible depending on what you want to do. Also, if you want to assert that interacting with your popup does something to a page that’s currently open, you’re out of luck, since you need to navigate your browser to the popup to be able to interact with it.

How We Can Test Chrome Extensions

Creating a Chrome extension that’s testable is possible, but we need to get a bit creative. To begin, we need a way to modularize our program into bite-sized chunks that each have a specific purpose. Then, we need to figure out a solution to our integration test issues, including setting up some sort of headless browser that can run tests. Finally, we need to actually write tests.


There are two different solutions to modularization. The first is to use a background HTML page instead of a background script. This HTML page’s role is to simply “require” a bunch of different Javascript files in order of dependency using <script> tags. For example, this is what Microsoft does in their OneNote extension. This configuration allows you to modularize your scripts in separate files and then unit test each individually.


The other option is to use some sort of build system. By using a build system (such as Gulp), you can develop the extension in any form you’d like, then allow the build system to concatenate files together, thus creating Chrome’s required single script at build-time. By using a task runner like Gulp, you could have a setup where Gulp runs unit tests on your modules, then minifies and compiles everything into Chrome’s required format, and finally runs integration tests on the bundled package. This solution requires Node as a development dependency, but it offers more flexibility than the background.html option.

Integration Tests

Writing functional integration tests is a little harder. Assuming we want to use some sort of CI system, we’ll probably want our integration tests running in a headless browser, so we should first focus on trying to get a headless browser running with our Chrome extension.

Selenium is a popular choice for browser automation, and has wrappers in just about every major language. It also works with ChromeDriver, which is great news, since we can’t load our Chrome extension into any other kind of browser. Depending on your wrapper, you’ll either be given a method to load in an extension or you’ll have to use the driver flag --load-extension=extension_path. If you’re running the browser in headless mode, it may also be required to pass --no-sandbox and --disable-impl-side-painting.

The extension path you give the driver should be the path to the folder containing your manifest.json, and it should not be zipped. If either of these aren’t true, Chrome crashes. Once you’ve gotten Selenium to navigate to your extension popup at http://(your extension id)/popup.html, you’re ready to put together some tests.

Actually Writing Tests

At this point, writing tests is fairly simple. We can use Karma and Jasmine to test our Javascript modules in the same way that we could normally use Karma and Jasmine. For integration tests, I chose to use RSpec paired with Watir, a Ruby Selenium wrapper, but if you want to keep the entire testing suite running in Node, WebDriver.io + Jasmine is a great option. We can use these frameworks like we normally would, now that our project is setup correctly.

A Working Example

I’ve put together a working, testable example implementing everything mentioned in this blog post. It can be found on GitHub here. This can be used either as a proof of concept or as a base for anyone writing a Chrome extension. The extension has a super simple background module that we test with a unit test, and some text on a popup that we test using an integration test. Unit tests are done using Karma and Jasmine. Integration tests are done using Watir and RSpec. Hopefully this example helps people write better, more testable Chrome extensions in the future.

Questions? Have a better method for testing Chrome extensions? We’d love to hear from you. Feel free to shoot us an email at contact@tinfoilsecurity.com with any awesome browser extensions you’re building and how you’ve gone about testing them.

Nathaniel Woodthorpe

Nathaniel is a Software Engineering Intern at Tinfoil. He's currently studying at the University of Waterloo and working on finishing his degree in Computer Science. He got into programming writing bots for online games back in elementary school. When he isn't coding, he's usually outside climbing something.

Announcing Tinfoil Security's Login Recorder

Ensuring our scanner can successfully authenticate into any given site has always been a priority at Tinfoil Security, and has helped us stay ahead. After all, if authentication fails, the scanner may not be able to access huge parts of a site, and can miss many vulnerabilities. Today, we’re announcing the Tinfoil Security Login Recorder: the easiest way for our scanner to authenticate into your site.

The Login Recorder is a browser extension that allows you to teach our scanner how to authenticate into your site by recording your login sequence in your own browser. Simply install our Chrome extension and follow the on-screen instructions to submit your recording.

Begin by providing API keys found here. If you’ve never created Tinfoil API keys before, the first step in the extension has a link to an interactive tutorial that will show you how to make one. Ensure your keys are “Full Access”, otherwise the extension won’t be able to send data to our server.

After you’ve authenticated, you’ll be asked to select which site you want to provide a recording for. Choose the site from the menu then hit ‘Next’. You’ll be warned about which sites have existing recordings, and submitting a recording for a site that already has one will overwrite the existing recording.

When a site has been selected, you’re ready to record! Make sure you’re on the login page for the correct site, then hit the ‘Record’ button.

At this point, the extension icon will turn red to indicate we are recording. Authenticate with your site as you normally would, then open the extension popup and click ‘Finished’ to end your recording.

Next, you’ll be asked to select some ‘verifiers’ on the page after you’ve logged in by clicking on them in your browser. A verifier is an element on the current page that is not present before authentication. For example, if you authenticate with your site and are greeted with a “Hello, Nathaniel!” banner, this would make a perfect verifier. We use these verifiers to help figure out if our scanner successfully logged in to your site, or if something went wrong in the process. We recommend selecting at least 3 verifiers.

When this is done, all that’s left to do is submit! Once you hit the ‘Submit’ button, your recording will be securely sent to our servers, and your next Tinfoil scan will mimic your recorded login to authenticate into your site.

This new method of authentication will work on many different types of login flows. Whether your site uses a traditional style login page, or a complex multi-page authentication flow built to your site’s needs, the Login Recorder provides a simple way to get your account setup quickly and easily.

Note that Tinfoil Security is dedicated to handling your sensitive information privately and securely. After submitting, your recording and API keys are removed from your system. On Tinfoil’s servers, your recording is encrypted using the same security practices as the rest of Tinfoil Security’s sensitive information. See https://www.tinfoilsecurity.com/security for more information.

Visit the Chrome Web Store to try the extension today!

Nathaniel Woodthorpe

Nathaniel is a Software Engineering Intern at Tinfoil. He's currently studying at the University of Waterloo and working on finishing his degree in Computer Science. He got into programming writing bots for online games back in elementary school. When he isn't coding, he's usually outside climbing something.