Stryker

Stryker

Measure the effectiveness of JavaScript tests.

Stryker: Live at TechDays 2017

What is mutation testing?

Bugs, or mutants, are automatically inserted into your production code. Your tests are run for each mutant. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. The higher the percentage of mutants killed, the more effective your tests are.

It's really that simple.

But wait, what about code coverage?

Well... code coverage doesn't really tell you anything about the effectiveness of your tests. Think about it, when was the last time you saw a test without an assertion, purely to increase the code coverage.

Imagine a sandwich covered with paste. Code coverage would tell you the bread is 80% covered with paste. Mutation testing, on the other hand, would tell you it is actually chocolate paste and not... well... something else.

Meet: Stryker

With Stryker, mutation testing finally arrives at the JavaScript world. It takes the best practices of existing mutation testing frameworks like PIT and Humbug, but with the open mentality of nodejs and the web. This is why Stryker is not a test runner or test framework. Indeed, why would it? There are already awesome test runners and test frameworks out there. Instead, Stryker empowers you to use what you like and focusses on being the best Mutation testing framework it can be.

An example

Say you're building an online casino, users can only enter the casino when they're over 18. So you write the following piece of code to check if someone is allowed to use the website:

function isUserOldEnough(user) {
  return user.age >= 18;
}

Stryker will find the return statement and decide to change it in a number of ways:

/* 1 */ return user.age > 18;
/* 2 */ return user.age < 18;
/* 3 */ return false;
/* 4 */ return true;

We call those modifications mutants. After the mutants have been found, they are applied one by one and your tests will be executed against them. If at least one of your tests fail, we say the mutant is killed. That's what we want! If no tests fail, it survived. The better your tests, the less mutants survive.

Output

Stryker will output the results in various different formats. One of the easiest to read is the clear text reporter which output. You can activate it by adding 'clear-text' to the reporter list in stryker.conf.js.


  Mutant killed!
  /yourPath/yourFile.js: line 10:27
  Mutator: BinaryOperator
  -                 return user.age >= 18;
  +                 return user.age > 18;

  Mutant survived!
  /yourPath/yourFile.js: line 10:27
  Mutator: RemoveConditionals
  -                 return user.age >= 18;
  +                 return true;


The clear text reporter will output how exactly your code was modified and which mutator was used. It will then tell us if a mutant was killed, meaning that at least one test failed, or if it survived. The second mutation in this example is marked as a survivor. This means there is probably a test missing that explicitly tests for an age lower than 18.

Features

Mutations

Control more than 30 supported mutations, or write your own.

Speed

Use code analysis and parallel test runner processes to speed things up.

Test runner agnostic

Run tests with Karma, Mocha, or write a simple plugin for any test runner you like.

Open source

Free as in Speech. Maintained by the open source community at GitHub.

Extendable

Create your own plugins to mutate, report or run tests using the Stryker API.

Clever reports

Use clever reports to sniff out surviving mutants and improve test effectiveness.

For someone who hates mutants... you certainly keep some strange company.
Professor X
Oh, they serve their purpose... as long as they can be controlled.
William Stryker

Free swag

Contribute and get a free t-shirt or sticker. As long as stock lasts.

Stryker T-shirt