Chutzpah

by Matthew Manela on Chutzpah

Chutzpah 2.4 – Code Coverage is a go!

With the Chutzpah 2.4 release one of the most asked for features is now available: built-in code coverage. In addition this release contains a bunch of bug fixes and added configurability.

You can get the new bits from CodePlex, NuGet or go to the Visual Studio Gallery to get the updated Visual Studio Context Menu Extension  and Unit Test Explorer adapter for Visual Studio 2012.

Changes and Fixes

  • Added Code Coverage support (item 113)
  • Updated versions of QUnit(v1.11.0), TypeScript(v0.8.3) and CoffeeScript(v1.6.2)
  • jasmin-jQuery toContain tests fail with reason “Passed.” (item 95)
  • FileSearchLimit=300 cancels test run with more than 300 tests (item 89)
  • /failOnScriptError exits with 1 even when there are not errors (item 97)
  • ‘undefined’ is not a function after updating to 2.3 (item 96)
  • No tests found when using QUnit.test instead of test (item 94)
  • Allow including files without building them (item 93)
  • Improve “/openInBrowser” (item 91)
  • Add TestFileTimeout setting to chutzpah.json (item 110)
  • Need a later version of phantom bundled with Chutzpah (item 107)
  • QUnit test runner module gotcha in 2.3 (item 109)
  • Support for unicode (item 111)
  • Improve CoffeeScript parse error reporting (item 80)

 

Code Coverage

Chutzpah’s new code coverage features is built on top of the great open source library Blanket.js. When you run with code coverage enabled it will use Blanket.js to instrument the files under test and gather the coverage results. You can perform code coverage from the command line and both Visual Studio plugins.

Command Line

To get code coverage from the command line you pass the /coverage flag to chutzpah.console.exe.

chutzpah.console.exe /coverage /path test.js

After this command finishes you will have a two files placed in your current directory

_Chutzpah.coverage.html – The rendered html results
_Chutzpah.coverage.json – The raw json results

When you open the results html file you will see something like:

image

 

Visual Studio Context Menu Extension

The Visual Studio Context Menu Extension now contains a Show Code Coverage command which will perform code coverage and open the results in a browser.

image

Visual Studio 2012 Test Adapter Extension

For the Unit Test Explorer adapter for Visual Studio 2012 Chutzpah piggybacks on the existing “Analyze Code Coverage” menu option and uses it to run with code coverage and show the results in a browser. Ideally, Chutzpah would integrate into the code coverage reporting built into VS but I was unable to figure out a good way to plug in to that. The side effect is that when you run this command VS will open their results window first (with nothing in it) and then Chutzpah will launch its results in a browser.

image

NOTE: I have noticed some performance issues when running “Analyze Code Coverage for All Tests” command. I think it might be VS doing extra work for no reason since it is trying to generate its own report.

 

Configuration Code Coverage

Being able to run code coverage is not enough since in many cases the results will not be accurate. For example, if your project uses jQuery you do not expect your test to cover all the lines of jQuery’s source code (that would be one impressive test). Therefore you need to exclude files/directories from participating in code coverage. There are two way you can do that with Chutpah: command line arguments or a chutzpah.json file.

If you are running Chutzpah from the command line there are two additional arguments you can pass coverageIncludes and  coverageExcludes. These are a comma separated list of files/paths you want to include or exclude from coverage. They can use a glob format where * matches zero or more characters and ? matches one character. For example:

Exclude files that ends in matt.js

chutzpah.console.exe /path test.js /coverage /coverageExcludes *matt.js

 

Include only dog.js and any file with a single character extension

chutzpah.console.exe /path test.js /coverage /coverageIncludes "*\dog.js, *.?"

 

Include *query.js but exclude jQuery.js

chutzpah.console.exe /path test.js /coverage /coverageIncludes *query.js /coverageExcludes *\jquery.js

 

The other option for configuring code coverage is the chutzpah.json file. There are two new setting which you can use CodeCoverageExcludes and CodeCoverageIncludes. These settings work in the same way as the their command line counterpart. The main difference is that you must escape backslashes.

{
  "CodeCoverageExcludes": ["*\\jquery.js"],
  "CodeCoverageIncludes": ["*query.js", "*\\dog.js"]
}

Test timeout setting

Chutzpah already has a test timeout setting but you previously could only configure it from the command line or with a setting in VS. This was inflexible especially if you want a different timeout for different tests. To solve this Chutzpah now include a TestFileTimeout setting in the chutzpah.json file. For example if you want a 10 second time out for a certain directory just place a chutpah.json file with the following in it:

{
    "TestFileTimeout": "10000"
}

Chutzpah Specific Reference

With the addition of TypeScript support a few releases ago several people reported issues when using Chutzpah with the TypeScript plugin in VS. The problem occurs since both TypeScript and Chutzpah are using the <reference path…> comments for different purposes. To get around this issue Chutzpah now supports two ways to reference dependent files.

1. The existing general way

/// <reference path="foo.js" />

2. The new Chutzpah specific way

/// <chutzpah_reference path="foo.js" />

Using the latter will let Chutzpah know about your dependency graph and should not confuse the TypeScript compiler.

Excluding References

The reference syntax is also used by VS to give intellisense in JS files. This leads to an issue since sometimes you may reference a version of a file just for intellisense but you do not want Chutzpah to copy it. In this case you now can include the chutzpah-exclude attribute to have it ignored by Chutzpah:

/// <reference path="someDependency.js" chutzpah-exclude="true" />

Running against a remote url

Chutzpah now supports executing a remote test harness. If you have a html test file on a remote server (like http://somesite.com/test.html) you can tell Chutzpah to execute that file directly

chutzpah.console.exe http://somesite.com/test.html

Note: Chutzpah does not support running anything but .html files remotely. If you want to run Chutzpah directly on a .js, .ts or .coffee file they must be local on disk.