Chutzpah 2.4 – Code Coverage is a go!

With the Chutzpah 2.4 release one of the most asked for features is now available: built-in code coverage. In addition this release contains a bunch of bug fixes and added configurability.

You can get the new bits from CodePlex, NuGet or go to the Visual Studio Gallery to get the updated Visual Studio Context Menu Extension  and Unit Test Explorer adapter for Visual Studio 2012.

Changes and Fixes

  • Added Code Coverage support (item 113)
  • Updated versions of QUnit(v1.11.0), TypeScript(v0.8.3) and CoffeeScript(v1.6.2)
  • jasmin-jQuery toContain tests fail with reason “Passed.” (item 95)
  • FileSearchLimit=300 cancels test run with more than 300 tests (item 89)
  • /failOnScriptError exits with 1 even when there are not errors (item 97)
  • ‘undefined’ is not a function after updating to 2.3 (item 96)
  • No tests found when using QUnit.test instead of test (item 94)
  • Allow including files without building them (item 93)
  • Improve “/openInBrowser” (item 91)
  • Add TestFileTimeout setting to chutzpah.json (item 110)
  • Need a later version of phantom bundled with Chutzpah (item 107)
  • QUnit test runner module gotcha in 2.3 (item 109)
  • Support for unicode (item 111)
  • Improve CoffeeScript parse error reporting (item 80)

 

Code Coverage

Chutzpah’s new code coverage features is built on top of the great open source library Blanket.js. When you run with code coverage enabled it will use Blanket.js to instrument the files under test and gather the coverage results. You can perform code coverage from the command line and both Visual Studio plugins.

Command Line

To get code coverage from the command line you pass the /coverage flag to chutzpah.console.exe.

chutzpah.console.exe /coverage /path test.js

After this command finishes you will have a two files placed in your current directory

_Chutzpah.coverage.html – The rendered html results
_Chutzpah.coverage.json – The raw json results

When you open the results html file you will see something like:

image

 

Visual Studio Context Menu Extension

The Visual Studio Context Menu Extension now contains a Show Code Coverage command which will perform code coverage and open the results in a browser.

image

Visual Studio 2012 Test Adapter Extension

For the Unit Test Explorer adapter for Visual Studio 2012 Chutzpah piggybacks on the existing “Analyze Code Coverage” menu option and uses it to run with code coverage and show the results in a browser. Ideally, Chutzpah would integrate into the code coverage reporting built into VS but I was unable to figure out a good way to plug in to that. The side effect is that when you run this command VS will open their results window first (with nothing in it) and then Chutzpah will launch its results in a browser.

image

NOTE: I have noticed some performance issues when running “Analyze Code Coverage for All Tests” command. I think it might be VS doing extra work for no reason since it is trying to generate its own report.

 

Configuration Code Coverage

Being able to run code coverage is not enough since in many cases the results will not be accurate. For example, if your project uses jQuery you do not expect your test to cover all the lines of jQuery’s source code (that would be one impressive test). Therefore you need to exclude files/directories from participating in code coverage. There are two way you can do that with Chutpah: command line arguments or a chutzpah.json file.

If you are running Chutzpah from the command line there are two additional arguments you can pass coverageIncludes and  coverageExcludes. These are a comma separated list of files/paths you want to include or exclude from coverage. They can use a glob format where * matches zero or more characters and ? matches one character. For example:

Exclude files that ends in matt.js

chutzpah.console.exe /path test.js /coverage /coverageExcludes *matt.js

 

Include only dog.js and any file with a single character extension

chutzpah.console.exe /path test.js /coverage /coverageIncludes "*\dog.js, *.?"

 

Include *query.js but exclude jQuery.js

chutzpah.console.exe /path test.js /coverage /coverageIncludes *query.js /coverageExcludes *\jquery.js

 

The other option for configuring code coverage is the chutzpah.json file. There are two new setting which you can use CodeCoverageExcludes and CodeCoverageIncludes. These settings work in the same way as the their command line counterpart. The main difference is that you must escape backslashes.

{
  "CodeCoverageExcludes": ["*\\jquery.js"],
  "CodeCoverageIncludes": ["*query.js", "*\\dog.js"]
}

Test timeout setting

Chutzpah already has a test timeout setting but you previously could only configure it from the command line or with a setting in VS. This was inflexible especially if you want a different timeout for different tests. To solve this Chutzpah now include a TestFileTimeout setting in the chutzpah.json file. For example if you want a 10 second time out for a certain directory just place a chutpah.json file with the following in it:

{
    "TestFileTimeout": "10000"
}

Chutzpah Specific Reference

With the addition of TypeScript support a few releases ago several people reported issues when using Chutzpah with the TypeScript plugin in VS. The problem occurs since both TypeScript and Chutzpah are using the <reference path…> comments for different purposes. To get around this issue Chutzpah now supports two ways to reference dependent files.

1. The existing general way

/// <reference path="foo.js" />

2. The new Chutzpah specific way

/// <chutzpah_reference path="foo.js" />

Using the latter will let Chutzpah know about your dependency graph and should not confuse the TypeScript compiler.

Excluding References

The reference syntax is also used by VS to give intellisense in JS files. This leads to an issue since sometimes you may reference a version of a file just for intellisense but you do not want Chutzpah to copy it. In this case you now can include the chutzpah-exclude attribute to have it ignored by Chutzpah:

/// <reference path="someDependency.js" chutzpah-exclude="true" />

Running against a remote url

Chutzpah now supports executing a remote test harness. If you have a html test file on a remote server (like http://somesite.com/test.html) you can tell Chutzpah to execute that file directly

chutzpah.console.exe http://somesite.com/test.html

Note: Chutzpah does not support running anything but .html files remotely. If you want to run Chutzpah directly on a .js, .ts or .coffee file they must be local on disk.

Posted in Chutzpah | 5 Comments

Using SignalR in an ARR cluster

My team ran into an issue when deploying an application which uses SignalR on a clustered environment using Application Request Routing (ARR). We created a simple three server setup with one server as an ARR node and 2 servers running the application. When accessing the site we saw some odd behavior.

1. All SignalR requests fail to use the more efficient transports (ServerSentEvents/ForeverFrame) and fall back to using Long Polling

2. Some messages get replayed multiple times in IE9

After looking into the issue for a while we figured out it was caused by a setting in ARR called Response Buffer Threshold.

Response buffer threshold (KB)

Lists the size of the buffer, in kilobytes, that ARR uses to retain a response before sending it to the client. Content that is smaller in size than the threshold can be cached.

By default this is set to 256kb which means that it will buffer responses until they reach that amount. By setting this to 0 ARR will no longer buffer and SignlR will function correctly.

clip_image002

Posted in ARR, SignalR | 4 Comments

Chutzpah 2.3 Released

This release contains a couple important new features and bug fixes. Chutzpah now gives you more control in how you want youe tests run using the new chutzpah.json settings file (described in detail below).

You can get the new bits from CodePlex, NuGet or go to the Visual Studio Gallery to get the updated Visual Studio Context Menu Extension  and Unit Test Explorer adapter for Visual Studio 2012.

Changes and Fixes

  • Added Chutzpah.json settings file support
  • Consistent conversion error (item 70)
  • System.IO.IOException: The handle is invalid when running from inside WebStorm (item 77)
  • Exception has been thrown by the target of an invocation. inside Visual Studio 2012 (item 65)
  • Tests are hanging (ChutzpahTimeoutException) (item 72)
  • Reference root path should be project (item 83)
  • Smarter coffeescript compilation (item 75)
  • Running with parallelism gives faulty timing (item 81)
  • Running with parallelism gives System.IO.IOException (item 74)
  • Compile all TypeScript files at the same time (item 82)

The Chutzpah Settings File

The Chutzpah settings file is an .json file that you can place anywhere in your testing directories to inform Chutzpah of test specific settings. The Chuzpah settings file is completely optional and if the file or any of its properties are omitted Chutzpah will use the default values.

When Chutzpah is run on a file (lets say test.js) it scans the current directory for a file named chutzpah.json. If it does not find one it will recursively traverse up the directory chain looking for one. This behavior provides a lot of flexibility since you can provide multiple chutzpah.json files in your tree and each one will work over its sub-folder. This lets you have different settings for different groups of tests.

Chutzpah.json Format

The settings file is a text file with json content of the following format:

{
    "Framework": "qunit|jasmine",

    "TestHarnessLocationMode": "TestFileAdjacent|SettingsFileAdjacent|Custom",

    "TestHarnessDirectory": "<Path to a folder>",

    "TypeScriptCodeGenTarget" : "ES3|ES5",

    "RootReferencePathMode":"DriveRoot|SettingsFileDirectory"
}

The chutzpah.json file supports the following properties

Framework – Determines what testing framework to use. This will override the other detection methods. This is helpful if Chutzpah is having trouble figuring out what your intended framework is. If left out Chutzpah will automatically try to detect the framework.

TestHarnessLocationMode – Determines where to place the generated html test harness files. The default mode is TestFileAdjacent which means the harness is placed in the same directory as the test file. SettingsFileAdjacent means it is placed in the same directory as the chutzpah.json file. Custom mode lets you specify a custom path to the directory.

TestHarnessDirectory – When TestHarnessLocationMode is set to Custom this is either the relative or absolute path to the directory where to place the test harness.

TypeScriptCodeGenTarget – When running on a TypeScript file this property will tell the TypeScript compiler what type of code to generate. ES3 is for ECMAScript 3 and ES5 is for ECMAScript 5.

RootReferencePathMode – This property determines what directory a rooted reference path refers to. In a test file Chutzpah parses the <reference path=”” /> tags to determine your test dependencies. By default  (the DriveRoot setting) if you write <reference path=”\a.js” /> Chutzpah will look for a.js under C:\ (assuming you are running your test in the C drive). However, if you set this property to SettingsFileDirectory Chutpah will now look for a.js relative to the location of the chutzpah.json file. This is useful if you have a large project and want to minimize the number of long reference paths.

Examples:

This file tells Chutzpah to use QUnit and to place the test harness next to the chutzpah.json file:

{
    "Framework": "qunit",
    "TestHarnessLocationMode": "SettingsFileAdjacent"
}

 

This file tells Chutzpah to convert TypeScript to ES5 code and to place the test harness in a custom directory:

{
    "TypeScriptCodeGenTarget" : "ES5",
    "TestHarnessLocationMode": "Custom",
    "TestHarnessDirectory": "../Some/Folder"
}
Posted in Chutzpah | 3 Comments

Chutzpah 2.2 with TypeScript support

This release contains a few new features most notably that Chutzpah can now work directly on TypeScript files. You can get the new bits from CodePlex, NuGet or go to the Visual Studio Gallery to get the updated Visual Studio Context Menu Extension  and Unit Test Explorer adapter for Visual Studio 2012.

Changes

  • Added support for writing tests in TypeScript (item 67
  • Fixed encoding problem with .coffee files (item 63)
  • Allow reference paths for folders (item 68)
  • Some Unit Test Explorer bug fixes
  • Added clearer error messages when test file does not exist

TypeScript Support

Chutzpah now supports running unit tests you write in TypeScript. Chutzpah will automatically generate temporary JavaScript files for your referenced TypeScript files and use those to gather the test results.
For example given a TypeScript test file that references another TypeScript file:

/// <reference path="../code/code.ts" />

test("will add 5 to number", function () {
    var res:number = mathLib.add5(10);

    equal(res, 15, "should add 5");
});

It will parse that file as well as the referenced code.ts files and generate corresponding .js files.

If you are writing QUnit tests be aware that you need to write QUnit.module instead of just module since module is a reserved word in TypeScript.

Referencing Folders

In your test files you can now reference folders using the reference declarations. For example when Chutzpah sees this line:

/// <reference path="../someFolder/" />

It will traverse that folder and add the files in it as references. This can be easier to manage than referencing many files individually.

Posted in Chutzpah, TypeScript | 16 Comments

Quickly moving up a directory tree

I am lazy and I have been working recently on optimizing my PowerShell profile to become faster from the command line. My most recent target was a command I run often to move up a directory:

> cd ../

Now that may not look like a lot of characters but it gets worse when I want to jump 3 or 4 levels up:

> cd ../../../../

Annoying! To fix this I added this little snippet to my PowerShell profile:

for($i = 1; $i -le 5; $i++){
  $u =  "".PadLeft($i,"u")
  $unum =  "u$i"
  $d =  $u.Replace("u","../")
  Invoke-Expression "function $u { push-location $d }"
  Invoke-Expression "function $unum { push-location $d }"
}

This beautiful bit of code generates two sets of functions. One lets me move up a directory a given number of times by running a command like:

> u4

This will move up four directories. The second set of functions does the same thing but by typing u the number of times you want to traverse up:

> uuuu

Now I can hear you saying “Hey Matt, You claim to be all about being fast from the command line so why would you ever run the command uuuu when you can just run u4?”

That is an astute observation nameless reader! But sometimes I am in a mood where it is satisfying to press one key repeatedly. Ok?

Posted in PowerShell | 4 Comments