Chutzpah and Snippet Designer now support Visual Studio 2013

I released updates for both Chutzpah (2.5.0) and Snippet Designer (1.5.0) which adds support for Visual Studio 2013.

In addition Chutzpah’s command line client now has a /trace argument that will dump detailed trace information into a file named chutzpah.log. This is useful for both debugging and filing bug reports. I plan to add much more information into this log in the upcoming releases.

 

Chutzpah 2.4.3

This release of Chutzpah contains a few bug fixes, upgraded versions of Blanket.js and TypeScript and a new feature for injecting HTML templates.

TypeScript Upgraded

Chutzpah now uses TypeScript 0.9.0.1. This change may require an update for your .ts files. In this version of TypeScript the compiler errors if you do not specify the .d.ts file corresponding to your third party JS library. For example, if you are using QUnit you must reference the QUnit.d.ts file.

Improved Code Coverage

Code coverage has improved with two changes. The version of Blanket.js was upgraded to 1.1.5. In addition, Chutzpah will now prevent files you exclude from code coverage from getting instrumented. This helps improve performance of the coverage runs.

HTML Template Feature

This feature came in as pull request from Matthew Osborn. This allows you to specify an HTML template file from your tests. Chutzpah will inject this templates content into the head section of the generated test harness. For example, lets say you have an HTML template file named template.tmpl.html with the contents:

<script id="testTemplateId" type="text/html">
    <div id="testTemplateDiv">
        Cool!
    </div>
</script>

Then you can reference this file from your test with the syntax:

/// <template path="template.tmpl.html"/>

For example, this is a test case I wrote in Chutzpah to assert this feature works correctly:

/// <reference path="qunit.js" />
/// <template path="template.tmpl.html"/>
/// <reference path="../jquery-1.7.1.min.js" />

module("Html Template Test - QUnit");
test("Will load html template", function () {
    // Grab template text, convert to html, append to body
    $($("#testTemplateId").text()).appendTo("body");

    var templateDiv = $("#testTemplateDiv");

    equal(templateDiv.length, 1);
});

Which results in an HTML test harness which looks something like this:

<!DOCTYPE html>
<html>
<head>
  <link rel="stylesheet" type="text/css" href="qunit.css"/>
  <script type="text/javascript" src="qunit.js"></script>
  <script id="testTemplateId" type="text/html">
      <div id="testTemplateDiv">
          Cool!
      </div>
  </script>
  <script type="text/javascript" src="query-1.7.1.min.js"></script>
  <script type="text/javascript" src="template-qunit.js"></script>
</head>

<body>
    <h1 id="qunit-header">Unit Tests</h1>
    <h2 id="qunit-banner"></h2>
    <h2 id="qunit-userAgent"></h2>
    <ol id="qunit-tests"></ol>
    <div id="qunit-fixture"></div>
</body>
</html>

The content from the template was injected into the head section above. This ability is useful for things like jQuery widgets and JS template renderers.

 

Chutzpah 2.4 – Code Coverage is a go!

With the Chutzpah 2.4 release one of the most asked for features is now available: built-in code coverage. In addition this release contains a bunch of bug fixes and added configurability.

You can get the new bits from CodePlex, NuGet or go to the Visual Studio Gallery to get the updated Visual Studio Context Menu Extension  and Unit Test Explorer adapter for Visual Studio 2012.

Changes and Fixes

  • Added Code Coverage support (item 113)
  • Updated versions of QUnit(v1.11.0), TypeScript(v0.8.3) and CoffeeScript(v1.6.2)
  • jasmin-jQuery toContain tests fail with reason “Passed.” (item 95)
  • FileSearchLimit=300 cancels test run with more than 300 tests (item 89)
  • /failOnScriptError exits with 1 even when there are not errors (item 97)
  • ‘undefined’ is not a function after updating to 2.3 (item 96)
  • No tests found when using QUnit.test instead of test (item 94)
  • Allow including files without building them (item 93)
  • Improve “/openInBrowser” (item 91)
  • Add TestFileTimeout setting to chutzpah.json (item 110)
  • Need a later version of phantom bundled with Chutzpah (item 107)
  • QUnit test runner module gotcha in 2.3 (item 109)
  • Support for unicode (item 111)
  • Improve CoffeeScript parse error reporting (item 80)

 

Code Coverage

Chutzpah’s new code coverage features is built on top of the great open source library Blanket.js. When you run with code coverage enabled it will use Blanket.js to instrument the files under test and gather the coverage results. You can perform code coverage from the command line and both Visual Studio plugins.

Command Line

To get code coverage from the command line you pass the /coverage flag to chutzpah.console.exe.

chutzpah.console.exe /coverage /path test.js

After this command finishes you will have a two files placed in your current directory

_Chutzpah.coverage.html – The rendered html results
_Chutzpah.coverage.json – The raw json results

When you open the results html file you will see something like:

image

 

Visual Studio Context Menu Extension

The Visual Studio Context Menu Extension now contains a Show Code Coverage command which will perform code coverage and open the results in a browser.

image

Visual Studio 2012 Test Adapter Extension

For the Unit Test Explorer adapter for Visual Studio 2012 Chutzpah piggybacks on the existing “Analyze Code Coverage” menu option and uses it to run with code coverage and show the results in a browser. Ideally, Chutzpah would integrate into the code coverage reporting built into VS but I was unable to figure out a good way to plug in to that. The side effect is that when you run this command VS will open their results window first (with nothing in it) and then Chutzpah will launch its results in a browser.

image

NOTE: I have noticed some performance issues when running “Analyze Code Coverage for All Tests” command. I think it might be VS doing extra work for no reason since it is trying to generate its own report.

 

Configuration Code Coverage

Being able to run code coverage is not enough since in many cases the results will not be accurate. For example, if your project uses jQuery you do not expect your test to cover all the lines of jQuery’s source code (that would be one impressive test). Therefore you need to exclude files/directories from participating in code coverage. There are two way you can do that with Chutpah: command line arguments or a chutzpah.json file.

If you are running Chutzpah from the command line there are two additional arguments you can pass coverageIncludes and  coverageExcludes. These are a comma separated list of files/paths you want to include or exclude from coverage. They can use a glob format where * matches zero or more characters and ? matches one character. For example:

Exclude files that ends in matt.js

chutzpah.console.exe /path test.js /coverage /coverageExcludes *matt.js

 

Include only dog.js and any file with a single character extension

chutzpah.console.exe /path test.js /coverage /coverageIncludes "*\dog.js, *.?"

 

Include *query.js but exclude jQuery.js

chutzpah.console.exe /path test.js /coverage /coverageIncludes *query.js /coverageExcludes *\jquery.js

 

The other option for configuring code coverage is the chutzpah.json file. There are two new setting which you can use CodeCoverageExcludes and CodeCoverageIncludes. These settings work in the same way as the their command line counterpart. The main difference is that you must escape backslashes.

{
  "CodeCoverageExcludes": ["*\\jquery.js"],
  "CodeCoverageIncludes": ["*query.js", "*\\dog.js"]
}

Test timeout setting

Chutzpah already has a test timeout setting but you previously could only configure it from the command line or with a setting in VS. This was inflexible especially if you want a different timeout for different tests. To solve this Chutzpah now include a TestFileTimeout setting in the chutzpah.json file. For example if you want a 10 second time out for a certain directory just place a chutpah.json file with the following in it:

{
    "TestFileTimeout": "10000"
}

Chutzpah Specific Reference

With the addition of TypeScript support a few releases ago several people reported issues when using Chutzpah with the TypeScript plugin in VS. The problem occurs since both TypeScript and Chutzpah are using the <reference path…> comments for different purposes. To get around this issue Chutzpah now supports two ways to reference dependent files.

1. The existing general way

/// <reference path="foo.js" />

2. The new Chutzpah specific way

/// <chutzpah_reference path="foo.js" />

Using the latter will let Chutzpah know about your dependency graph and should not confuse the TypeScript compiler.

Excluding References

The reference syntax is also used by VS to give intellisense in JS files. This leads to an issue since sometimes you may reference a version of a file just for intellisense but you do not want Chutzpah to copy it. In this case you now can include the chutzpah-exclude attribute to have it ignored by Chutzpah:

/// <reference path="someDependency.js" chutzpah-exclude="true" />

Running against a remote url

Chutzpah now supports executing a remote test harness. If you have a html test file on a remote server (like http://somesite.com/test.html) you can tell Chutzpah to execute that file directly

chutzpah.console.exe http://somesite.com/test.html

Note: Chutzpah does not support running anything but .html files remotely. If you want to run Chutzpah directly on a .js, .ts or .coffee file they must be local on disk.

 

Using SignalR in an ARR cluster

My team ran into an issue when deploying an application which uses SignalR on a clustered environment using Application Request Routing (ARR). We created a simple three server setup with one server as an ARR node and 2 servers running the application. When accessing the site we saw some odd behavior.

1. All SignalR requests fail to use the more efficient transports (ServerSentEvents/ForeverFrame) and fall back to using Long Polling

2. Some messages get replayed multiple times in IE9

After looking into the issue for a while we figured out it was caused by a setting in ARR called Response Buffer Threshold.

Response buffer threshold (KB)

Lists the size of the buffer, in kilobytes, that ARR uses to retain a response before sending it to the client. Content that is smaller in size than the threshold can be cached.

By default this is set to 256kb which means that it will buffer responses until they reach that amount. By setting this to 0 ARR will no longer buffer and SignlR will function correctly.

clip_image002

 

Chutzpah 2.3 Released

This release contains a couple important new features and bug fixes. Chutzpah now gives you more control in how you want youe tests run using the new chutzpah.json settings file (described in detail below).

You can get the new bits from CodePlex, NuGet or go to the Visual Studio Gallery to get the updated Visual Studio Context Menu Extension  and Unit Test Explorer adapter for Visual Studio 2012.

Changes and Fixes

  • Added Chutzpah.json settings file support
  • Consistent conversion error (item 70)
  • System.IO.IOException: The handle is invalid when running from inside WebStorm (item 77)
  • Exception has been thrown by the target of an invocation. inside Visual Studio 2012 (item 65)
  • Tests are hanging (ChutzpahTimeoutException) (item 72)
  • Reference root path should be project (item 83)
  • Smarter coffeescript compilation (item 75)
  • Running with parallelism gives faulty timing (item 81)
  • Running with parallelism gives System.IO.IOException (item 74)
  • Compile all TypeScript files at the same time (item 82)

The Chutzpah Settings File

The Chutzpah settings file is an .json file that you can place anywhere in your testing directories to inform Chutzpah of test specific settings. The Chuzpah settings file is completely optional and if the file or any of its properties are omitted Chutzpah will use the default values.

When Chutzpah is run on a file (lets say test.js) it scans the current directory for a file named chutzpah.json. If it does not find one it will recursively traverse up the directory chain looking for one. This behavior provides a lot of flexibility since you can provide multiple chutzpah.json files in your tree and each one will work over its sub-folder. This lets you have different settings for different groups of tests.

Chutzpah.json Format

The settings file is a text file with json content of the following format:

{
    "Framework": "qunit|jasmine",

    "TestHarnessLocationMode": "TestFileAdjacent|SettingsFileAdjacent|Custom",

    "TestHarnessDirectory": "<Path to a folder>",

    "TypeScriptCodeGenTarget" : "ES3|ES5",

    "RootReferencePathMode":"DriveRoot|SettingsFileDirectory"
}

The chutzpah.json file supports the following properties

Framework – Determines what testing framework to use. This will override the other detection methods. This is helpful if Chutzpah is having trouble figuring out what your intended framework is. If left out Chutzpah will automatically try to detect the framework.

TestHarnessLocationMode – Determines where to place the generated html test harness files. The default mode is TestFileAdjacent which means the harness is placed in the same directory as the test file. SettingsFileAdjacent means it is placed in the same directory as the chutzpah.json file. Custom mode lets you specify a custom path to the directory.

TestHarnessDirectory – When TestHarnessLocationMode is set to Custom this is either the relative or absolute path to the directory where to place the test harness.

TypeScriptCodeGenTarget – When running on a TypeScript file this property will tell the TypeScript compiler what type of code to generate. ES3 is for ECMAScript 3 and ES5 is for ECMAScript 5.

RootReferencePathMode – This property determines what directory a rooted reference path refers to. In a test file Chutzpah parses the <reference path=”” /> tags to determine your test dependencies. By default  (the DriveRoot setting) if you write <reference path=”\a.js” /> Chutzpah will look for a.js under C:\ (assuming you are running your test in the C drive). However, if you set this property to SettingsFileDirectory Chutpah will now look for a.js relative to the location of the chutzpah.json file. This is useful if you have a large project and want to minimize the number of long reference paths.

Examples:

This file tells Chutzpah to use QUnit and to place the test harness next to the chutzpah.json file:

{
    "Framework": "qunit",
    "TestHarnessLocationMode": "SettingsFileAdjacent"
}

 

This file tells Chutzpah to convert TypeScript to ES5 code and to place the test harness in a custom directory:

{
    "TypeScriptCodeGenTarget" : "ES5",
    "TestHarnessLocationMode": "Custom",
    "TestHarnessDirectory": "../Some/Folder"
}