Chutzpah 3.0: Mocha, RequireJS and more…

Chutzpah 3.0 resolves two of the most requested features: Mocha test framework and better RequireJS support.  In addition to those this release contains many bug fixes and a few smaller additions. Below I will describe in detail how to take advantage of these new features. As always  you can get the new bits from CodePlex, NuGet or go to the Visual Studio Gallery to get the updated Visual Studio Context Menu Extension  and Unit Test Explorer adapter.

Changes

Improved RequireJS Support

The most common source of issues for Chutzpah was people trying to get it to work with RequireJS. Although it was possible to make it to work it was very tricky and would crumble if you tried anything complicated. The new support for RequireJS should be much more flexible and allow you to run individual test files however it does require a few specific settings to work. I checked in examples for how to set this up for Mocha, QUnit, Jasmine and TypeScript. I will walk through a simplified version of the  QUnit example in both JavaScript and in TypeScript.

RequireJS with QUnit using JavaScript

In this sample there is a RequireJS module in the base folder and a test file in the tests/base folder. Below I show the file structure of the sample and the source of the test file and the code file tests.

File Structure

chutzpah.json
require-2.1.8.js 

base/
  core.js 

tests/
  base/
    base.qunit.test.js

core.js

define(function () {
    return {
        version: 8
    };
});

base.qunit.test.js

define(['base/core'],
    function (core) {
        module("base/core");
        test("will return correct version from core", function () {
            var version = core.version;
            equal(version, 8);
        });
    });

The key to being able to run the base.qunit.test.js test file is to set up the chutzpah.json file. This sample uses the following chutzah.json file:

{
    "Framework": "qunit",
    "TestHarnessReferenceMode": "AMD",
    "TestHarnessLocationMode": "SettingsFileAdjacent",
    "References" : [
        {"Path" : "require-2.1.8.js" } 
    ]
}

The chutzpah.json file is making use of a few new settings in version 3.0.  Setting TestHarnessReferenceMode to AMD tells Chutzpah that we are running tests that use the AMD style. In this mode Chutzpah will not insert references (discovered from ///<reference comments and the test file into the test harness. Instead it will inject a require statement in the test harness with the right path to the file under test.  I then list explicitly which references I want injected into the test harness (in this case just require.js). The settings file should be place at the root of your AMD project (i.e. where you want all your paths relative to) and you should set the TestHarnessLocationMode to be SettingsFileAdjacent which places the generated harness in the same folder.

With this setup you can run chutzpah as follows:

chutzpah.console.exe tests\base\base.qunit.test.js

RequireJS with QUnit using TypeScript

This is the same example as above expect with everything converted to TypeScript. To be able run this example there are a couple extra settings in the Chutzpah.json file.

File Structure

chutzpah.json
require-2.1.8.js
require.d.ts
qunit.d.ts 

base/
  core.ts 

tests/
  base/
    base.qunit.test.ts

core.ts

export var version = 8;

base.qunit.test.ts

import core = require('base/core');

QUnit.module("base/core");
test("will return correct version from core", function () {
    var version = core.version;
    equal(version, 8);
});

The TypeScript versions of these files make use of TypeScript’s nice import/require syntax which make its cleaner to write code in the AMD style.

The chutzpah.json file needed to run this sample is:

{
   "Framework": "qunit",
   "TestHarnessReferenceMode": "AMD",
   "TestHarnessLocationMode": "SettingsFileAdjacent",
   "TypeScriptModuleKind": "AMD",
   "References" : [
      { "Path": "require-2.1.8.js" },
      { "Path": "require.d.ts", "IncludeInTestHarness": "false" },
      { "Path": "qunit.d.ts", "IncludeInTestHarness": "false" },
      { "Path": "base", "IncludeInTestHarness": "false" }
    ]
}

This chutzpah.json file is a bit more complicated than the one in the previously.It sets TypeScriptModuleKind to AMD, which tells the TypeScript compiler to emit AMD define statements. It also has a few additional References path entries. There are references for the .d.ts files to satisfy type checking. Then we reference the base folder so that Chutzpah will recursively scan that folder and find all files in it. This is important since Chutzpah needs to find all the .ts files in order to convert them to JavaScript. Chutzpah will also use these discovered .ts files to generate a RequireJS map setting that maps from the original file names to the temporary names that Chutzpah creates after conversion. All of these additional path references have IncludeInTestHarness set to false. This tells Chutzpah to not generate a script reference for these files inside of the generated test harness. For the .d.ts files we don’t want script tags and for all the files in the base directory are going to load them using RequireJS so we must not include them a second time using a script tag.

For more complicated examples browse the samples folder in the Chutzpah repository.

Mocha Test Framework Support

One of the most request features was to add the Mocha test framework to the supported frameworks. Mocha presents some challenges since it is a super flexible and open ended test framework. This required several changes in Chutzpah as well as some new chutzpah.json options to make sure it is flexible enough. In the future (based on user feedback) I will try to open up the Mocha integration further to let people do things like use custom interfaces.

Chutzpah supports three of the Mocha interfaces: QUnit, BDD and TDD. It will try to auto detect which one you are using but you can also specify explicitly. For example here is a basic Mocha test using the BDD interface:

/// <reference path="mocha.js" />
/// <reference path="chai.js" />

var expect = chai.expect;

it("A basic test", function() {
    expect(true).to.be.ok;
    var value = "hello";
    expect(value).to.equal("hello");
});

Note that this file includes a ///<reference comment referencing mocha.js. This file doesn’t need to exist since Chutzpah uses its own version but this line is needed to let Chutzpah know you are using Mocha. Since Mocha’s interfaces can look like Jasmine/QUnit Chutzpah won’t auto detect that you are using Mocha. In addition, since Mocha lets you choose your assertion library the test above is referencing chai.js (this does need to exist on disk since Chutzpah does not ship with this). Putting those reference comments in every file can be annoying so you can use the Chutzpah.json file remove the need for them:

{
    "Framework": "mocha",
    "MochaInterface": "bdd",
    "References" : [
      { "Path": "chai.js" }
    ]
}

This Chutzpah.json example shows that you can set the Framework to Mocha, set the interface to BDD and then list the references you want Chutzpah to use. While you don’t always need to explicitly set the Mocha interface it doesn’t hurt to be clear with your intent.

Customizable HTML harness generation

Chutzpah tries its best to work for as many types and configurations of projects. However, there will always be projects where Chutzpah’s assumptions just don’t hold true. To help support these projects Chutzpah now allows you to take complete ownership of the template it uses to generate the html test harness. Be warned this is an advanced feature, by taking ownership of this template you are responsible for keeping it up-to-date with newer additions Chutzpah makes in later releases. Also, by editing this template Chutzpah can no longer make any guarantees that features like Code Coverage will still work.

If that red text did not scare you off continue on :)

To use this functionality you should start by making a copy of existing copy of the Chutzpah test harness template that is checked in (QUnit, Mocha, Jasmine). These templates contain a bunch of JavaScript and HTML embedded with special Chutzpah placeholders. These placeholders are where Chutzpah inserts text and tags it generates when examining your test files. As of version 3.0 the harness contains the following placeholders:

Placeholder Description
@@TestFrameworkDependencies@@ The test framework file references (like QUnit.js) and any references marked as IsTestFrameworkFile.
@@CodeCoverageDependencies@@ The code coverage framework file references (like Blanket.js). This will be empty when not running code coverage.
@@ReferencedCSSFiles@@ The CSS references.
@@TestHtmlTemplateFiles@@ Any test html template files discovered.
@@ReferencedJSFiles@@ Reference for all JS files discovered that your test files are dependent on.
@@TestJSFile@@ The reference for the file under test. This will be empty if in AMD mode.
@@AMDTestPath@@ The AMD path to the file under test.
@@AMDModuleMap@@ A mapping of AMD paths from original test dependencies to ones generated after running TypeScript or CoffeeScript conversion.

You can edit this copy to your liking and then place it somewhere in your project. To get Chutzpah to use this template set the following in your chutzpah.json file:

{
  "CustomTestHarnessPath": "path\\to\\customQUnit.html"
}

Chutzpah will use this template and generate a test harnesses out of it.

Command line failure report flag

When running Chutzpah on a large test suite from the command line it becomes difficult to see everything that failed because the failed test may be several pages up. To alleviate this problem chutzpah.console.exe contains a new flag: /showFailureReport . This flag will print a report showing all test errors and test failures at the ends of the test run.

Chutzpah.json additions

To support some of the new features and to fix existing issues the Chutzpah.json file has several new additions (many of which have been mentioned earlier in this post). Below is a complete list of the additions along with usage examples:

References

Description

The references setting allows you to specify which files/folders to use/scan to find references. This is useful since it replaces the need to the ///<reference comments. This setting is a list of path entries each which can contain the following arguments:

Path The path to either a file or a folder. If given a folder, it will be scanned recursively. This path can be relative to the location of the chutzpah.json file.
Include This is an optional include glob pattern. This is used when the Path is a folder. Only files matching the Include pattern will be added.
Exclude This is an optional exclude glob pattern. This is used when the path is a folder. All files matching the exclude pattern will not be added.
IncludeInTestHarness This determines if the reference should be injected into the test harness. When referencing files like .d.ts or files that you plan to load using require.js you should set this to false. Defaults to true.
IsTestFrameworkFile Indicated that this references should be placed directly after the test framework files (like QUnit.js) in the test harness. This ensures that this file is injected into the test harness before almost all other files. Defaults to false.

Examples

Include files named a.js and b.js

{
    "References": [
        { "Path": "../a.js" },
        { "Path": "b.js"}
    ]
}

Include all .js files in a folder unless their name contains the word Resource

{
    "References": [
        { "Path": "src/Code", "Include": "*.js", "Exclude": "*Resource*" }
    ]
}

TypeScriptModuleKind

Description

The TypeScriptModuleKind setting determines how the TypeScript compiler handles import/export statements. The options are either CommonJS or AMD. The default is CommonJS.

Example

{
   "TypeScriptModuleKind": "AMD"
}

TestHarnessReferenceMode

Description

The TestHarnessReferenceMode setting determines how Chutzpah should treat references to scripts and to the test file. The setting can either be Normal or AMD. Normal is how Chutzpah has always worked where it injects a script reference for your test file as well as all of its references into the HTML test harness. The AMD mode tells Chutzpah that tests using RequireJS are being used. In this case Chutzpah will not inject references or the test script into the test harness. The default value is Normal.

Example

{
   "TestHarnessReferenceMode": "AMD"
}

CustomTestHarnessPath

Description

The CustomTestHarnessPath setting allows you to override the default template Chutzpah uses for the HTML test harness. This is an advanced feature which was explained in detail above.

Example

{
  "CustomTestHarnessPath": "path\\to\\customQUnit.html"
}

EnableCodeCoverage

Description

The EnableCodeCoverage settings provides a way to have Chutzpah always run code coverage. This is important for when running on a build server where you may want Chutzpah to always collect code coverage. The default value is false.

Example

{
  "EnableCodeCoverage": "true"
}

CoffeeScriptBareMode

Description

The CoffeeScriptBareMode indicates if you want CoffeeScript compilation to output scripts in bare mode or not. In bare mode CoffeeScript won’t wrap the generated JavaScript in a function to protect the scope. The default value is true.

Example

{
  "CoffeeScriptBareMode": "false"
}

 

  • Qube

    This is a tremendous release. Thanks for all your hard work Matt!

  • Pierre

    This is great news Matthew.
    We’ve been working around the previous limitations for require.js.

    Feels great to be able to refactor those specs as well as replacing jasmine with mocha.

    Thank you

  • Marco

    This is an awesome release! Really like the native requirejs support!

    Unfortunately I do have some problems with it. I upgraded from 2.4.3 to 3.0 and the Test Adapter (VS2013) isn’t finding any of my Jasmine tests anymore.
    The test-filenames are foo_tests.js
    I’ve even added the basic-jasmine.js testfile to our project but that didn’t appear in the test explorer either.
    I uninstalled the Chutzpah test-runner completely and re-installed it but to no avail.
    Our chutzpah.json looks like this:
    {
    “TestFileTimeout”: “10000”
    }

    Is there some additional setting (since 3.0) we need to set?

    Ps. I also enabled the tracing but in the chutzpah.log it only says “Discover test started” and “Discover test finished”. No additional errors or something.

    • https://www.google.com/accounts/o8/id?id=AItOawlrLeowWytSCscAcNv3ky4tdtP7AcgDAC8 Matthew

      Sorry for the issues. I have been getting several people reporting this but I am having a *very* hard time reproducing it. I am actively looking into but until I get a repro its hard to figure out why its working for me and not others.

      • Marco

        Hi Matthew,

        I’ve got a small test-project that shows the issue, the test in it are discovered on a machine with version 2.4.3 but not on a machine with version 3.0. Not sure if you would like to have that project, if so let me know.
        Furthermore I’ve been trying to debug the TestAdapter to see what’s going on. I followed the instructions you blogged about here http://matthewmanela.com/blog/anatomy-of-the-chutzpah-test-adapter-for-vs-2012-rc
        For some reason the breakpoints i set in the TestAdapter are not loaded, so they are not hit. I did debug the Adapter from VS2013 and it did start the VS Experminetal version. Attached to the vsTest.discoveryEngine but to no avail. Maybe you know if i need to do something different to be able to debug the discovery?

      • https://www.google.com/accounts/o8/id?id=AItOawlrLeowWytSCscAcNv3ky4tdtP7AcgDAC8 Matthew

        I discovered an issue in version 3.0 with the Test Adapter integration inside of VS 2013 that many users are hitting. I am actively working on fixing this issue but until then if you encounter this you can either use the context menu extension or use version 2.5 of the Test Adapter integration

        For debugging, I don’t think you are doing anything wrong, part of the bug is that the test adapter is not getting called! So the break points won’t it.

      • https://www.google.com/accounts/o8/id?id=AItOawlrLeowWytSCscAcNv3ky4tdtP7AcgDAC8 Matthew

        I released the fix for the issue in version 3.0.1

  • Dave

    Hey, this is a fantastic update. Thanks for your work on Chutzpah!

    I haven’t tried it yet, but is anybody using Chutzpah with the newly-released 2.0 version of Jasmine?

    Thanks again-

    • https://www.google.com/accounts/o8/id?id=AItOawlrLeowWytSCscAcNv3ky4tdtP7AcgDAC8 Matthew

      Chutzpah does not support Jasmine 2.0 yet.

  • Ben

    I saw a video that showed Chutzpah running tests continually. I can’t get 3.0.1 to do that for me. The button that was being used in the video just gives me options to group the tests instead of enabling ‘Run tests after build’. Is this feature still in 3.0.1?

    • https://www.google.com/accounts/o8/id?id=AItOawlrLeowWytSCscAcNv3ky4tdtP7AcgDAC8 Matthew

      In my VS I still have the run tests after build button. What version of VS do you have?

  • james

    It’s too bad you couldn’t make chutzpah depend upon whichever typescript version is installed instead of embedding it.

    • https://www.google.com/accounts/o8/id?id=AItOawlrLeowWytSCscAcNv3ky4tdtP7AcgDAC8 Matthew

      Its not that it can’t happen its just that is hasn’t happened yet. There is a work item on chutzpah.codeplex.com to allow Chutzpah to let you specify where your compiler is.

  • http://www.photon.in Senthil Kumar T D

    Hi,

    The chutzpah will support angular.js unit tests?

    Regards,
    Senthil

    • https://www.google.com/accounts/o8/id?id=AItOawlrLeowWytSCscAcNv3ky4tdtP7AcgDAC8 Matthew

      Does Chutzpah need to do anything special to support Angular.js? I know other people have already been able to use it with Angular.