Chutzpah 4.0 – Batching, Inheritance and more

Chutzpah 4.0 cleans up some of the legacy baggage while adding the much requested feature of test file batching.

Changes


Test File Batching

Chutzpah has always created an HTML harness for each of your test files. This has the advantage of isolated one test from another but it has the big draw back of limiting throughput of running all your tests. Especially for people with large suite of tests parallel execution of individual test harnesses will not be as fast as batching them all together in one harness. With this release Chutzpah now adds support for both modes. When you set the following in your chutzpah.json file:

"EnableTestFileBatching": true

It tells Chutzpah to put all the test files under this chutzpah.json file into one HTML harness (see full sample). Because the settings file determines the grouping of files that get combined into one harness it gives you the ability to determine what tests you want to get loaded into one harness. Chutzpah can then run those harness in parallel which can give you increased throughput. In the future (based on user feedback) batching may become the default.

 

Settings File Inheritance

The addition of the test file batching feature above encourages the use of multiple chutzpah.json files to help control how files are batched and parallelized. This presents a problem though since your project will often have a set of common files and you would rather not list them multiple times in different settings files. To address this you can now inherit settings from a parent chutzpah.json file. In your chutzpah.json file (see full sample) you write:

"InheritFromParent": true

Which instructs Chutzpah to traverse recursively up the folder hierarchy and look for another chutzpah.json file. If found it will merge the two files and combine their settings. The inheritance between a parent and child settings works as follows when both the parent and child files contain the same setting:

Setting Name What happens?
References References from child template are prepended with the references from the parent template
Templates Templates from the child template are prepended with the templates from the parent template
Tests Not inherited
*All Others* Child template overwrites the parent template setting

 

Multiple Includes/Excludes for Tests and References Settings

The Tests and References settings now have Excludes and Includes properties (in addition to the old singular Exclude and Include setting). The new plural settings let you specify an array of patterns. This provides more flexibility when describing what files you do and do not want to include for a folder. For example:

{
    "Tests": [
        { "Path": "someFolder/someOtherFolder", "Includes": [ "*Dir1/*.js", "*Dir2/*.js"], "Excludes": [ "*Dir1/test4.js", "*Dir2/test2.js" ] }
    ]
}

 

Legacy Compilation Support Removed

The 4.0 release removes the legacy compilation support. Chutzpah no longer ships with a built in TypeScript and CoffeeScript compiler. Please use the compile setting now to configure how Chutzpah handles TypeScript or CoffeeScript files.

 

Chutzpah 3.3.0

Chutzpah 3.3.0 is now available at your usual locations.

Changes

  • Update to QUnit 1.16
  • Update to Jasmine 2.1.3
  • Improve parsing of Chutzpah.json files (finally allow comments)
  • Skip paths that are too long while discovering tests
  • Integrate test result transformers into Chutzpah.json file
  • Add LCOV test result transformer
  • Add TRX test result transformer
  • Add source map support for code coverage
  • Fix issue with requirejs urls containing querystrings
  • Move HTML template injection from HEAD tag to BODY tag

 

Test Result Transforms

Chutzpah has long had support to transform your test results into the JUnit XML format but this was only exposed from the command line. With recent additions this feature has been greatly improved: Chutzpah now supports JUnit, LCOV (for code coverage) and TRX file formats. In addition, you can now specify what transforms you want and where the output files should live using the chutzpah.json file.

For example the settings below will output the results as lcov, trx and junit to the specified paths whenever Chutzpah runs.

{
    "Transforms": [
        { "Name": "lcov", "Path": "C:\temp\lcov.dat" },
        { "Name": "junit", "Path": "C:\temp\junit.xml" },
        { "Name": "trx", "Path": "C:\temp\trx.trx" }
    ]
}

 

Code Coverage Source Map Support

One annoyance with Chutzpah’s code coverage was that is always showed coverage in terms of the .js files even if your source files were TypeScript or CoffeeScript. To address that problem this release contains a new option as part of the compile settings called UseSourceMaps. This will instruct Chutzpah to look for source map files in your configured compile output directory. If it finds them it will map your coverage results to the lines of the source files and generate the coverage HTML accordingly.

Here is an example config using this setting:

{
    "Compile": {
        "UseSourceMaps": true,
        "Mode": "External",
        "Extensions": [ ".ts" ],
        "ExtensionsWithNoOutput": [ ".d.ts" ]
    },
    "Tests": [
        { "Path": "Tests.ts" }
    ],
    "References": [ 
        { "Path": "SourceMappedLibrary.ts" }
    ],
    "CodeCoverageExcludes": ["*Tests.ts"]
}
 

Chutzpah is now on GitHub

Today, I moved Chutzpah from CodePlex to GitHub. I made this move for a few reasons

  1. The feature set GitHub offers has grown increasingly compelling.
  2. There is a more active community on GitHub and I am hoping this will lead to more user engagement.
  3. I have received feedback from multiple Chutzpah users that they would like to have it moved to GitHub.

 

Source Code

Migration of the source code was simple since it’s all git.

 

Documentation

The migration of the documentation was simple but time consuming. CodePlex uses a different syntax than MarkDown so I manually ported all the pages. If you notice any issues likes broken links please let me know.

 

Issues

The issue migration was interesting. I have a couple hundred closed/active issues on the CodePlex site. I considered just leaving the old ones on Codeplex but I really like having them all in one place. When I searched to find if anyone wrote a tool to do this I found the CodePlex Issues Importer, perfect! Well, there was a catch. That project hasn’t been updated in three years which meant it was using an old version of the GitHub API and was screen scraping looking for non-existent elements on the CodePlex site. I still started playing with this project and with some minor changes I was able to get it to work and import all of my issues to GitHub! I sent a PR to the CodePlex Issues Importer project with my fixes to help anyone else who wants to migrate.

 

What’s Left

There are still a few links that point to the CodePlex site in the VSIX packages. I will update these during the next release.

 

Avoiding side effects when traversing an IVsHierarchy in Visual Studio

There was a bug filed against Chutzpah reporting that Chutzpah’s Visual Studio extension was causing database connection error popup. I was surprised by this since all Chutzpah really does is traverse the project hierarchy and I wasn’t sure how this could result in that error. After digging in and searching online a bit I found the issue. The act of traversing a node in the IVsHierarchy can cause arbitrary side affects (like establishing a database connection!).  In order to determine if this can happen you must check the __VSHPROPID.VSHPROPID_HasEnumerationSideEffects property on an IVsHierarchy item. Once I found this the fix was simple, just skip any node that has that property set.

 

Chutzpah 3.2 – A smarter approach to compilation

Chutzpah 3.2 introduces a new and improved method for supporting languages which compile to JavaScript. This version puts the power in your hands to tell Chutzpah how to compile and lets Chutzpah get out of your way. As always  you can get the new bits from CodePlex, NuGet or go to the Visual Studio Gallery to get the updated Visual Studio Context Menu Extension  and Unit Test Explorer adapter.

Changes

Smarter Compile

When I first added support for compiled languages (like TypeScript and CoffeeScript) to Chutzpah I was trying to make it super simple to use. I embedded both CoffeeScript and TypeScript inside Chutzpah and executed them at test time by hosting the IE script engine to produce the needed .js files. This approached worked ok but led to a few major issues/pain points:

  1. Versioning: Chutzpah needed to ship with a specific version of TypeScript since the api’s needed to run the compilation kept changing. This led to a bad issue of a user wanting to use version X but Chutzpah only supporting Y.
  2. Performance: Chutzpah’s compile was slow and not smart about when it needed to run.
  3. Memory: The hosting of the IE script engine led to a native memory leak when you run your tests using VS Test Explorer.
  4. Philosophical:  Chutzpah should not be responsible for determining how to compile your source code. It was a mistake to mix this concept into Chutzpah. It should have been decoupled from the start. Live and learn ;)

Given those pain points I still like the idea that you can point Chutzpah at your source files and you should still be able to get test results that point back to the test files. This led to the new chutzpah.json compile setting. This setting lets you describe in the Chutzpah.json file how to execute a command which can compile your source files to .js files. You tell Chutzpah what to execute and some information about what your executable does (like where to find the generated .js files). Then after running the executable Chutzpah can associate each source file with each output file to still give the nice behavior of mapping tests back to their original files.

You should ideally configure this compile setting to use the same mechanism you have for compiling the code in your project. This will give you a nice performance gain when running Chutzpah since it will re-use the .js files you have already built. This becomes really powerful when you are taking advantage of IDE integration features that TypeScript and CoffeeScript plugins have that provide a compile on save experience. If configured correctly Chutzpah will be able to use the the output of that feature and never need to actually invoke the compile step. This results in a *very* fast test running experience.

When Chutzpah sees that you have the compile configuration set it will not use the legacy integrated compilation.

Below are the options for the Compile setting:

Field Default Description
Extensions [] The extensions of the files which are getting compiled (e.g. .ts).
ExtensionsWithNoOutput [] The extensions of files which take part in compile but have no build output. This is used for cases like TypeScript declaration files which share a .ts extension. They have .d.ts extension and are part of compilation but have no output. You must tell Chutzpah about these if you want the SkipIfUnchanged setting to work. Otherwise Chutzpah will think these are missing output.
SourceDirectory Chutzpah.json directory The root directory where all the sources the command compiles are below. This lets Chutzpah know where in the out dir it should find each reference file
OutDirectory Chutzpah.json directory The directory where the compiled files are output to
WorkingDirectory Chutzpah.json directory This is the working directory of the process which executes the command.
Executable null The path to an executable which Chutzpah executes to perform the batch compilation. Chutzpah will try to resolve the path relative to the settings directory. But if can’t find the file there you must give it a full path.
Arguments null The arguments to pass to the command
Timeout 30000 (5 minutes) How long to wait for compile to finish in milliseconds?
SkipIfUnchanged true Skips the execution if all files Chutzpah knows about are older than all of the output files. This is defaulted to true but if you hit issues since it is possible Chutzpah might not know about all the files your compilation is using then you can turn this off. Ideally you should tell Chutzpah about these files using the references and tests settings since this setting helps Chutzpah not need to even invoke the executable if it figures out it’s not needed.
Mode Executable Determines how this compile setting is used. By default it is in Executable mode where it will require you provide an executable which Chutzpah will run if it sees it finds missing .js for input file. If you set this to External then Chutzpah will ignore the Executable, Arguments settings and assume you have some external process which is compiling. In this case Chutzpah will use the SourceDirectory and OutDirectory options to try to find your .js files for the input files. If it can’t find them it will trace an error but still attempt to proceed.

 

For the Executable, Arguments and OutDirectory fields Chutzpah lets you put in variables which get expanded at execution time. These can be system environment variables like %comspec% as well as the set of Chutzpah provided variables below:

%chutzpahsettingsdir% The directory the chutzpah.json is in
%clrdir% The directory to the folder where the CLR is. This folder contains programs like msbuild.exe
%msbuildexe% The path to msbuild.exe
%powershellexe% That path to powershell.exe
%cmdexe% An alias for %comspec% which is the path to cmd.exe.

 

Integration with compilation tooling

Many users of TypeScript and CoffeeScript use IDE tooling which will already provide compilation of those files to JavaScript. In that case you should Compile Mode setting to External. This tells Chutzpah to assume that some external force is doing the compilation and that Chutzpah can just count on the JavaScript file to exist where you said they would be.

Examples

Enough talk! As part of this release I have included several examples which show different combinations of these settings. Below I will highlight three examples. One runs your compilation code using a .bat file, one uses a .ps1 file and the last runs an MSBuild project file. For each of these samples I will cover some interesting parts but for a complete picture please browse the full code of the samples.

All of the samples below assume you have tsc.exe in your system path.

Using a .bat file

This sample demonstrates compiling TypeScript files in place using a .bat filed called compile.bat.

chutzpah.json

{
 "Compile": {
   "Extensions": [".ts"],
   "ExtensionsWithNoOutput": [".d.ts"],
   "Executable": "compile.bat"
  },
 "References": [
   {"Include": "*/src/*.ts", "Exclude": "*/src/*.d.ts" }
 ],
 "Tests": [
   { "Include": "*/test/*.ts", "Exclude": "*/test/*.d.ts" }
 ]
}

 

compile.bat

@echo off
tsc src/code.ts test/test.ts --sourcemap

 

A couple interesting things about this example:

  1. It doesn’t set the outdirectory since the compile.bat command is compiling them in place.
  2. It gives a relative path for compile.bat since it is in the same directory as the chutzpah.json file
  3. It provides .d.ts as the ExtensionsWithNoOutput since although .d.ts are part of the compilation process they produce no output. This helps Chutzpah be smart about when it needs to run the compile executable.

 

Using a .ps1 file

This sample demonstrates compiling TypeScript files to a out directory using a PowerShell file.

chutzpah.json

{
 "Compile": {
   "Extensions": [".ts"],
   "ExtensionsWithNoOutput": [".d.ts"],
   "OutDirectory": "out",
   "Executable": "%powershellexe%",
   "Arguments": "-NoProfile %chutzpahsettingsdir%\\compile.ps1"
  },
 "References": [
   {"Include": "*/src/*.ts", "Exclude": "*/src/*.d.ts" }
 ],
 "Tests": [
   { "Include": "*/test/*.ts", "Exclude": "*/test/*.d.ts" }
 ]
}

 

compile.ps1

tsc src/code.ts test/test.ts --sourcemap --declaration --outDir out

A couple interesting things about this example:

  1. It sets the outdirectory in the chutzpah.json to match the outDir setting in the call to tsc.exe in compile.ps1. This tells Chutzpah where to find the .js files.
  2. It uses the %powershellexe% variable that Chutzpah provides to get the full path to powershell.exe in the executable field.
  3. It uses the %chutzpahsettingsdir% variable in the arguments field to build the full path to the compile.ps1.

 

Using an MSBuild project file

This sample demonstrates compiling TypeScript files to an out directory by executing a target in an MSBuild project file.

chutzpah.json

{
 "Compile": {
   "Extensions": [".ts"],
   "ExtensionsWithNoOutput": [".d.ts"],
   "Executable": "%msbuildexe%",
   "Arguments": "/t:CompileTS ",
   "OutDirectory": "out"
  },
 "References": [
   {"Include": "*/src/*.ts", "Exclude": "*/src/*.d.ts" }
 ],
 "Tests": [
   { "Include": "*/test/*.ts", "Exclude": "*/test/*.d.ts" }
 ]
}

 

compile.proj

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="12.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

 <PropertyGroup>
   <TscExe>tsc</TscExe>
 </PropertyGroup>

 <Target Name="CompileTS">
   <Exec Command="$(TscExe) src/code.ts test/test.ts --declaration --sourcemap --outdir out" />
 </Target>

</Project>

 

A couple interesting things about this example:

  1. The %msbuildexe% variable that Chutzpah provides is used in the executable field to get the full path to msbuild.exe.
  2. The arguments field calls the compile.proj using a specific target (CompileTS). You may notice this command doesn’t mention the project file compile.proj. In this sample it is assuming that is the only project file. If you wanted to be explicit you could change the arguments to: compile.proj /t:CompileTS

 

Future of the legacy TypeScript and CoffeScript integration

With the addition of the compile configuration setting I am deprecating the legacy (that is what I am calling it) existing integrated compilation functionality. For this release the legacy code is still there but it will no longer receive updates (besides upgrading TypeScript to 1.0 when it is released). In a future release I may REMOVE the legacy compilation support so please start the process of upgrading to the new compile setting.

 

Custom test function pattern

When Chutzpah reports test results it also tells you what line they are on. The way this is accomplished is by having a regex for each testing framework which describes where to find the names of your tests. For example for QUnit Chutzpah uses the following:

((?<!\.)\b(?:QUnit\.)?(test|asyncTest)[\t ]*\([\t ]*["'](?<TestName>.*)["'])

This works pretty well but if you use a library which provides a different syntax for specifying tests then this will fail. This results in all tests having no line number information. To fix this problem Chutzpah now offers a TestPattern setting in the chutzpah.json file. This lets you specify your own regex for Chutzpah to use. The requirement for this regex is that it needs to capture the name of the test in a capture group called TestName. For example lets say this is how your test file looks:

function TestClass() {

}

TestClass.prototype.test = function(name, callback) {
    QUnit.test(name, callback);
};

TestClass.prototype.testCase = function (name, callback) {
    QUnit.test(name, callback);
};

TestClass.prototype.registerTests = function(callback) {
    callback.call(this);
};

var testClass = new TestClass();

testClass.registerTests(function() {
    this.test("Pattern 1", function() {
        ok(true, "this test is fine");
    });

    this.testCase("Pattern 2", function () {
        ok(true, "this test is fine");
    });

});

 

Then you can use the following chutzpah.json to get correct line number information.

{
    "Framework": "qunit",
    "TestPattern": "this\\.(testCase|test)\\([\"'](?<TestName>.*)[\"']"
}

 

Require.js BaseUrl support

One of the issues that users reported after the addition of the improved Require.js support in the 3.0 release was that it doesn’t work correctly when you set a custom BaseUrl in the Require.js config or if you set a custom test harness location in the chutzpah.json file.  With this release Chutzpah contains a setting in the chutzpah.json file called AMDBasePath. This setting is a path relative to the settings file directory which should point to the same folder as your require.js baseurl. In addition, Chutzpah will also now correctly work when you use its AMD mode while setting a custom TestHarnessDirectory. To demonstrate the use of these changes I created three new samples.

CustomBaseUrl Sample – This sample demonstrates using Require.js with a baseurl setting along with the Chutzpah AMDBasePath setting.

CustomHarnessLocation Sample – This sample demonstrates placing your tests in a separate folder than your source but placing the generated HTML file at the root of your source.

CustomBaseUrlAndCustomHarnessLocation Sample – This sample shows a combination of both samples above.