Code Contracts and Test-Driven Development

Introduction

This post will be mostly about Microsoft’s Research in Software Engineering (RiSE) group’s Code Contracts. I’ll also cover the role of code contracts in conjunction with test-driven development. There seems to be a debate over the role of test-driven development when using code contracts. I’m going to weigh in on that debate.

In order to demonstrate code contracts, I’m going to return to the Category class that I used as an example in my blog post Advanced Test-Driven Development. This post may also be a little on the longer side.

An Introduction to Code Contracts

What are code contracts? Where did they come from? Why should I care?

The term ‘code contract’ was coined by Bertrand Meyer while he was developing the Eiffel programming language around 1986. In fact, code contracts are a core part of the Eiffel programming language to this day.

Code contracts are a way of specifying pre-conditions, post-conditions for methods and invariants for objects. Pre-conditions are predicates that specify conditions that must be met in order for a method to be in a state that can result in successful execution. Post-conditions are predicates that must be met upon any exit from a method. Invariants are predicates that validate state that must hold at all throughout the lifetime of an object.

Microsoft & Code Contracts

Microsoft’s RiSE team first introduced code contracts to .NET as an extension to the C# language known as Spec#. Spec# was basically a DSL especially for defining code contracts when programming in C#. Fast forward a few years, and shortly after the release of the .NET Framework 3.5, the RiSE team released the successor to Spec#, Code Contracts, in the form of an assembly that you can reference from your project. This assembly replaced the DSL with a native C# language implementation. For the first time in .NET (aside from the Spec# augmented language), programmers could employ code contracts (and not just simple guard conditions at method entry and rudimentary forms of post-conditions). With the release of the .NET Framework 4.0, Code Contracts have found their way into the base class library (BCL) itself. Code contracts can be found in the System.Diagnostics.Contracts namespace. You no longer need to reference an external assembly to employ code contracts.

In order to make full use of code contracts in your .NET projects, you will need to install the necessary tooling from Microsoft’s RiSE Code Contracts site. Installing this tooling installs a new project properties tab for controlling how contracts are incorporated into your assemblies, whether contracts will be enforced at runtime, and/or whether or not the static analyzer will perform an analysis of your code. For the full details, check out the Code Contracts documentation. Furthermore, the tooling installs a code contracts re-writer. The re-writer is the component responsible for weaving the necessary IL into your assemblies in order to enforce the contracts according to the options specified. OK, enough of the history lesson. Let’s get to the fun part: the code.

Using Code Contracts with the Category Class

As I stated in the introduction to this post, I’m going to be reusing the Category class example I went through in my post on advanced TDD. This class represents a category that you might find in some online e-commerce product catalog. I'm going to highlight just a few areas of the Category class as it pertains to implementing code contracts. The first part of the class we’re going to look at is instantiating the class given a name, since the pre- and post-conditions are pretty easy to follow. Also, because of this easiness, it will be easier to introduce how to incorporate unit testing when using Code Contracts.

So, starting with the a category's name: it cannot be null; it must not be the empty string; and it must not consist solely of whitespace. In my advanced TDD post, we enforced these requirements with the following code:

public class Category
{
private string name;

public Category(string name)
{
if (string.IsNullOrWhiteSpace(name))
throw new InvalidCategoryNameException();

this.name = SanitizeName(name);
}
}

The guard clause is known as a pre-condition. This particular form of pre-condition is considered a defensive programming style. If the name parameter is null, the empty string, or only whitespace, an InvalidCategoryNameException exception is thrown. Notice that the constructor has no post-conditions—there is no code to ensure that the private class field name also satisfies these conditions. (In all honesty, it's hard to think of a reason why, if the name parameter meets the pre-conditions, the corresponding class field would not end up being assigned.) Let's see what this same constructor would look like using Code Contracts.

using System;
using System.Diagnostics.CodeContracts;

namespace DDDSportsStore.Domain
{
public class Category
{
private string name;

public Category(string name)
{
Contract.Requires<InvalidCategoryNameException>(
!string.IsNullOrWhiteSpace(name)
);
Contract.Ensures(!string.IsNullOrWhiteSpace(this.name));

this.name = SanitizeName(name);
}
}
}

As you can see, the guard clause was replaced with a call to Contract.Requires<T>(bool precondition). (Important: read the documentation before using the generic form of the Requires method in your release code.) This line of code guarantees that if the name parameter does not meet the specified pre-condition, the specified exception will be thrown. Similarly, we have stated a post-condition: namely that when this constructor exits, the class's name field will also not be null, an empty string, or whitespace. In my advanced TDD post, the constructor ended up having a lot more guard clauses in it for other properties of the Category class. Don't worry, we'll come to those soon.


Code Contract Violations


When a pre-condition, post-condition, or invariant condition is violated, an exception is thrown. The exception is the System.Diagnostics.Contracts.ContractException. The catch is, this exception’s access modifier is internal; so you can’t catch this exception directly. This is actually a very good design decision on the part of the RiSE team, even thought it causes headaches for us developers—especially when it comes to unit testing. Programmers should not be able to circumvent contract violations by catching these exceptions and “handling” them. Contracts are guarantees. Pre-conditions guarantee that, given parameters that satisfy the stated pre-conditions, the method can perform its duties. Similarly, post-conditions guarantee to the caller that upon exit from the method, certain conditions will be true. And invariants guarantee that certain conditions hold throughout the lifetime of the object before and after each method call. So while this is a bit of a pain when it comes to unit testing, this was a very good decision.


Code Contracts and Test-Driven Development


As I alluded to in the introduction to this post, there is a debate that’s been going on in Internet land regarding the roles of code contracts and test-driven development and when and how they are used together. I’ve seen many people giving bad advice in regards to these development practices and I wish to argue the case for my opinion in the next few paragraphs.


What, traditionally, has been the purpose of unit tests (especially those unit tests written as part of a test-driven development effort)? Traditionally, they’re written to ensure that a method does what it’s supposed to do. In fact, there’s a whole style of test-driven development known as behavior driven development (BDD). BDD’s main focus is tests that are named after the behavior that the function is performing. BDD tests strive to be as human readable as possible in an effort to be somewhat of a requirements document for the code base. (I’m over simplifying a bit—I urge you to become familiar with it, as the style has many merits over regular TDD.) But no matter the testing style you use, tests are written to verify that: 1) method parameters are checked to ensure successful execution of a method; 2) the method executes successfully and the proper outputs are returned; and 3) that the state of the object is valid when the method has finished executing (not all tests require this last point to be checked). The first point sounds a lot like pre-conditions—and in a sense, that’s exactly what’s being checked. The second point sounds exactly like post-conditions—and it is. The third point, if part of the test, might be an object invariant (I say might, because what’s being tested in the third point may only be true for that particular case and not a true invariant.)


So now the question is, if I use code contracts, do I need to test pre-conditions and post-conditions? Emphatically, I say YES!! You will find many who disagree with me. Let’s examine the two main arguments against writing tests that exercise code contracts.


You Don’t Write Unit Tests that Exercise Debug Asserts—Why Write Tests That Exercise Your Contracts?


First, I would like to say that this is an invalid statement to begin with. Of course you don’t exercise debug asserts. There’s nothing you can do with them in the testing framework. Your test will fail if a debug assert fails because the program aborts (and therefore, your test runner aborts, too)! So that’s just a ridiculous comparison to make. Furthermore, pre-conditions are not debug asserts. I think this argument comes about because there is a Contract.Assert method in Code Contracts. But this method is not used as a pre- or post-condition. The Contract.Assert method is used as a hint to the static analyzer that, at some point in your code, some condition should hold. That’s it. It’s just a hint to the static analyzer—not a contract. These hints are sometimes needed in order to give the static analyzer a little bit of help when it can’t quite figure out what should hold or not.


Testing Code Contracts Pre-Conditions is Akin to Testing the BCL


This argument is almost plausible; but ultimately, it is flawed. First, it is true that Code Contracts are now a part of the .NET base class library. That’s what makes this argument so plausible. However, when you write tests to test your pre-conditions, the goal is not to test that Code Contracts “does the right thing” when a contract is violated. Indeed, that would be silly. It’s 3rd party code that I have no control over, so why test it? No, what I am testing is that I didn’t accidentally mistype the condition that is checked as part of the pre-condition contract!


Seriously, you must write unit tests for your pre-conditions. While adding code contracts to the Category class used in this blog post, I did mistype a pre-condition specification. The specification was supposed to ensure that when adding a subcategory to a category, the category being added as a subcategory is not null and was not the NullCategory. So when I ran my unit tests and two of them failed (again, I had added the contracts after I had already written the majority of the class in this instance using the standard defensive coding practice of guard clauses at method entry), I was surprised. What happened is that I had accidentally used an || instead of an && in the pre-condition (I probably had copied the guard clause condition; pre-conditions are often the opposite of traditional guard clauses—guard clauses check that something isn’t true, pre-conditions check that something is true). So again I say, YES, you must test your pre-conditions to ensure, not that Code Contracts is behaving, but that you didn’t mistype your pre-condition!


I do find, however, that it’s not practical to test your post-conditions, even though the same possibility of mistyping the post-condition exists. This is because 1) it’s really hard to make your method fail a post-condition on purpose, and 2) chances are, if your post-condition is wrong, a ContractException exception will be thrown. Again, this argument is not water tight—but it is difficult to test post-conditions, and therefore I believe, not worth the effort.


I think that these two arguments are the most frequent reasons given for not testing your code contracts; and I think I have demonstrated why these arguments don’t hold water.


Test-Driven Development with Code Contracts


Now that we have that out of the way, I want to share with you how you can perform test-driven development while using code contracts. This did take me quite a bit of time to figure out. So let’s re-introduce unit tests for the constructor as it's currently implemented above.


The first thing we need to take care of is the fact that when a contract is violated, a private ContractException exception is thrown, and since it’s private, it cannot be caught. So how can we test our pre-conditions? Code contracts provides an event on the static Contract class called ContractFailed. You can subscribe to this event to get access to the contract exception that was thrown. You can also flag the exception as handled from within this event. (So apparently, even thought you can’t catch the exception, you can subscribe to this event and effectively handle it there. Why any good developer would do this is beyond me, after all, why use code contracts in the first place, then? But for unit testing, this is just the hook we need for testing our contracts—but this is the only place where you should ever write this code.)


Now if you remember back to our constructor example, I was using the generic form of the Contract.Requires method that allowed me to specify the exception that is thrown. How this works is, upon a contract violation, code contracts will still throw the System.Diagnostics.Contracts.ContractException exception. But there is a class that is emitted as part of the code contracts re-writer that is able to catch this exception. When the exception is caught, the exception’s InnerException property contains an instance of the exception specified for the generic type parameter. This inner exception is then thrown back to the application. (Read the documentation before using the generic version of Contract.Requires.)


Shown below, I have written a class that subscribes to the Contract.ContractFailed event,  which is meant to serve as a base class for all test classes that will be testing objects written with code contracts.

using System;
using System.Diagnostics.Contracts;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace DDDSportsStore.Domain.Tests
{
// Probably not the best named class for this purpose...
// but good enough.
[TestClass]
public abstract class UnitTestBase
{
private EventHandler<ContractFailedEventArgs> ContractFailedHandler =
new EventHandler<ContractFailedEventArgs>((s, e) => e.SetHandled());

[TestInitialize]
public void HookContractFailed()
{
Contract.ContractFailed += ContractFailedHandler;
}

[TestCleanup]
public void UnhookContractFailed()
{
Contract.ContractFailed -= ContractFailedHandler;
}
}
}

This class has both a setup and a teardown method. It hooks and unhooks an event handler to handle the Contract.ContractFailed event. In the official documentation, the RiSE team show that you should call e.SetUnwind(). If you call that method, your unit tests will stop running the moment a contract is violated. That’s because that method causes another exception to be thrown. Any exception thrown from the ContractFailed event handler winds up causing the application to terminate—which means no other tests will run because the test runner will be terminated. But in this case, we want the exception to be thrown because it verifies that when we pass invalid parameters to the method under test, that the pre-condition contract was specified correctly, and therefore, the correct exception is thrown. So now you can use your test framework’s method for asserting thrown exceptions and the tests will pass. Let’s look at an example test that tests our Category constructor by passing a null name, an empty string name, and a name consisting entirely of whitespace.

[TestClass]
public CategoryTests : UnitTestBase
{
private void AssertCannotCreateCategoryWithName(string name)
{
ExceptionAssert.Throws<InvalidCategoryNameException>(
() => new Category(name)
);
}

[TestMethod]
When_creating_a_category_with_an_invalid_name_then_an_InvalidCategoryNameException_is_thrown()
{
AssertCannotCreateCategoryWithName(null);
AssertCannotCreateCategoryWithName(string.Empty);
AssertCannotCreateCategoryWithName("\n");
}
}

If you look back to my previous post, you will see my implementation of the ExceptionAssert class that I created to handle testing for thrown exceptions. I prefer using this utility class to using the ExpectedExceptionAttribute attribute. Pretty much, that's all there is to incorporating unit tests with code that uses code contracts and maintaining the ability to test that you typed your pre-conditions correctly. If you don't want to use the generic form of the Contract.Requires method (and there may be good reasons for you to do so, see the code contracts documentation for details), then you may need to modify how you test your pre-conditions. But the basic elements remain the same. I'll leave that as an exercise for you.


For the remainder of this blog post, I'm going to skip showing unit tests. As, apart from what I showed above, there's nothing different about my unit tests than what I demonstrated in my previous blog post. For the rest of this post, I want to focus on Code Contracts and incorporating more of its features into your classes.


Code Contracts and Inheritance


The great thing about Code Contracts is that you don’t need to redefine them on all of your sub-types. Once you define them on a base class, they’re defined for all sub-types. In this way, using Code Contracts can help ensure that your code follows the Liskov Substitution Principle (LSP). Because code contracts are inherited, all derived types can be used in place of their base types and clients are guaranteed that all pre-conditions, post-conditions, and invariants of the base type will still hold on all derived types.


This does have some implications for derived types however. Derived types are not allowed to define pre-conditions that are stronger than their base types (in fact, doing so would most likely be a direct violation of the LSP to begin with). It is interesting to note, that while weaker pre-conditions should be allowed, they are not. The RiSE team, in their work with Spec#, found that allowing weaker pre-conditions often provided little value but added a large amount of complexity to the analyzer. So in effect, you cannot even add any pre-conditions to derived types. This is even true for interfaces (after all, interfaces are nothing more than pure, abstract classes for which C# has simply provided syntactic sugar (and built-in compiler constraints) in the form of the interface keyword). It should also be noted that if you don’t specify any pre-condition on a method in a base class, then that method is considered to have the weakest of all pre-conditions: true.


These same rules, however, do not apply to post-conditions. You are allowed to specify stronger post-conditions on derived types. This just provides further guarantees for clients that call those methods on derived types compared to their base class method counterparts.


Code Contracts and Abstract Class Methods and Interfaces


Naturally, the question should arise: how can I implement contracts on abstract class methods and interfaces, neither of which are permitted to have method bodies? The answer is to create a special class that overrides the abstract class methods or implements the interface. Code Contracts provides two attributes, one that adorns the abstract class or interface (ContractClassAttribute), and the other that adorns the contract class itself (ContractClassForAttribute).


Keeping Your Contracts DRY


If you find yourself repeating many contracts over and over again in various methods throughout your objects, you can group them together into methods adorned with the ContractAbbreviatorAttribute attribute and call them directly from methods that require the contracts. Let me show you an example. For any Category that is created, it must have a non-null list of subcategories. It’s parent category must also not be null (if a category has no parent, it should have the NullCategory Category.None assigned as its property value). And of course, a category should not have a null, empty string, or whitespace only name after it’s been instantiated. Here’s how I can “group” my contracts together:

public class Category
{
public static readonly Category None = new NullCategory();

private string name;
private Category parentCategory;
private List<Category> subCategories;

protected Category()
{
subCategories = new List<Category>();
}

public Category(string name) : this(name, None) { }

public Category(string name, Category parent) : this()
{
Contract.Requires<InvalidCategoryNameException>(
!string.IsNullOrWhiteSpace(name)
);
Contract.Requires(parent != null);
EnsureThisCategoryIsValid();

this.name = SanitizeName(name);
if (parentCategory == null)
parentCategory = parent;
else
SetParentCategory(parent);
}

[ContractAbbreviator]
private void EnsureThisCategoryIsValid()
{
// Obviously, separate calls to Ensures or Requires are ANDed together...
Contract.Ensures(subCategories != null);
Contract.Ensures(
this == None ||
!(string.IsNullOrWhiteSpace(name) || parentCategory == null)
);
}
}

Again, while this example is trivial, if you have a lot of methods with the same pre- and/or post-conditions, this can be a valuable tool in keeping your code contracts code DRY.


Specifying Object Invariants


Code contracts has a way of telling the analyzer the various invariants of your objects. Invariants are conditions that must be met by your object after each public method call. In the Category class, for example, the following invariants are true: a category’s name is not null, a category’s parent is not null, and a category’s list of subcategories is not null. These conditions can be checked by placing them within calls to Contract.Invariant inside of a method adorned with the ContractInvariantMethodAttribute attribute as shown below:

[ContractInvariantMethod]
public void CategoryInvariants()
{
Contract.Invariant(Name != null);
Contract.Invariant(Parent != null);
Contract.Invariant(Subcategories != null);
}

Invariants can be especially useful with automatic properties (properties that don't have explicit backing fields that are retrieved/assigned from the property's getters/setters).


Code Contract Helper Methods


Let's say you have some helper method to help you in determining whether certain conditions hold (either for pre- or post-conditions on a method) due to some complex logic. You don't really want all that complex logic in the Contract.Requires or Contract.Ensures method call. So instead, you'll have something like Contract.Ensures(ThisMethodsResultsAreVaild());. In order to use this method without code contracts emitting an error at compile time, you should mark the method as pure using the PureAttribute attribute. This tells code contracts that this method has no visible side-effects and will remove the compiler error. The effects of the PureAttribute attribute are also inherited by derived classes.


Giving the Analyzer a Helping Hand


Determining the various states that the source code could leave the object in at any particular time is a fairly difficult task. You’ll see this is true when you start to use Code Contracts because it will slow down your build times a bit (static analysis runs as a post-build activity). For this reason, Microsoft does suggest that you only run contract checking for a custom build configuration that you would create in addition to your normal Debug and Release builds (e.g., Checked).


Due to the complexity involved in analyzing the source code, sometimes the static analyzer will ask for your help. It does this by writing informational messages to the Error Output window in Visual Studio. Sometimes it tells you exactly what it’s looking for in the form of suggesting appropriate Contract.Requires or Contract.Ensures statements. Other times, it tells you that it’s assuming one thing or another (and suggests that you specify these assumptions for next time).


Code Contracts has two other static methods on the Contract class called Assume and Assert. These are not pre- or post-conditions or even object invariants. In the case of Assume, you’re informing the static analyzer that it should assume the condition specified for the argument to the method is true at that point in the execution of the code. It is merely a hint to the analyzer. On the other hand, Assert tells the static analyzer that some condition must be true at that point in code execution in order for successful execution to continue. If at runtime (for builds where asserts are left in the code, such as your Checked build confiuration) an assertion is violated, your program will terminate.


Conclusion


I've touched upon many of the features of Microsoft's RiSE Code Contracts in this blog post along with a few code illustrations. If you find Code Contracts interesting and an acceptable way of practicing Design by Contract, I encourage you to check out the Code Contracts website and the corresponding documentation. Below, find the complete Category class re-written to include all Code Contracts. Except for having to hook the ContractFailed event in a base class for all my unit tests in order to test that expected exceptions where thrown, the unit tests did not need to change in any other way, and so I have not repeated them here. Until next time, take care.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Diagnostics.Contracts;
using System.Text.RegularExpressions;

namespace DDDSportsStore.Domain
{
[DebuggerDisplay("[Name: {Name}]")]
public partial class Category
{
public static readonly Category None = new NullCategory();
private string name;
private Category parentCategory;
private List subCategories;

protected Category()
{
subCategories = new List();
}

public Category(string name) : this(name, None) { }

private Category(string name, Category parent) : this()
{
Contract.Requires<InvalidCategoryNameException>(
!string.IsNullOrWhiteSpace(name)
);
RequiresCategoryIsNotNull(parent);
EnsureThisCategoryIsValid();

this.name = SanitizeName(name);

if (parentCategory == null)
parentCategory = parent;
else
SetParentCategory(parent);
}

protected virtual int Id { get; set; }

public virtual string Name
{
get { return name; }
set
{
RequiresValidName(value);
name = SanitizeName(value);
}
}

public virtual Category Parent
{
get { return parentCategory; }
set
{
RequiresCategoryIsNotNull(value);
Contract.Assume(parentCategory != null);
SetParentCategory(value);
}
}

public IReadOnlyCollection Subcategories
{
get
{
Contract.Ensures(
Contract.ForAll(
Contract.Result<IReadOnlyCollection<Category>>(),
c => c != null && c != Category.None
)
);
Contract.Assume(subCategories != null);

return subCategories;
}
}

/// <summary>
/// Creates a new <see cref="Category"/> as a subcategory of the
/// current instance.
/// </summary>
/// <param name="name">The name of the new category.</param>
/// <exception cref="InvalidOperationException">
/// Thrown if the current instance is <see cref="Category.None"/>
/// </exception>
/// <exception cref="InvalidCategoryNameException">
/// Thrown if the <paramref name="name"/> is <c>null</c>,
/// the empty string, or consists entirely of whitespace.
/// </exception>
public virtual Category AddSubcategory(string name)
{
RequiresValidName(name);

Category subCategory = new Category(name, this);
AddSubcategory(subCategory);
return subCategory;
}

private void AddSubcategory(Category c)
{
Contract.Requires<InvalidOperationException>(
this != Category.None && !(c == None || c == null)
);
Contract.Requires<InvalidOperationException>(
!(
Contract.Exists<Category>(
subCategories, sc => sc.Name == c.Name
)
)
);

this.subCategories.Add(c);
}

public bool Equals(Category other)
{
if ((object)other == null)
return false;

return this.ToString() == other.ToString();
}

private string SanitizeName(string proposedName)
{
Contract.Assume(!string.IsNullOrWhiteSpace(proposedName));
return Regex.Replace(proposedName, @"\s+", " ");
}

private void SetParentCategory(Category c)
{
RequiresCategoryIsNotNull(c);
RequiresCategoryIsNotNull(parentCategory);
Contract.Requires<InvalidOperationException>(
c == Category.None || !this.HasDescendant(c),
"Attempted to set the parent of this category to " +
"one of this category's descendants."
);

parentCategory = c;
if (parentCategory != None)
parentCategory.AddSubcategory(this);
}

#region System.Object Overrides

public override bool Equals(object obj)
{
if (obj == null)
return false;

Category c = obj as Category;
if ((object)c == null)
return false;

return this.ToString() == c.ToString();
}

public override int GetHashCode()
{
return this.ToString().GetHashCode();
}

public override string ToString()
{
return Parent == Category.None ?
Name : string.Concat(Parent.ToString(), "/", Name);
}

#endregion

#region Operator Overrides

public static bool operator ==(Category a, Category b)
{
if (object.ReferenceEquals(a, b))
return true;

if ((object)a == null || (object)b == null)
return false;

return a.ToString() == b.ToString();
}

public static bool operator !=(Category a, Category b)
{
return !(a == b);
}

#endregion

#region Code Contracts Helper Methods

[ContractInvariantMethod]
private void CategoryInvariants()
{
Contract.Invariant(Name != null);
Contract.Invariant(Parent != null);
Contract.Invariant(Subcategories != null);
}

[Pure]
private bool HasDescendant(Category c)
{
RequiresCategoryIsNotNull(c);

bool result = false;

if (this == c)
result = true;
else if (c.Parent != None)
result = this.HasDescendant(c.Parent);

return result;
}

[ContractAbbreviator]
private void EnsureThisCategoryIsValid()
{
Contract.Ensures(subCategories != null);
Contract.Ensures(this == None ||
!(string.IsNullOrWhiteSpace(name) || parentCategory == null));
}

[ContractAbbreviator]
private void RequiresValidName(string name)
{
Contract.Requires<InvalidCategoryNameException>(
this == None || !string.IsNullOrWhiteSpace(name)
);
}

[ContractAbbreviator]
private void RequiresCategoryIsNotNull(Category c)
{
Contract.Requires(c != null);
}

#endregion
}
}

Mighty Moose and Contextual (a/k/a Hierarchical) TDD

Introduction

In my last few blog posts, I introduced Mighty Moose and advanced TDD using nested, hierarchical context classes. If you have started using Mighty Moose and tried your hand at contextual TDD (that’s my new name for it, it seems to fit), also known as hierarchical testing, you may have noticed a problem.

The Test Runner Behind Mighty Moose

Mighty Moose is compatible with the following testing frameworks:

  • MS Test
  • NUnit
  • xUnit
  • MbUnit
  • SimpleTest
  • MSpec

What may be surprising is that Mighty Moose does not necessarily use the native test runners for these frameworks. Mighty Moose has its own test runner called AutoTest.Net. From what I can tell, it appears that Mighty Moose implements a bunch of Adapters that it uses to interact with the various testing frameworks it supports. The Adapters don’t make use of the native test framework engines themselves. The Adapters contain their own implementation. Unfortunately, it appears the execution of the Adapters may not be 100% compatible with the test framework you’re using to write your tests.

So, imagine my surprise when I refactored all of my tests to use nested, hierarchical classes in order to constrain the boundaries of the various setups I would need (see my last blog post on advanced TDD). Mighty Moose started reporting that all of my abstract base class tests were broken! (The actual error was that AutoTest.Net was unable to instantiate an abstract class. No, really?) I ran my tests in Visual Studio using the CTRL+R, T shortcut. This invokes the native MS Test test runner. I pop on over to the Test Explorer window, and what do I find? All of my tests pass.

On the one hand, having these Adapters are great; you don’t have to learn a different testing framework API in order to use Mighty Moose. On the other hand, my confidence in Mighty Moose is now lower than I would otherwise like because the output is not 100% compatible with the native test runner. How can I be sure that in all instances the tests are actually passing or failing (or even that AutoTest.Net is reporting the right result, for that matter)?

AutoTest.Net is Broken

So obviously, there’s a bug in AutoTest.Net, which is what Mighty Moose uses to run your tests; and it’s a pretty big one. AutoTest.Net recognizes that you have various test classes (a là the TestClassAttribute attribute). And so it tries to instantiate any class decorated with that attribute without checking if it’s valid to instantiate said class. But wait, it gets worse. It finds the nested inner class (which derives from the abstract base classes) and only runs the tests found in the concrete derived class (none of the inherited base class tests run with the derived class).

Is there a way around this? Well, yes, but you should be careful. The way to avoid this problem with Mighty Moose is to not make the base classes abstract. However, this will lead to the base class tests executing for every class which derives from that base class. And this won’t be true just for AutoTest.Net, but also for MS Test. (I don’t know about NUnit and xUnit since I’ve never used them.)

So what, you say? That’s ok, as long as Mighty Moose works. Well, consider this. You have 100 tests in your base class, and 100 tests in a derived class. How many tests will get executed? 300 tests. 100 for the base class, 100 for the derived class’s inherited base class methods, and 100 tests contained in the derived class itself. OK, so you don’t have 100 tests in a single test class. However, in a production system, you may have upwards of 5000 tests. (And that’s just when they’re running once!) So now multiply a good chunk of those tests and pretty soon you’re pushing 15000 tests. (And keep in mind that that's assuming that you're only nesting one level. If you nest two levels deep, now you have an explosion of the possible number of tests that will execute.) It’s wasteful of time and computing resources.

Conclusion

If you have a really small project and don’t mind tests being executed more than once, by all means, go for it. But it’s really not a good of a solution. AutoTest.Net should take into account whether or not a class is abstract and ignore it if it’s marked with the TestClassAttribute attribute. Furthermore, AutoTest.Net should look to see if a test class is a derived class and ensure that the derived class’s base class hierarchy is properly instantiated and that all inherited members are executed as part of the derived class.

Hopefully the authors of AutoTest.Net will fix this bug soon. Until then, if you have a small project, go for it. Otherwise, it’s probably best not to use Mighty Moose. If you really need a continuous test runner, check out NCrunch. I’ve been using it at work and I really like that continuous test runner, too. (Oh, and as far as the cost for NCrunch, while I don’t want to pay for it for personal use at this time, it’s really not that expensive; and if things were different as far as the level of development I do personally, I’d definitely pay for it.)

Mighty Moose a/k/a ContinuousTests

Introduction

At my current place of employment, I was introduced to NCrunch, a continuous testing solution for .NET and Visual Studio. It’s a great tool that I really enjoy using at work. However, for at home use, I can’t justify paying their asking price for a personal license. Don’t get me wrong, I really love the tool, and if I were doing some serious development on my own (i.e. creating products for sale as a side job), I’d probably pay for it, because it’s fast and shows exactly where an exception occurred or a test failed.

So I really like this concept of continuous testing and went on an Internet search to see what was available out there for free. I wanted something that would run in Visual Studio 2012/2013. A lot of the “free” tools use external test runners which display their output in either the browser (as an XML/HTML page) or in a console window. I don’t like those tools because they interrupt my context, having to switch between Visual Studio and, say, a console window. Plus, then I have to go digging for the file and line number when a test fails.

Enter Mighty Moose a/k/a Continuous Tests

During my search, I found Mighty Moose a/k/a ContinuousTests, by Greg Young. Mighty Moose has a slightly different philosophy about continuous testing. Instead of a code coverage margin in Visual Studio, they have a risk margin. The risk margin shows a colored circle next to a method under test with a number inside. The number inside tells you how many tests are covering the method. The color: red, green, or yellow, indicates the risk of changing that method.

How is risk calculated? It’s based on how close the method was called from the test. According to Greg, while code coverage tells you whether or not the code was executed by a test, that information alone is not enough to classify how risky it is to change the code. What if the code that’s “covered” was called from a method 10 frames down the call stack? By changing that code, you could be affecting not just the method under test, but all 8 other methods in the call stack.

In addition to the risk margin, Mighty Moose contains graph “overlay” graphics (for those of us who can’t afford Visual Studio Ultimate ;) ) that show the call graph of all methods called during the test. This can help you gauge “how far away” a method was invoked from the test, and in general, just help you see the entire call graph, which in itself, could be enlightening.

Getting Mighty Moose to Work with Visual Studio 2013

Mighty Moose is currently at version 1.0.47. It officially only supports Visual Studio 2008, 2010, and 2012. However, there’s an excellent question over on Stack Overflow that I contributed to which shows you how to get Mighty Moose to work with Visual Studio 2013.

AllMargins 2012 v2.8.x Releases

On 1 September 2013 I released version 2.8 of the AllMargins 2012 Visual Studio extension. Since that release, there have been two additional "dot" releases. Briefly, here's what v2.8.x has to offer:

  • Visual Studio 2013 RC and RTM support

  • All the registry value names for the extension's settings have been renamed to prevent naming conflicts with the Productivity Power Tools for Visual Studio 2012 and beyond (more on that below)

  • Fixed the registry value name for method separator line color

  • Fixed the StructureAdornment extension so that method separator lines are drawn using the specified color

  • Added C/C++ union support

Extension Option Registry Value Name Clashes

David Pugh (MSFT) ported a lot (but not all) of the functionality of the original AllMargins extension into the Productivity Power Tools for Visual Studio 2012. The extension option registry value names were left the same. So, when I re-released the AllMargins extension as AllMargins2012, the two extensions would step over each other in terms of their settings for the common extensions. Starting with v2.8, the registry value names have changed to prevent this. As a result, those who installed v2.7 and subsequently customized the extension options by editing the registry, upon installing v2.8, will see that the extension is using the default values for all settings—it appears as though your customizations were changed. They are not. They are still present in the registry. However, new settings are now used by the AllMargins 2012 extension. You need only copy the setting value from the v2.7 registry value to the new v2.8 registry value. I apologize for the inconvenience.

For more details about what the various registry settings are, and their old and new names, see the description tab for the extension at the Visual Studio Gallery.

What to expect in the next major release

Many have spoken, including myself, about what you would like to see in the next major release. Work is underway; here is a quick rundown of what to expect (of course, this is subject to change—nothing is set in stone unless otherwise noted):

  • The ability to set the extension's options from the Visual Studio Tools|Options dialog. This is a top priority, non-negotiable item for the next major release. I have been wanting this very feature myself since David Pugh originally wrote this extension. Now that I'm maintaining this code base, this task falls on me.

  • The ability to specify the line widths displayed in the editor for the StructureAdornment extension

  • Add an option to show/hide the StructureMargin preview tool tip

The items above are what I can reasonably expect to fit into the next major release, though only the first item is guaranteed to make it through. Whatever doesn't make it through will be in a follow-up release. One item that is a planned feature for a future release is the availability to select a color theme for the StructureMargin and StructureAdornment extensions. As many have noted, and I agree, the "default" colors don't work well with Visual Studio 2012's dark theme. In addition, I plan on changing the default color for the class structure adornment from black to some other color, because with the Visual Studio light theme the black line can cause exclamation points to appear to not be present in C# and C/C++ preprocessor statements (e.g. #define !SOME_PREPROCESSOR_VAR), as one user pointed out to me. Black was the original color used by David Pugh—but due to this potential problem, it's best if this color is changed.

If there are any other features you'd like to see, please feel free to leave a comment here on my blog or on the Q AND A tab at Visual Studio Gallery. Thanks again for all of your comments, feedback, suggestions, and ratings!

Announcing the Release of AllMargins for Visual Studio 2012

In August of last year, I wrote about upgrading Visual Studio 2010 extensions so that they will work with Visual Studio 2012. In particular, I discussed upgrading the AllMargins Visual Studio extension, which was quite popular.

For some time now, David Pugh of Microsoft has pulled this extension off of Visual Studio Gallery. It is only available if you happened to have downloaded it and saved it to your hard drive or have downloaded the source code (which you can find the link to in the article I wrote back in August). The problem with the source code is that it's a few versions behind the last release that was available on Visual Studio Gallery and contains a few bugs and is missing support for things like C#'s class initializer syntax.

The extension was pulled because David ported most of the functionality into the Visual Studio Productivity Power Tools. However, I'm not a big fan of the particular implementation of the enhanced scrollbar in VSPPT and prefer the AllMargins scrollbar instead.

Today, I'm announcing general availability once again of the AllMargins 2012 extension on Visual Studio Gallery. I modified the available source code, bringing it up to parity with version 2.6, the last available release on Visual Studio Gallery. In addition, I modified the extension so that it will install in Visual Studio 2012. The extension is now at version 2.7.

Thanks to David Pugh at Microsoft for giving the Visual Studio community this wonderful extension and releasing the source code for it a while back!

As I have time, I hope to make a few additions to the extension. Some planned enhancements include providing the ability to change various colors used for drawing the lines in code files delineating various code blocks or for drawing various markers in the overview margin (e.g. drawing a mark for all occurrences in a document of a word under the cursor in the overview margin) right from the Visual Studio Options dialog.

A few notes about AllMargins 2012. First, because the original extension is linked to David Pugh's Windows Live ID, I had to change the identifier of the extension. (This is good, afterall, you wouldn't want Microsoft to allow people to hijack author's extensions possibly replacing them with malicious code, now would you?!) Since the ID, well, identifies the extension, it's essentially a different extension. So I renamed it AllMargins 2012 while I was at it. What this means is that if you want to use this extension in Visual Studio 2010, you must uninstall David Pugh's original AllMargins extension before installing this extension. I don't know what would happen if you have both installed (Double overview scrollbars? Crash Visual Studio?)

I hope that you will continue to find this extension as useful as I have! The Read Me for the top-level AllMargins 2012 extension is presented below.

AllMargins 2012

Introduction:
This is a compilation of the OverviewMargin and several other extensions that use the OverviewMargin. The OverviewMargin shows
a margin on the right side of the editor that logically maps to the entire file (similar the the vertical scroll bar). Unlike
the scroll bar, it maps to the entire file and can contain other margins that provide more information about the file.


Included extensions:
CaretMargin
ErrorsToMarks
MarkersToMarks
OverviewMarginImpl
StructureAdornment
StructureMargin

BlockTagger The definition of an API to get the structure of a code file.
BlockTaggerImpl An implementation of the BlockTaggerAPI for C/C# and VB files.

OverviewMargin The definition of an API for creating margins that map to the entire file.

SettingsStore The definition of an API to load and save editor options across sessions.
SettingsStoreImpl An implementation of the SettingStore API that uses IVsSettingsStore to access the system registry.

Usage:
See other extensions.

Options:
None.

Notes:
The full source for this extension is at http://code.msdn.microsoft.com/OverviewMargin/Release/ProjectReleases.aspx?ReleaseId=3957
The source code is not yet available for download.

History:
v1.0 David Pugh 2/26/2010
Initial release
V1.1 David Pugh 3/4/2010
Updated included extensions.
V1.2 David Pugh 4/19/2010
Updated included extensions.
V1.3 David Pugh 4/20/2010
Updated included extensions.
V1.4 David Pugh 4/22/2010
Updated included extensions.
V1.5 David Pugh 4/27/2010
Updated included extensions.
V1.6 David Pugh 4/28/2010
Updated included extensions to fix VB parser error.
V1.7 David Pugh 4/28/2010
Updated version number to work around install issue.
V1.8 David Pugh 4/28/2010
More VB parse fixes (Public, REM, Interface, Structure).
V1.9 David Pugh 4/28/2010
Updated CaretMargin, VB, C# and C parsers.
V2.0 David Pugh 5/03/2010
Bumped version number to fix installation problem.
V2.1 David Pugh 6/10/2010
Updated OverviewMargin, BlockTagger, StructureAdornment & StructureMargin.
Changed namespaces and DLL names to Microsoft.VisualStudio.Extensions.....
V2.2 David Pugh 6/29/2010
Updated StructureAdornment.
V2.3 David Pugh 6/30/2010
Picked up fix for VB parser, tweaks for structure adornment & margin.
V2.4 David Pugh 7/08/2010
Updated StructureAdornment.
V2.5 David Pugh 7/12/2010
Updated included BlockTaggerImpl to handle C/C# array initializers.
V2.6 David Pugh 7/31/2010
Updated included CaretMargin to fix problem with margin not updating correctly.
Updated included OverviewMarginImpl to fix scrollbar mapping problem.
V2.7 Craig E. Shea 1/16/2013
Added Visual Studio 2012 support.
Removed the attribute MaxVersion="4.0" on the SupportedFrameworkRuntimeEdition element.
Changed the author and description to indicate who is now responsible for the extension and that it is a re-release of a previously available extension.

Updating a Visual Studio 2010 Extension for Visual Studio 2012



UPDATE - 16 January 2013

The contents of this article have been updated to correct a problem that existed at the time of publication. In addition, I have uploaded this updated Visual Studio extension to the Visual Studio Gallery.

Hello everyone. It's been way too long. It seems I'm always doing this and that and never have time to write down anything, anywhere—whether that be on this blog, or on Facebook. But today, I have something special for those of you who have that favorite Visual Studio 2010 Extension that they just can't live without, but it hasn't been updated for Visual Studio 2012.

In order to update the extension, you will need access to the extension's source code; so these instructions will have limited impact for most of you out there unless you can get your hands on the source code for that favorite extension.

Fortunately for me, the author of my favorite extension, AllMargins, did post his source code, especially since this extension is no longer available from the Visual Studio Gallery.

Install the Visual Studio 2012 and 2012 SDKs

Get Visual Studio Professional 2012 with MSDN from the Microsoft Store

In order to author extensions for Visual Studio, one must have the Visual Studio SDK installed. The same is true for modifying them. If you don't already have it installed, go out and download it and install it.

Prepare the Source Code for Visual Studio 2012

If you want to maintain backward compatibility for the extension, it would be best not to open the original Visual Studio 2010 solution in Visual Studio 2012. That's because Visual Studio Extension projects for Visual Studio 2010 are not compatible with Visual Studio 2012. Well, that's not exactly true. The real truth is that the VSIX manifest XML schema has changed between Visual Studio 2010 and Visual Studio 2012. The old schema still works for Visual Studio 2012, but Microsoft would have you use the new schema going forward. The problem is, if you use the new schema, you won't be able to use the extension in Visual Studio 2010. Don't worry though, Visual Studio 2012 will not upgrade the schema, just the project file(s); but in doing so, the projects will no longer open in Visual Studio 2010.

Why should you care? Well, for one, testing. You can't simply instruct VsixInstaller to install the extension to an experimental instance of Visual Studio, unfortunately. That really makes it hard to test the extension for more than one version of Visual Studio. Even so, you really should be testing your extension on a virtual machine with the installed version of Visual Studio to test it with and which does not have the Visual Studio SDK installed. If the Visual Studio SDK is installed on your test machine, your extension will always work but, you may find that it doesn't work users who don't have the Visual Studio SDK installed. Your extension should not explicitly require the Visual Studio SDK be installed (unless you're writing a Visual Studio Extension project template! Lol...).

If you want to be able to develop and test in both Visual Studio 2010 and Visual Studio 2012 on the same machine, here's my suggestion.

  1. Create a new empty solution called <Your Extension>.2012 in the same place as the original solution.

  2. Open Windows Explorer to the location of one of the projects that are in the Visual Studio 2010 solution.
    1. Select the project file and copy and paste it into the same folder. The file will be renamed 'Copy of <Your Project File>.csproj' (or .vbproj, if you use Visual Basic). Rename the file to <Your Project File>.2012.csproj.

    2. Back in Visual Studio 2012, add <Your Project File>.csproj to your solution.


  3. Add all the files that are part of the Visual Studio 2010 project to the newly created Visual Studio 2012 project.

  4. Repeat these steps for each of the other projects that were part of the original Visual Studio 2010 solution.

Now you can work on the source code in both Visual Studio 2010 and Visual Studio 2012, opening the respective solution for the version of Visual Studio you're using to make changes. When you build the Visual Studio 2012 solution, it will be deployed to the Visual Studio 2012 experimental instance. The same is true for 2010, the extension will be deployed to the Visual Studio 2010 experimental instance. Also, it allows you to work on a single code-base that should work in both environments so long as you don't use features exclusive to Visual Studio 2012.

While upgrading this extension to work with Visual Studio 2012, I discovered that there is an enumeration value in the VSIX XML schema for specifying the Express editions of Visual Studio. However, the only "extensions" that Express SKUs support are project/item template extensions.

We're now ready to start modifying the various files to make this extension compatible with Visual Studio 2012!

Update the *.vsixmanifest file

The first order of business in updating the extension is to modify the *.vsixmanifest file. Here's an example of one of the files from the AllMargins.vsix extension:

<?xml version="1.0" encoding="utf-8"?>
<Vsix xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Version="1.0.0" xmlns="http://schemas.microsoft.com/developer/vsx-schema/2010">
<Identifier Id="SettingsStore.dpugh.ce3ce74a-4966-4a66-b9ad-a3adf4f9e940">
<Name>SettingsStore</Name>
<Author>dpugh</Author>
<Version>1.2</Version>
<Description xml:space="preserve">This defines an API for persisting IEditorOptions across sessions.</Description>
<Locale>1033</Locale>
<MoreInfoUrl>http://code.msdn.microsoft.com/OverviewMargin/Release/ProjectReleases.aspx?ReleaseId=3957</MoreInfoUrl>
<SupportedProducts>
<VisualStudio Version="10.0">
<Edition>Ultimate</Edition>
<Edition>Premium</Edition>
<Edition>Pro</Edition>
</VisualStudio>
</SupportedProducts>
<SupportedFrameworkRuntimeEdition MinVersion="4.0" MaxVersion="4.0" />
</Identifier>
<References />
<Content>
<MefComponent>|%CurrentProject%|

</Content>
</Vsix>

You need to update the SupportedProducts and SupportedFrameworkRuntimeEdition elements as follows:

<SupportedProducts>
<VisualStudio Version="10.0">
<Edition>Utimate</Edition>
<Edition>Premium</Edition>
<Edition>Pro</Edition>
</VisualStudio>
<VisualStudio Version="11.0">
<Edition>Utimate</Edition>
<Edition>Premium</Edition>
<Edition>Pro</Edition>
</VisualStudio>
</SupportedProducts>
<SupportedFrameworkRuntimeEdition MinVersion="4.0" />

By removing the MaxVersion attribute for the SupportedFrameworkRuntimeEdition element, we're telling Visual Studio that so long as the machine supports the .NET Framework 4.0, the extension can be used.

Also for good measure, I updated the Version value (e.g. 1.3) and updated the Description element noting that I changed the VSIX package and it is not the original. As an aside, every time you make a change to the extension that you wish to distribute, you must increment the value for the Version element.

Now I ran across a tricky problem with updating the AllMargins extension. This extension has several "sub-extensions" that are contained within the AllMargins extension itself. And some of these "sub-extensions" have yet other "sub-extensions". The long and short is, during testing, Visual Studio had a problem locating some of the required DLLs.

What I found out is that some of the sub-extension projects contained an additional file—a *.pkgdef file. This is an older file that was used with VSPackages back in the VSI days of Visual Studio 2005/2008. This file is not much unlike a *.reg file, though with a slightly different syntax.

This file for the SettingsStore sub-extension (for which the VSIX manifest file was shown above) looks like this:

[$RootKey$\BindingPaths\{8ADD56CC-E512-4CB0-AF58-33ECCC0B4912}]
"$PackageFolder$"=""

This creates the GUID under HKCU\Software\Microsoft\VisualStudio\<Maj. Version Num>_Config\BindingPaths key. $PackageFolder$ expands to the folder where the extension was installed by VSIXInstaller.

Apparently, in Visual Studio 2010, just placing this file within your VSIX package was enough for Visual Studio to go ahead and create the requisite registry keys and values. However, this does not work with Visual Studio 2012 and attempts to use extensions relying on this extension result in a System.IO.FileNotFoundException. In order to resolve this issue, you must place some additional content in your VSIX manifest file as shown below.

<References>
<Reference Id="Microsoft.VisualStudio.MPF" MinVersion="10.0" />
<Name>Visual Studio MPF</Name>
</Reference>
<!-- Any other references you may have -->
</References>
<Content>
<MefComponent>|%CurrentProject%|</MefComponent><!-- Expands to the assembly name -->
<VsPackage>SettingsStore.pkgdef</VsPackage><!-- Not required by VS2010, but is required by VS2012 in order for the file to be processed -->
</Content>

With all of that out of the way, save the file and we're done with the *.vsixmanifest file. If there are several VSIX extensions that make up the extension you're updating (as is the case for AllMargins.vsix), you will need to do this for each *.vsixmanifest file in the solution.

At this point, at the original date of publication (22 August 2012), I had you retarget the projects in the solution and replace some DLL references. If you want to maintain compatibility with Visual Studio 2010, this is not necessary. So I have removed that section of the original blog entry.

Recompile a Debug Build

You should now recompile the solution using a Debug build. This will automatically install the extension(s) into the Visual Studio experimental instance(i.e. it won't muck up your Visual Studio that you use for everyday coding). Make sure that everything works correctly. If all is well, you're ready for the final step.

Recompile in Release Mode

If your testing above went OK, then change the build mode from Debug to Release and recompile. Then simply navigate to the bin\Release directory under the project folder for the extension and double-click the VSIX file to install it.

Now, in this case, I was not required to change a single line of code. This may not be the case for every extension you wist to upgrade. YMMV.

Enjoy using Visual Studio 2012

That's it! Enjoy using Visual Studio 2012 with your beloved extension.

Put feedlink markup here.