I have wrestled with a problem for days The problem was that I didn’t get coverage for all assemblies that were tested. No errors, no exceptions, no problem whatsoever in the builds and tests. But no matter what I did, the tests did not produce the expected coverage. NCover  and NUnit-console just spits out a coverage report with large holes in it.

As it turns out, all boils down to how NUnit treats the assemblies and the program database files (PDB) while testing. And avoiding shadow copies while executing tests on the build server.

Some background

A couple of weeks ago we made the move from CC.NET to TeamCity. At the same time we restructured the solution folders, having problem with dependencies on builds and that scripts took forever to finish. So to speed up the feedback loop for the day-to-day development we needed the new structure.

Setting up the new structure took about one day of work and creating the initial build and test loop for development another couple of days. This without any build scripts at all. The deployment took a bit longer and involved a few MSBuild scripts and it works well now. Also in this is a setup for executing selenium tests, this took some more scripting but finally it’s in place as well.

The problem

The final piece of script was the coverage. Unfortunately there are no integrated support for NCover in TeamCity so you have to revert to good (or bad) old MSBuild scripting. As there a few long running tests in this soup we opted for running coverage at nightly builds. The dev coverage can be executed on the developer machine so there's no problem with this.

So after setting up a converge build for each of the solutions I found that it never produced the same result for all solutions. With no apparent problem, assemblies was omitted from the result. And sometimes it worked as expected. After more analysis it turns out that when one solution has been built it affects the coverage of other builds. More than was explained by the interface points between the solutions.

It got even worse, I did a build for project A, checked the coverage for A, then built project B and the result for project A changed. Checking the files used for instrumenting project A I could not see one single difference down to the timestamp and version for each and every file of the A build.

The solution

Finally I sat down with FileMon from SysInternals and did a check on one specific assembly used in the test to get a grip on what was tested and where it came from. Hmm, what is all this shadow copy used by NUnit? What happens here?

Well it turns out that NUnit copies assemblies to a temporary folder while executing the unit tests, this to allow the user to continue develop while the tests are executed. Well, this is not the problem on the build server. In the coverage run that I created the assemblies built are copied to an analysis folder and the tests are executed there. And the simple and working solution is to add the /noshadow option to the NUnit command. After checking with FileMon and the resulting coverage I finally got this last nagging problem solved.

After the various coverage results are merged with coverage explorer, it is now published with the working builds on our demo server. Automatic build configuration finally done. And less copying of files while testing might even shave a few seconds of build times.

By the way, now that I know exactly the problem and the solution, it was easy enough to find a blog post concerning NCover problems, that explains it all.

// Håkan Reis

For a while know I have been using CruiseControl.NET for my Continuous Integration setup. It works OK on most occasions, but it seems to get old, no new version for since June last year. I want and expect better interaction and setup experience from a modern tool today.

So along comes TeamCity from JetBrains as a promising new tool, free for three build agents, 20 users and 20 different builds. And for a couple of weeks I have been test driving it. The system itself is built as two parts, the Continuous Integration server and the Build Agents. These need not to be on the same system and you can mix and match Windows and Linux with .NET and Java. It all depends on the build agents. For the test setup I used a server with TeamCity and a build agent on the same machine. The setup is really straightforward, just run the installer and answer a few simple questions. Next up is setting up your project and builds.

There are a few different scenarios I have been working with and to start with the most simple:

Scenario one - build and test:

• Subversion
• Visual studio 2008 solution
• NUnit 2.4.6 unit testing

Well, it couldn't get much easier than this when installing. There are seven steps for each build setup, but not all has to be used. For the basic setup you just point to your subversion repository and select a project folder to checkout into.

Then you select the solution files, set the unit test to the correct version and specify the test assemblies for example **/*.test.dll. Don't forget to unselect the tests in the */obj/ folders, there is a special setting for this as well **/obj/*.test.dll would do the trick.

Last thing was to set the build triggers, I just went along with 60s delayed check in build, then you can also trigger time scheduled builds (good for integration testing and deployment).

Actually it is that simple, no bloated config files to go through. Just make sure the correct version of .NET framework is in place. I had an early beta of 3.5 in place and it took me a while to figure that out.

So for the client then? Is there a tray icon and stuff? Sure is, on the server, set up the users that should access TeamCity, and the access level. When you login you have your settings, and there you will find plugins for Eclipse and Visual Studio (will link to the code from the site and show the tests and your modifications) as well as a tray icon.

What's next?

The second scenario, build, test, and coverage was trickier, when adding the coverage part. Being on .NET there aren't that many free coverage tools. The only viable one is the old NCover 1.5.8 version. This is sad, $149 is a bit steep but it's ok in the right project. But I certainly don't want to pony up $299.00 just to get 64bit support. Supposedly the pro version don't support 64bit. There is a coverage tool called PartCover but it seems to be dead since August 2007, and there is too little information about it.

The third scenario was one that I didn't quite get around to. This was to use the build in Visual Studio Team System Developer support with Microsoft test and coverage. These are good when you run it in your environment and it would be great to have it running in TeamCity.

I will get to the second and third scenarios in upcoming blog posts.

A few weeks back I blogged about Unit testing event handlers using InternalsVisibleTo. Today I run across a, to put it mild, annoying problem that took far to long to solve. The project I'm working on has come to a point where the user interface has to go through a metamorphoses. I'm redoing the application in full WPF (for speed and convenience I will keep a hosted property grid, but that's another story).

When added a user control to the application I started to get intermittent errors. I mix the development with Blend and Visual Studio 2005. It would always build in both environments but from time to time I kept getting these pesky exceptions. This went on for quite some time. As it started when I introduced the custom controls I began my quest there. I recreated the controls and it worked for a while an the it returned. This was driving me crazy. The exception looks like this:

Could not load file or assembly '...., Version=...., Culture=neutral, PublicKeyToken=......' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)  Error in markup file '....;component/Window.xaml' Line 2 Position 5.

It usually worked when I removed all for a full rebuild but not always and sometimes a restart seemed to do the trick. But not always, and there is no way to work like that. I was ready to give up when I remembered a similar hard to track down problem. There is some kind of snag or bug a bug when you sign the assembly and at the same time leave the assembly version in the assembly info to automatic increment. For example, the value:

[assembly: AssemblyVersion("1.0.0.*")]

will give you a terrible headache. So make sure you set this to a constant value when you sign your assemblies. Something like this should work:

[assembly: AssemblyVersion("1.0.0.0")]

I hope I at least saved someone a few hours of frustration.

I always work close to the UI. Actually, most of the time elbow deep in the GUI. But I also love testing. One thing you don't want to do, is drive you GUI for unit testing purpose (there are other uses for that but not unit testing...trust me on that). So it would be great to fire events at will, or in the test code but that's not an option either. To much trouble.

Another way is to just open up the event handlers to the public and execute the code and that, in a testing sense feels a lot better. But again, event handlers is not a good thing to open up to the public, especially when you are building libraries.

I remember Marcus Widerbeg and I had a few discussions around this, how to maintain testability but retain protection. I think the following can develop further but It will do for most scenarios.

There are an attribute that actually seems to be made for this scenario. InternalsVisibleTo... This will present internal methods and fields to a specific strong named and signed assembly, that should be the unit test assembly. It's quite simple. Just change the protection from private to internal. Sign your test assembly and set the attribute in the AssemblyInfo.cs, together with the public key for your test assembly, like this:

[assembly: InternalsVisibleTo("Haqwin.Gui.UnitTests, PublicKey=00...cb")]

To find the public key, just build your solution and run the sn.exe tool like this:

sn -Tp Haqwin.Gui.UnitTests.dll

So now I can use this to unit test most of my UI code as well as automate the Function and Integration tests. Driving them almost as close as executing the actual interface.  Here is an example of testing the EventHandler for a button:

[Test]

internal void NavigateMetaDataEditor()

{

    Assert.AreEqual(DockStyle.Fill, _InitializeMetaData.Dock);

    Assert.AreEqual(DockStyle.None, _EditMetaDataControl.Dock);

    _InitializeMetaData.NextControl = _EditMetaDataControl;

    _InitializeMetaData.NextControlButton_Click(this, new EventArgs());

    Assert.AreEqual(DockStyle.None, _InitializeMetaData.Dock);

    Assert.AreEqual(DockStyle.Fill, _EditMetaDataControl.Dock);

}

The other solution to this problem, is to use reflection to get to the private methods but I'll save that for another post