I have wrestled with a problem for days The problem was that I didn’t get coverage for all assemblies that were tested. No errors, no exceptions, no problem whatsoever in the builds and tests. But no matter what I did, the tests did not produce the expected coverage. NCover  and NUnit-console just spits out a coverage report with large holes in it.

As it turns out, all boils down to how NUnit treats the assemblies and the program database files (PDB) while testing. And avoiding shadow copies while executing tests on the build server.

Some background

A couple of weeks ago we made the move from CC.NET to TeamCity. At the same time we restructured the solution folders, having problem with dependencies on builds and that scripts took forever to finish. So to speed up the feedback loop for the day-to-day development we needed the new structure.

Setting up the new structure took about one day of work and creating the initial build and test loop for development another couple of days. This without any build scripts at all. The deployment took a bit longer and involved a few MSBuild scripts and it works well now. Also in this is a setup for executing selenium tests, this took some more scripting but finally it’s in place as well.

The problem

The final piece of script was the coverage. Unfortunately there are no integrated support for NCover in TeamCity so you have to revert to good (or bad) old MSBuild scripting. As there a few long running tests in this soup we opted for running coverage at nightly builds. The dev coverage can be executed on the developer machine so there's no problem with this.

So after setting up a converge build for each of the solutions I found that it never produced the same result for all solutions. With no apparent problem, assemblies was omitted from the result. And sometimes it worked as expected. After more analysis it turns out that when one solution has been built it affects the coverage of other builds. More than was explained by the interface points between the solutions.

It got even worse, I did a build for project A, checked the coverage for A, then built project B and the result for project A changed. Checking the files used for instrumenting project A I could not see one single difference down to the timestamp and version for each and every file of the A build.

The solution

Finally I sat down with FileMon from SysInternals and did a check on one specific assembly used in the test to get a grip on what was tested and where it came from. Hmm, what is all this shadow copy used by NUnit? What happens here?

Well it turns out that NUnit copies assemblies to a temporary folder while executing the unit tests, this to allow the user to continue develop while the tests are executed. Well, this is not the problem on the build server. In the coverage run that I created the assemblies built are copied to an analysis folder and the tests are executed there. And the simple and working solution is to add the /noshadow option to the NUnit command. After checking with FileMon and the resulting coverage I finally got this last nagging problem solved.

After the various coverage results are merged with coverage explorer, it is now published with the working builds on our demo server. Automatic build configuration finally done. And less copying of files while testing might even shave a few seconds of build times.

By the way, now that I know exactly the problem and the solution, it was easy enough to find a blog post concerning NCover problems, that explains it all.

// Håkan Reis