Introduction
I recently needed to set up a Continuous Integration server for some projects I'm about to start. There are quite a few options, but the two main players I looked at in .NET land were CruiseControl.NET (v1.4) from ThoughtWorks and TeamCity (v3.1 and v4 EAP) from JetBrains. I started my evaluation by taking a project I'm working on, which has a clean, one-click MSBuild powered build system (builds, runs tests, and generates both test and code coverage reports), and hooking it up to each CI server. Below is an overview of my experience setting each system up and my first impressions.
Project
Before diving into the nuts and bolts of CC.NET and TeamCity here's a quick overview of my project. It's a Silverlight app which has several assemblies, including NUnit test projects. Initially I investigated using NAnt to drive the build but after looking at MSBuild I dropped NAnt immediately. Using MSBuild allows you to use the vs2008 .sln and .csproj files directly so you get exactly the same experience building outside vs2008 as you do inside - this makes NAnt scripts mostly redundant. There are many more NAnt Tasks available, but if you really need them you can kick off NAnt from an MSBuild script, which leverages your already existing MSBuild files. You don't need vs2008 installed to run the build. The .csproj files essentially are very similar to NAnt scripts with some hooks to allow you to customise the build process (pre and post-build targets). Currently I have tapped the post-build target of the NUnit test projects to kick off NCover to run the unit tests and produce both test and code coverage reports using a Conditional construct in the target tag (if a property RunTests is set to True, the target executes). This is enough to integrate very easily with CC.NET, but with TeamCity I needed to create an extra build script which executes the build above and has a further target to run NCoverExporer (to build the Coverage Report in xhtml) and MSBuild Community Task XSLT using NUnitReport.xsl (to build the Test Report in xhtml).
The AfterBuild targets in each of my NUnit test .csproj files looks like this:
1: <Import Project="..\..\UtopiaTest.targets" />
2: <PropertyGroup>
3: <CoverageAssemblyName>Utopia</CoverageAssemblyName>
4: </PropertyGroup>
5: <Target Name="AfterBuild">
6: <CallTarget Targets="RunTestsAndCoverage"
7: Condition="$(RunTests)=='True'"/>
8: </Target>
The CoverageAssemblyName property tells NCover what assembly to run coverage over for this particular project assembly. It imports the following reusable test and coverage target:
1: <Project ToolsVersion="3.5"
2: DefaultTargets="RunTestsAndCoverage"
3: xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
4:
5: <PropertyGroup>
6: <NUnitPath>$(MSBuildStartupDirectory)\3rdParty\Externals\NUnit\</NUnitPath>
7: <NCoverPath>$(MSBuildStartupDirectory)\3rdParty\Externals\NCover\</NCoverPath>
8: <NCoverExplorerExtrasPath>$(MSBuildStartupDirectory)\3rdParty\Externals\NCoverExplorer.Extras\</NCoverExplorerExtrasPath>
9: <NUnitReportsPath>$(MSBuildStartupDirectory)\TestReports\</NUnitReportsPath>
10: <NCoverReportsPath>$(MSBuildStartupDirectory)\CoverageReports\</NCoverReportsPath>
11: </PropertyGroup>
12: <UsingTask TaskName="NCoverExplorer.MSBuildTasks.NCover"
13: AssemblyFile="$(NCoverExplorerExtrasPath)bin\NCoverExplorer.MSBuildTasks.dll"/>
14: <Target Name="RunTestsAndCoverage">
15: <Message Text="##teamcity[progressStart 'Running Tests and Coverage on $(AssemblyName).']"/>
16: <MakeDir Condition="!Exists('$(NUnitReportsPath)')"
17: Directories="$(NUnitReportsPath)"/>
18: <MakeDir Condition="!Exists('$(NCoverReportsPath)')"
19: Directories="$(NCoverReportsPath)"/>
20: <Exec Command="regsvr32 /s $(NCoverPath)\CoverLib.dll"/>
21: <NCover
22: ToolPath="$(NCoverPath)"
23: LogLevel="Quiet"
24: WorkingDirectory="$(MSBuildProjectDirectory)"
25: CommandLineExe="$(NUnitPath)bin\nunit-console.exe"
26: CommandLineArgs="$(OutputPath)$(AssemblyName).dll /xml=$(NUnitReportsPath)$(AssemblyName).Results.xml /nologo /nodots"
27: CoverageFile="$(NCoverReportsPath)$(AssemblyName).Coverage.xml"
28: Assemblies="$(CoverageAssemblyName).dll" />
29: <Exec Command="regsvr32 /u /s $(NCoverPath)\CoverLib.dll"/>
30: <Message Text="##teamcity[progressFinish 'Running Tests and Coverage on $(AssemblyName).']"/>
31: </Target>
32:
33: </Project>
Incidentally, editing a vs2008 project file in vs2008 is easy - just right click on the project in the solution explorer, select Unload Project, and right click again and select Edit. When done, save your changes, close the file and right click again, selecting Reload Project.
CruiseControl.NET
Setting up was a little more involved than I expected. You need to have IIS installed before you install CruiseControl .NET. If you install IIS after CC.NET, you need to go in and manually set up a virtual directory. Once everything is installed, you need to dig in to a xml config file to set up how your build is pulled from source control and executed. One gotcha with CC.NET and subversion is that initially it doesn't work! CC.NET fails to pull the source from svn complaining about a Certificate problem. You can solve this by using TortoiseSVN to manaully pull the source down and "Accept Forever" the certificate. Feels a little hacky. You can also specify the various artifacts generated by your build that you want published, like the xml output of NUnit, NCover, or a host of other tools. The great thing about CC.NET is that's all you need to do to get reports presented for your build. Under the hood, CC.NET has .xsl transformation files for all the usual suspects, so once it knows where to look for these xml reports, it can do the rest. That said, the .xsl files that come with CC.NET don't produce pretty reports. The reports generated fit quite consistently with the overall look of CC.NET - think web circa 1996!
TeamCity
TC was a breeze to set up, initially. Once installed, you create a project and configurations by walking through a web UI setup dialog where you provide the same kind of details as the CC.NET xml config file, so you don't need to get your hands dirty with xml. Literally, a few minutes after installing I had my build working in TC. However, there's no such thing as a free lunch in TeamCity .NET land. I tried a few Build runners (MSBuild, sln2008 and commandline) but the only one that allowed me to incorporate test reports without any extra work on my part was the sln2008 runner which has a complementary NUnit task allowing you to specify your test assemblies. This is great, but if you want to run NCover, you're out of luck. If you use any of the other Build Runners mentioned, while you can run tests yourself and generate test output, you need to manually transform the xml into xhtml and dig into the TC xml config file to tell TC about it. The process isn't exactly painless if you're new to this.
I used NCoverExplorer to merge my coverage reports and transform them into a xhtml summary file. TC is already configured to expect a coverage report in the form of a zipped web site called coverage.zip at the project root - you just need to build that file, and register it as a build artifact. This gives you a Coverage tab on the build report page. Because I used NCover to run the unit tests, and didn't want to run them twice (once with the TC NUnit task, which gives you a very basic test report in TC, and once inside your build script with NCover to generate the coverage reports), I ultimately used the CommandLine build runner. This meant I only had to run my tests once, but also manually transform the xml test reports into a html based report and register it with TC. This is a bit more work than the basic CC.NET setup. It does end up being prettier though. That said you could do exactly the same with CC.NET.
The process of transforming the NUnit xml test reports into a xhtml form involved using the MSBuild XSLT task. You simply need a .xsl transformation file (NUnitReport.xsl), which you'll find the the source distribution of the MSBuild Community Tasks, not the binary (.msi) distribution, for some reason. The report generated is clean and fits well with TC.
Here is the MSBuild script I created which builds the project proper, and then merges the test and coverage reports, spitting out xhtml, ready for TC:
1: <Project ToolsVersion="3.5"
2: DefaultTargets="Build"
3: xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
4:
5: <PropertyGroup>
6: <ApplicationName>Utopia</ApplicationName>
7: <Configuration Condition=" '$(Configuration)' == '' ">Release</Configuration>
8: <RunTests>True</RunTests>
9: </PropertyGroup>
10:
11: <!-- Build the project and Run all Tests/Coverage -->
12:
13: <Target Name="Build">
14: <MSBuild Projects="utopia.sln"
15: Properties="Configuration=$(Configuration);RunTests=$(RunTests)"/>
16: </Target>
17:
18: <!-- Collect all Coverage report xml and build final report -->
19:
20: <PropertyGroup>
21: <NCoverExplorerPath>$(MSBuildStartupDirectory)\3rdParty\Externals\NCoverExplorer\</NCoverExplorerPath>
22: <NCoverExplorerExtrasPath>$(MSBuildStartupDirectory)\3rdParty\Externals\NCoverExplorer.Extras\</NCoverExplorerExtrasPath>
23: <MSBuildCommunityTasksPath>$(MSBuildStartupDirectory)\3rdParty\Externals\MSBuild.Community.Tasks\Build</MSBuildCommunityTasksPath>
24: </PropertyGroup>
25:
26: <UsingTask TaskName="NCoverExplorer.MSBuildTasks.NCoverExplorer"
27: AssemblyFile="$(NCoverExplorerExtrasPath)bin\NCoverExplorer.MSBuildTasks.dll"/>
28:
29: <Import Project="$(MSBuildCommunityTasksPath)\MSBuild.Community.Tasks.Targets" />
30:
31: <ItemGroup>
32: <nunitReportXslFile Include="$(MSBuildCommunityTasksPath)\NUnitReport.xsl">
33: <project>$(ApplicationName)</project>
34: <configuration>$(Configuration)</configuration>
35: <msbuildFilename>$(MSBuildProjectFullPath)</msbuildFilename>
36: <msbuildBinpath>$(MSBuildBinPath)</msbuildBinpath>
37: <xslFile>$(MSBuildCommunityTasksPath)\NUnitReport.xsl</xslFile>
38: </nunitReportXslFile>
39: </ItemGroup>
40:
41: <Target Name="Coverage"
42: DependsOnTargets="Build">
43: <Message Text="##teamcity[progressStart 'Merging Coverage on $(ApplicationName).']"/>
44:
45: <!-- Use NCoverExplorer to merge all the coverage reports and build a Html page -->
46:
47: <NCoverExplorer
48: CoverageFiles="$(MSBuildStartupDirectory)\CoverageReports\*.Coverage.xml"
49: FailMinimum="false"
50: HtmlReportName="index.html"
51: OutputDir="$(MSBuildStartupDirectory)\CoverageReports"
52: ProjectName="$(ApplicationName)"
53: SatisfactoryCoverage="60"
54: ReportType="ModuleClassFunctionSummary"
55: Exclusions="Assembly=*.Tests;Namespace=*.Tests*"
56: ToolPath="$(NCoverExplorerPath)"/>
57: <Zip Files="$(MSBuildStartupDirectory)\CoverageReports\index.html"
58: WorkingDirectory="$(MSBuildStartupDirectory)\CoverageReports"
59: ZipFileName="$(MSBuildStartupDirectory)\coverage.zip" />
60:
61: <!-- Use the NUnitReport.xsl to transform the test reports into a Html page -->
62:
63: <CreateItem Include="TestReports\*.xml">
64: <Output TaskParameter="Include"
65: ItemName="TestReportXMLFiless"/>
66: </CreateItem>
67: <Xslt Inputs="@(TestReportXMLFiless)"
68: RootTag="mergedroot"
69: Xsl="@(nunitReportXslFile)"
70: Output="TestReports\index.html" />
71: <Zip Files="$(MSBuildStartupDirectory)\TestReports\index.html"
72: WorkingDirectory="$(MSBuildStartupDirectory)\TestReports"
73: ZipFileName="$(MSBuildStartupDirectory)\tests.zip" />
74:
75: <Message Text="##teamcity[progressFinish 'Merging Coverage on $(ApplicationName).']"/>
76: </Target>
77:
78: </Project>
Quirks
I'm running Windows Vista as my development platform, and I have 2 bare minimum WinXP Pro VMs set up specifically for these CI evaluations. The VMs have just enough installed to allow the build to happen, and nothing extra - no vs2008 for example. I installed CC.NET and TC each on it's own VM with CC.NET running on IIS and TC running on Apache/tomcat. Initially my TC setup wouldn't trigger on a svn commit, despite being told to. I eventually tracked the problem down. Although the time was identical on my Vista machine (where svn lives) and my XP VM (where TC lives) the time zone wasn't. My VM was running in the future relative to my Vista machine, and it threw off the TC build trigger. As soon as I corrected that, the triggers worked normally. This problem never presented in CC.NET despite also having the same time zone difference. Both systems seem to use different ways of tracking changes in svn.
Conclusion
With respect to .NET development and the basic features considered above, there isn't a huge amount of substantial difference between the two, aside from cosmetics. What TeamCity gained in ease of setup, it quickly lost in it's lack of out-of-the-box reporting (at least in the .NET scenario I ended up with). CC.NET has a slightly more involved setup phase, but a simple and powerful reporting mechanism, if the default .xsl transformations are acceptable to you. Aside from kicking off an automatic build of your project, the meat of these packages is the reporting, and the presentation of that information is important. Overall, CC.NET and it's reports look really dated. You can of course create your own .xsl transformation files to compensate for this. TeamCity looks fresher, but (depending on the Build Runner you choose) doesn't provide good, out-of-the-box reporting, forcing you to do the heavy lifting generating reports that it can display. Unlike CC.NET, TeamCity is a commercial piece of software and while you are free to use it on small projects without any fee, you will have to pay for it if you scale up. See the JetBrains web site for details. To be fair to TC, my evaluation is exclusively on the .NET side, and it looks to me that .NET isn't entirely a first class citizen in TC just yet, although I think it's definitely heading that way. TC still seems slightly more oriented towards Java projects, particularly with respect to code coverage.
TeamCity does have many other bells and whistles that I haven't yet tried, like a vs2008 IDE plugin.
Ultimately, If you're working in a team environment, having a CI server is more important than which CI server you choose. That said, TeamCity feels like a modern piece of software. It has a sense of polish that CC.NET doesn't have. There's still a place for CC.NET but, for me at least, TeamCity comes out on top.