This post is a continuation of my Quest to test FxCop rules. The previous one was about writing pure unit tests for FxCop rules. This one is about writing Integration tests with a new framework I’ve developed called FxCopUnit.
For the past few days I’ve been struggling with FxCop – I’ve been Reflectoring the hell out of it (Reflector 5.0 is amazingly cool).
The purpose of all this fussing around is to create some sort of usable framework that would allow an easier testing experience for creating custom FxCop rules. I’ve already described how you can write real unit tests for custom FxCop rules here, but the framework I’m discussing here is meant for writing integration tests for FxCop rules . The main differences between the two is that integration tests usually take longer to run, harder to manage, need configuration and can’t be performed until the full (or at least some parts are full) system is in place.
Still, Integration testing has its advantages. For instance, the ability to realize that everything works together like clockwork. The analogy equivalent of Integration tests vs. unit tests for me (which I’m also using in my upcoming book) is that of a car’s engine. When you turn your key in the ignition, and the engine roars, you know all is well. Or maybe, you hear a slight noise from the engine. If you do, what’s the possibility that you’ll be able to recognize what the source of the problem is? That’s the classic integration test: all the parts work together as one single unit, and it’s just as hard to separate origins for problems – however- when they work, you know they work well together as a system. A unit test would test and run each of the engine parts on it’s own – easier to test, but you just won’t know the system really works until you get to that final system (integration) test.
Enough rambling.
I’ve used Sasha Goldstein’s code sample as a basis to create a more elaborate framework that will allow your custom rules to go through all the stages of running in FxCop, up to the XML report in the end. The main goals of the framework are:
- Easily specify specific rule you’d like to run
- Easily specific only specific method or type or member you’d like to test your rule against
- Easily tell FxCop where everything is located using code
- Easily run FxCop and get the Resulting report data
- Write as little lines of code as possible to do this
- Clean up the FxCop engine between tests
- Easily zoom in on type or member for testing, without needing full assemblies and multiple rules checked.
The full downloads for this framework are available here:
- Just the binaries
- Source and Binaries (including a new version of XtUnit)
In the full source download you’ll also find samples on how to use the framework but you’ll get a nice little 5 minute guide to it in both of them).
The FxCopUnit also uses an earlier framework I wrote called XtUnit which allows attaching custom attributes to tests which perform actions before or after the test. In this case I used it to create a custom attribute that initializes FxCop to the specified rules and types.
Here’s how you could write a simple Integration test for FxCop with FxCopUnit:
- Add the following references:
- from C:\Program Files\Microsoft Visual Studio 8\Team Tools\Static Analysis Tools\FxCop\
- FxCopCommon
- FxCopSDK
- Microsoft.Cci
- TeamAgile.FxCopUnit
- Add reference to XtUnit.Framework.dll
2. Create a new test fixture class
3. Inherit your class from FxCopTestFixture
Add the attribute to the top of the class to specify the location of your custom rules:
[TestFixture]
[FxCopCustomRuleLocation("TeamAgile.FxCopRuleTesting.SampleRules.dll")]
public class MytestsClass: FxCopTestFixture
{...}
4. Create a new test:
[Test]
[FxCopRuleToTest("Rule0001", "FxCopCustomRules.CustomRules",true)]
public void RunRuleAgainstMethod()
{
}
Note the FxCopRuleToTest attribute on the test. Specify the rule ID and Rule category as written in the Rule XML file manifest located inside your custom rules library (in the sample source it is RuleManifest.Xml – complied as Embedded resource).
The third parameter setup up FxCop engine so that the current test library will also be the main target to test the rules against.
Alternatively, you can specify the name of the target assembly to test against.
You can also specify the specific type that your rule will run against (all other types in the assembly will be ignored by the rules engine) .
Here’s a test that specifies a rule, and a specific method to test against.
It then tells FxCop to analyze the targets and does some simple asserts against the resulting FxCop report.
[Test]
[FxCopRuleToTest("Rule0001", "FxCopCustomRules.CustomRules",true)]
public void RunRuleAgainstMethod()
{
MethodInfo methodInfo = MethodBase.GetCurrentMethod() as MethodInfo;
FxRunner.EnableMethodToCheck(methodInfo);
FxRunner.Analyze();
FxReport.Assert.AtLeastOneAnalysisProblem();
Console.WriteLine(FxReport.Document.InnerXml);
}
The test uses two objects which are available once you inherit from FxCopTestFixture:
FxRunner:
Helper used to specify rules, types and other logic, as well as to start the analysis process by FxCop.
FxReport:
A report object that is generated only after calling FxRunner.Analyze(). It will be null before that.
Asserting against output:
Most of the asserts should be performed against the FxReport object, which already has a property named “Assert”. It is an AssertFxCop object which has some very simple asserts in it. It can be extended though- it’s a partial class. (so is the FxCopReport class)
FxCopUnit is very much work in progress. I wanted this out there so people can start using it and perhaps send me some interesting assert methods I can add built in to the current framework.
Your comments are appreciated.