Roy Osherove

View Original

The future of unit testing tools - MbUnit, NUnit, NMock and FIT

Roy Osherove

After I posted that I still recommend NUnit over MbUnit, and reading Andrew's response regarding how it's unlikely that MbUnit will freeze anytime soon, I started thinking some more.

What if MbUnit keeps going and becomes the de-facto standard?

If it's "freezability" is not an issue anymore, you're left to compare it by merits only to NUnit - which is interesting.

Currently, MbUnit surpasses NUnit in almost every facet: amount of features, types of attributes, ease of extensibility, amount of active contributors - almost everything.

Why almost?

  • It doesn't have a mocking framework (NUnit.Mock)  - not such a big deal since most people will use a mature framework such as NMock.
  • Documentation is abysmal at best (not even a cookbook that I know of, everything is on Peli's old blog)
  • and the only other thing NUnit has over MbUnit for me right now is simplicity. It's simple and easy to use NUnit without too much to worry about when you first learn it.

I'm not really sure that's such a big deal as well. To get effective unit tests done you need to be more proficient in the unit testing tools out there, including knowing about mocking frameworks, database testing, TDD, naming standards, and obviously know all the possible attributes. You might even need to extend the test framework when you get down to testing your real code, so simplicity is not such a big matter to me and shouldn't be to most devs out there.

So let's say that MbUnit becomes the de-facto standard out there.

What happens to NUnit?  I'm not saying that's such a big problem if it fades away, but what happens to all the people who contributed to it? Should they join the MbUnit effort? could they?

How mature is the MbUnit framework to stand on its own? Could it be a standard until it has at least decent docs and a mocking story?

I think several things separate it today from getting there"

  1. The user base is a tiny fraction of the NUnit user base
  2. that's it.

Getting #1 is what it's all about, and to get there they need:

  1. Better recognition form the online community (This post is a good example, so is my own post here)
  2. Better acceptance in "legit" docs such as MSDN, magazine prints etc
  3. Usage of MbUnit in conferences and user groups
  4. Support for running MbUnit tests inside NUnit (hard one! but possible)
  5. When VS 2005 testing tools will support using any unit test framework underneath as part of the MSTest.exe runner.

Personally, I like it more and more, but I think the UI still has a ways to go before I can call it sable enough. Still, I mostly use TD.NET inside vs 2005, or even better I use Resharper's Unit Testing support in VS 2005.

I think the Agile world is maturing into a better, more condensed community which better understands and reflects the features of developers. Currently, I think the unit testing frameworks in .NET are some of the most powerful - even more than Java, which was there first.

Interesting tidbit: MbUnit seems to be the only framework that is actively using the notion of Parameterized Unit Tests out of the box (no wonder that Peli, the original author who now works at Microsoft, is moving to that Microsoft research team!)

Other unit test frameworks that might be the future:

Mocking Frameworks

The mocking frameworks are getting better and better too. Look at ayende's RhinoMocks framework and tell me that' s not a possible future. Or NMock 2. Could those be the MbUnit Of the Mocking world? Maybe - but NMock 2 has been under construction for a year now, and RhinoMocks has but a fraction of the size of the MbUnit user base, if at all - again, recognition and the fact that only a single person is working on it.

Acceptance Tests

What about Fit and Fitness? It's going somewhere -slowly. I'd like to see better versions of both of these, since it's about time customer acceptance tests became a main player. Right now- it's nowhere near getting recognition.