a few days ago, we got into an interesting email conversation with a person in a large company, who had purchased isolator a while ago. Our customer lieason sent me this person’s recent response to the “how are things going with Isolator?” email we sent them.

He expressed concern because the customer had suggested they are looking into switching away from Isolator, towards using MOQ. First, here’s the email:

“Well, I am the one who introduced TypeMock at [company] so I suppose that is a natural delegation.
We started out with an organisation that had never done any kind of test driven development and had never done any work with dependency injection. We are gradually porting a lagacy vb6 application to .NET. We went with TypeMock Isolator so that we did not have to adjust the way we wanted to design and code and it went really well in the beginning. We also decided to work behaviour driven with a MVVM pattern in WPF.

However, we are about a fifth thru the migration process and have now passed 3000 tests. Those take about ten minutes to run on our build server. 15 000 tests would take close to an hour to run. Hey, that can't be right! So we investigated the reasons why our tests take so long to run and apart from the integration tests (yapp, they will eventually be set aside) the single most costly component seems to be having TypeMock Isolator enabled. Not necessarily running tests using TypeMock Isolator, but having it enabled. Due to the Profiling API I assume. I have sent you guys timing data previously indicating the big problem but I'll give you another example. A newly started project with 175 tests using Moq and StructureMap as dependency injection framework. No TypeMock Isolator whatsoever. Running the tests takes less than 8 seconds including integration tests. Enabling TypeMock Isolator it takes 17 seconds! More than 100% time increase! So that is a really big issue.

Apart from that we have had issues with prior versions where the Resharper runner stalled when running tests residing in different assemblies. We have an ongoing issue with tests not running after having created a first WPF-control in a test. For that we have a hack where we catch an exception in a dummy test (if I remember correctly) when creating the first WPF control. Following creation is no issue. I have tried to reconstruct it in stand alone environment but have failed at that (do not really have time to investigate further I'm afraid).

We are also under the impression that mocking without dependency injection works pretty well in a small scale, but as the code develops it gets harder and harder to maintain the tests. Partly because designing that way gives too much room to just add responsibilities to classes without having to rethink what restructuring/refactoring really should be done. Or put another way, it does not push my team mates in a test first direction. And I do not like that at all!

Well, as you gather we are moving on. Moq and StructureMap it is for now using dependency injection. It works really well. But then again we are still in the initial phase having a rather small code base.

If there is anything I could help you out with I will do my best to assist you. But time is kind of restricted in those areas here at my customer, and at home I am way to busy developing. So it might not be an instant response.
  Kind regards”

Wow.  I don’t think it gets any better than that when as a company we are looking for input from a real life customer. Yeah – we need to deal with perf issues, and minor bugs, but I’ve highlighted the best part of this email for me:

“Partly because designing that way gives too much room to just add responsibilities to classes without having to rethink what restructuring/refactoring really should be done. “

as far as I’m concerned, this is a success story for typemock (even though we have some things to fix). the reason? we solved the initial problem for that large client – become agile in the first place and learning to unit test – without needing to learn design first. they have successfully been able to split, delay and prioritize their learning, and still become better and more agile. because they were able to separate the task of unit testing from the task of learning design they were able to actually get things done and show results in a meaningful time, so the effort to drive agility was showing results quicker. if they had done them both at the same time chances for success would have been much lower, and time would be much longer.

as it stands, they went from zero agile to unit testing and TDD, and now they are finally at a stage where the design is starting to be their main concern. unit testing is just another tool in their belt. I feel that our product has enabled that company to achieve this stage, and may no longer ne needed for them. What they need now are tools that teach design, not tools that enable unit testing. and MOQ (or FakeItEasy, or RhinoMocks) are great for such a task.

They’ve outgrown us – and it’s sad to see them leave – but I’m happy they got to that stage – that we were so successful that we’ve made ourselves redundant.

That is pure awesome. this is why I come to work every day.

congratulations!

I promise we’ll work more on the perf issues mentioned so that our next customers will have an easier time.

interesting question – should we have a free version of isolator without the “power features” so that customers at least won’t have to learn a new syntax when they want to start focusing on design learning?