Multiple mocks, asserts, and hidden results

In his post “one mock per test considered harmful” , Phillip suggests that having multiple mocks used could make a more readable specification.  he also disagrees with the way I provide some of the advice in my book.

First, I’d like to get to the technical issue at the heart of this matter. Why do I not want to test more than one “thing” in the test? because mock verifications, much like asserts, throw exceptions. That means that any verifications below that line never get called and so you’ll never know if they passed or not.

if you have three assertions or mocks in a test and you verify all of them, you need to accept the fact that is the first one fails, you’ll never know if the other two passed or not.

You’re like a doctor that only sees one symptom in a patient, with two other symptoms hidden. them more symptoms you can see, the better you might deal with the problem, or look in a different place to fix it.

There is one exception which makes sense to use more than a single mock or assertion, even if you know it “hides” the other results.

ask yourself this simple question:

If the first assert\mock fails, do I really still care about the lower ones?

If the answer is “no", then you can go ahead and use multiple mocks\asserts. if you answered “yes” you’d be better off splitting that assert into a different test, with a good name that explains what is the behavior you are expecting.

For me, I find the “no, I don’t care about the other asserts\mocks if the first fails” answer is more likely when you are asserting on the same object (like checking multiple properties on the same object).

Test Frameworks need to support multiple asserts\mocks

If test frameworks did NOT throw exceptions form asserts\mocks, but instead gathered all of their data before ending the test as fail\pass, this would not be needed.

here’s an example of a small helper framework for .NET called “one assert per test” which tries to alleviate this situation, by generating a test-per-assert at runtime. Pretty clever idea!

Naming

When we have this support, then the naming problem might hit us – if you’re testing multiple things, can you make the spec\test name readable enough to fully understand all of its consequences? otherwise it would be very generic and not understandable.

Over Specification

90% of the time when I see tests using multiple mocks, they are actually guilty of over specification (some examples of that in this video)  - where you specify too much of the internal interaction and thus making the test fragile.

for newbies, thinking of multiple mocks as a short rule, is an encouragement to learn why and to make up their own rules later, but to get started with a foundation.

Learning Stages

I’m happy that Phillip has learned enough to avoid recommendations that feel “deterministic” in some way. He’s passed the stage where he needs to look and follow, and he’s making up rules that suit him better, and even making up better rules for the same situations.

My book is largely pointed at people new to unit testing. just like you start learning martial arts by following things step by step (katas), and then leveraging that knowledge to diversify and build upon and improvise new ways, so is my book. it’s not for people who are already doing bdd-like work. it’s for those who aren’t sure where to begin, or how to continue.

Comments

[Mark Seemann]

I find the additional rule that you can have multiple assertions if you don't care about subsequent assertions if the first fails helpful.

I regularly have more than one assertions (such as trying out a cast, and then testing against the downcasted value) in a single test, but always felt a little 'guilty' about it. Now I'm absolved :)

Another argument for single assertions come from the Red/Green/Refactor cycle. If you follow R/G/R to the letter, this would mean that you'd first need to run the unit test to see the first assertion fail, then fix only that assertion, then run the unit test to see the second assertion fail, then fix only that, etc.

Thus, it soon becomes more attractive to write a new test case for each assertion :)

[Philip Schwarz]

Hi Roy,

I really value your book. I must admit that I used to think that one of it's main take home points was your "one mock per test" guideline. It was the first time I saw it expressed in a book, and I thought it was a pretty important guideline. But I think I took it too literally: I did initially believe that you meant it to be a rule.

In Mocks aren't Stubs, the remix Rainsberger recommends against setting an expectation on a method (“mocking” it) when you really want to simulate a specific response from that method for the current test (“stubbing” it). He says that when one uses a mock in place of a stub, one tends to over-specify the collaboration between a Client and its Suppliers, and that leads to brittle tests. I commented saying that those who follow "one mock per test" avoid the mistake described by Rainsberger. Then Nat Pryce, one of the authors of Growing Object Oriented Software guided by tests [GOOS], replied to Rainsberger saying that he also practises Rainsberger's guideline and calls it "Stub Queries, Expect Actions".

So at that point I thought that GOOS might also advocate using "one mock per test". But when I mentioned this idea in Mocks suck reloaded Nat Pryce explained that GOOS does NOT advocate "one mock per test".

[Bradley Landis]

Philip,

Some of the confusion lies around the terminology of Mock/Stub/Fake ect. Especially since most mocking frameworks are moving away from explicitly creating mocks and stubs differently (I think). So when we say 1 Mock per test, it can be confusing. If we consider an expectation on a mock an assertion and stick to 1 assertion per test, it may be more clear.

Thanks,

Bradley