Minimizing unit Test Fragility – 8 features in Typemock Isolator to help

in continuation to my challenge (which no one had bothered answering, lazy web!)

One of the things that inhibits unit tests in organizations is the idea of fragile tests. a Fragile test is a test that can easily break when the production code changes. That’s not to say that tests should never break as you change production code, but there are ways to minimize this effect so that when tests do break, they break for the right  reason.

the right reason is when a feature in production code isn’t working.

the wrong reasons can be many and varied. Of the most common ones is the idea of over specification in tests. The more your test expects of your code’s internal implementation, the more “specified” it is, the more “expectations” it has on the internals of the code. since internal code is more prone to change, so will the test break more often. so the situation can occur that the production code changed, but still gets the job done, but since it’s doing it differently internally, the test breaks even though the end result is still OK.

One of the main ideas in the new Isolator API was to reduce the fragility of tests by default. there are several features in Typemock Isolator that remove lots of fragility from tests by default, compared to other tools:

1. All Fakes are non strict by default.
2. Recursive Fakes
3. Method stubs ignore parameters by default
4. “From now on” Method behavior as a default
5. Isolate.Swap.AllInstances
6. Isolated.NonPublic APIs will not break when refactoring from private to public
7. Ignore one or all static methods on a type
8. True Properties
9. All Method overloads are faked by default

All Fakes are non strict by default.

when a fake object is strict, then it will throw an exception when someone calls one of its methods that wasn’t specifically “expected” by the test (or a different number of times than expected). since internal code can change and start interacting with other code in different ways, strict fakes will almost always fail your test on the lightest nuance of change. Most frameworks used to have strict fakes by default.

Isolator has all fakes non strict by default. in fact, there is no “strict” mode at all.

var fake = Isolate.Fake.Instance<SomeType>();

//code under test can now call any method without the test needing to specify it. the test only needs to specify behavior of methods it cares about.

Recursive Fakes

This is a feature first introduced by Isolator more than a couple of years ago and then adopted by Moq and Rhino Mocks. the idea is simple : allow chained calls in your code without the test needing to specify fakes at each hierarchy level. for example, calling MyFake.SomeProperty.DoSomething() will, by default, do nothing since its root is a fake object. the “SomeProperty” will be initialized by default to a fake object as well, recursively down the line.

With other frameworks this will only work with interfaces, methods which are virtual and classes that are non sealed. Isolator can make this work with anything, including static methods and non sealed classes.

var fake = Isolate.Fake.Instance<SomeType>();

//code under test can

fake.SomeProperty.DoSomething()

//or

var propObj = fake.SomeProperty;

propObj.DoSomething();

As your internal code is refactored, or changed to call a method down the hierarchy, this should not break a test (as long as the method does not matter to the test)

Method stubs ignore parameters by default

all other frameworks except Isolator will use the values being sent to methods of fake objects during the “setup” phase, as expectations of these actual values. Isolator by default ignores values being sent in, and just returns the fake result we tell it to, no matter what was being sent in. Of course, you *could* tell it to use the exact argument values, but that is not the default behavior.

other frameworks will require that you put in special constraints on each parameter value to say that you don’t care about the actual value.

var o = Isolate.Fake.Instance<SomeType>();

const int IGNORED_VALUE=10;

object IGNORED_OBJECT= null;

Isolate.WhenCalled(()=> o.SomeMethodWithParams(“ignored string”,IGNORED_VALUE,IGNORED_OBJECT) ).WillReturn(3);

this method will return 3 no matter what parameters are being sent to it. so internal code can change parameters slightly and it will still work.

by specifying only on specify param values, we could be over specifying the test.

“From now on” Method behavior as a default

with Isolator, once you set behavior on a method (“Ignore”, WillReturn(..), WillThrowException(..) ), it will use that as its default behavior for all subsequent calls. That way, if the internal code ends up calling a method more than once, the test should not care about it (as long as the result does not hurt the test). This is great for ignore calls to “Log” objects and such, where you don’t know or care how many times it will be called. you just want to ignore them without having to worry if the code will use them or not.

for example, the method from the code above will *always* return 3, no matter how many times it will be called.

if we specified it twice with different return values, it will return the *last* behavior as the default, after it was called once with the previous behavior.

Isolate.Swap.AllInstances

This beauty will replace all existing and future instances of a specific type in the code under test with the fake type. that way you don’t care how many times some object might be created, or how many instances of it are used. for “util” type objects, this makes things a breeze to setup for a test that could be very complicated or impossible in other frameworks.

as the internal code changes with usage of these types, the test will still pass.

var fake = Isolate.Fake.Instance<SomeType>();

Isolate.WhenCalled( ()=> fake.Foo() ).IgnoreCall();

Isolate.Swap.AllInstances<SomeType>().With(fake);

//now production code can do this:

var x = new SomeType();//<—this will behave like the fake object we created

Isolated.NonPublic APIs will not break when refactoring from private to public

Isolator allows setting method behavior on non public methods as well:

var x = new SomeObject();

Isolate.NonPublic.WhenCalled(x,”SomeMethod”).WillReturn(3);

The beauty is that if you do refactor the method to make it public, the test won’t nudge you or throw an exception. It will just work with public methods as well. You can then refactor the test without being a slave to it.

Ignore one or all static methods on a type

Isolate.Fake.StaticMethods<SomeType>() will ignore all void methods on that type, and return recursive fakes on all functions. That way, if you add more methods to that type, the test won’t care about it.  great for utility style classes with lots of static methods.

True Properties

“True properties” is a feature that Isolator picked up from Rhino Mocks – you can set properties on a fake object as if they were real properties.

The nice thing is that if internal code then sets these properties to values you don’t care about, the test will work just fine. it’s also very readable.

var fake = Isolate.Fake.Instance<SomeType>();

fake.SomeProperty=3; //will behave like a real property and return 3;

Default Faked method overloads

Once you set a method’s behavior, if that method has any overloads, they will have the same behavior as well. So if you refactor your code under test to use a different overload of the same method the test will “just work” and won’t need to be changed to reflect this change.