Enterprise library -- Test usability and "runnability"

So, Enterprise Library is out and it's a great thing. It even has unit tests. But is anyone actually using them?

 

Enterprise Library is a great test case for the issue of unit tests usability because:

1. It will be used by thousands of developers
2. It will be maintained, changed, configured and extended by thousands of developers
3. It will  need to be understood by thousands of developers
4. It will need to live for a long long time as a generic block system base
5. It even has a survey asking people how they are dealing with the unit tests in the system

So - the first hurdle we've already come across - fatal clause #2:

 

This is a great example of how you can create a well tested system with automated tests, and then watch maintenance developers completely ignore them and go on changing the system without checking for breaking changes just because they find it hard to make the tests run in the first place.

• People can't just open the application and run the tests. They need to "figure stuff out" first.
• That means that they probably won't have time to "figure stuff out" and so they'll just start changing the code without making sure they didn't break anything
• More people will then use the "changed" and untested code thus they are more likely to not run the unit tests as well. Even if they ran the tests, who's to say there isn't a bug? suddenly the tests are not trustworthy. "They passed" -- "That doesn't mean anything. That's not even tested. We have to start debugging it."

The broken window theory rules here. The harder it is to run the tests, the less likely it will be for people to run them, meaning the less likely it will be to write new tests ("I'm gonna test it *now"?? half of this is untested anyway!")

 

A great hint you're missing something is if you find yourself starting up forum threads on this issue or posting blog posts on making stuff work. It's like comments in your code - if you write them you're probably not doing something right(for readability. In our case it's usability or "runnability").

 

I'd love to take a look at the replies to the EntLib test usability survey (not specified as one, but has some specific questions regarding the usage and helpfulness of the unit tests in the system) and I bet most of them are of the sort "I can't make it run so I gave up" or "I didn't know you have unit tests" or "I'm not sure what I should do with these".

 

So what's the solution here?

It's simple and painful all at once: make it automated. *everything*.

 

When I install EntLib, I want it to install all the test cases, all the test configurations, all the Database scripts, *everything* so that all I need to do after I install is press a button or click a menu. I don't want to do anything more than that, because 90% of the developers won't bother doing something that takes two-three clicks, or maybe run a batch file, god forbid.

I want the setup program to ask me for all the locations of my database and where I want to setup the test data. If it can't handle that, I'd rather have the tests divided into two or more "runnable" categories:

"Can run now", "Can run later if you have X", "Can run later if you have Y".

The more categories you have, the less likely it will be for someone to run *all* the tests and we're back into the vicious circle again.

 

The whole idea here is automation, so that's what people expect. If I have to setup a build machine for an hour just to see if my tests work - that's fine - just don't expect thousands of developers to be as dedicated as I am to the cause - they won't be. They'll mangle your code in ways you didn't think of - and they probably won't even be aware that there is an automated way to check if they broke anything!

 

To save that last point - you should make EntLib build configuration run a specific tool that checks if tests have been run at least once and alert the user if they didn't. It's easy to setup in the unit tests to set some global variable or registry key or whatever to be able to check it from the pre-build event or anything of this matter.

 

To summarize:

I'm not saying the EntLib guys didn't do a great job - they certainly did I think it's a breakthrough component in many ways.It's a great leap forward to have unit tests for all this code.

We need to think a bout the next step though - the one we're starting to experience today as people start to come of age with the notion of writing unit tests-- the next leap forward will be to actually make everyone run them at least once without breaking a sweat. They need to be runnable which would in turn make them usable. 

 

Then, my friends, we can start talking about productivity.