Subscribe!
Test Driven Source Control–Safe Points
Wednesday, February 2, 2011 at 4:49PM Having worked quite a bit now with Git and Mercurial, and tests, I got an interesting idea. An addin that runs inside your favorite IDE, and listens to test related events.
Every time the test suite is finished running, the addin performs a commit of all the current files and state into the local source control system. BUT, it also includes the current results of all the test runs.
essentially, it will associate the state of the system as part of an automated commit on every run of the tests.
What do you get?
You get a wonderful ability of reverting back to the time where all the tests passed, or to a time when a specific test passed.
You also don’t need to remember to commit locally when you get to any kind of stable state.
Each commit will be saved and visualized as a “Safe Point” – and you can travel up and down the source history, based on the test results that you’d like to get to.
This is great if you get down a rabbit hole and break everything. You just revert back to the time when everything was passing, or even partially passing. Today, without this, if you forgot to locally commit during the work or before it, you’re out of luck.
with this system you never forget to commit, and you get access to different states of the system
you also get, if you work in a Test Driven way, a nice way to perhaps visual the history of how some functionality was created. You might play-back the steps it took to make something.
If no one builds it, I will!
Reader Comments (10)
Very nice idea. My initial reaction was "well, no need to save the test results as they can be regenerated from the build." But, frankly, there's no harm. I think that a good next step beyond that would be to be able to link the test inputs and results to the source so that we can do a quick browse, and see what really goes in and comes out.
This is sort of an extension of an idea that Jonathan Edwards had with 'Subtext': programs should have visible computation, sort of like Excel. If you can hover and see the computation work on example values, you can gain deeper insight quickly.
Have you tried using hg/git bisect? It adopts a slightly different approach -- if you're looking for the last revision where a test passed, you can use it to do a binary search of your history for the offending revision.
Personally I'm not too happy about committing failing tests to source control -- unless of course you're using the TDD approach of write a failing test, commit, write code to fix it, commit, lather, rinse, repeat. But in general I personally think it's best not to commit revisions that break pre-existing tests.
It's an interesting idea, but I much prefer just using git bisect, which does exactly that. It can also help you when merging branches and so on. Automatic commiting can't possibly be good for the history (what will the messages be?)
+1 and might join again...
a. What happens in an auto test env. (e.g. https://github.com/acken/AutoTest.Net)? It seems every save will generate a commit and it will be more difficult to travel back when needed.
b. From my experience, it's quite rare to revert to old revisions - I tend to invest more on fixing the bug than finding where it was born. Might be that there is a path here to better locate the proper junctions in code evolution.
Regards
Using a continuous integration server also gets you this information. I use teamcity, and when a test fails, it will show you if this is the first time it failed, when it was last successful, and a complete history of the test including fail/pass and time taken.
I personally don't like the idea of auto-commit. Just because a test passes once doesn't mean I am done, or care to keep history at that point in time. If you're breaking tests in work-in-progress code but are so deep you can't figure out how to un-break the test, you're probably working on too much at once.
I think a few people are missing the fact that this is a Local commit only. It wouldn't trigger CI and if you don't like all the "little" commits in your commit history, you can squish them before you push to the master repository.
I think it is a brilliant idea.
You could install TeamCity locally, point it to whatever local repository you want and tell it to label after a build.
Rik, TeamCity watches a repository, so you'd still have to check it in manually every time.
I thought about it some more and came up with a solution. I have been using AutoBuild (http://code.google.com/p/autobuildtool/) lately. This does exactly what you want Roy. Just add an nant task to commit and it will build/run your tests/commit locally every time you save. Brilliant!
My Kata Template @ https://github.com/bradleylandis/Kata-Template now includes performing a local commit every time the tests pass.
I also wanted to mention that as an alternative or in addition to checking in the test results on every commit, you could commit the the report generated by something like TestDox for .NET (http://testdox.wordpress.com/)