In a previous post I explained the different between Continuous Integration tools and Build automation tools, but there’s also lots of be said for HOW to pick a specific CI or BA tool.

What do I look for in a CI tool?

• first and foremost – how intuitive and usable is it? how easy is it to add, remove or change build configurations? How long will it take me to explain to a new dev on my team how these builds are connected?, and how long will it take them to change or fix a build related problem?
• Does it have the ability to run builds on “slave” machines or just on the local machine where its installed?
• Do I need to use XML to configure it?
• How easy is it to see the status of a build, history of builds?
• can I “kill” and detect a “stuck” build?
• How long does it take to setup?
• Does it have good metrics?
• does it contain ability to launch random “exe” or “command line” batch files?
• Does it allow for build hierarchies? (builds that depend on each other’s output, or that trigger each other automatically, or that all have to pass )
• Is there a large community or product support in case of a commercial product?
• How much does it cost? will it cost me more to use a “free” tool given the questions above?

What do I look for in a build automation tool?

• first and foremost – how intuitive and usable is it? is the configuration interface visual? XML based? tree based? how easy is it to add, remove or change existing scripts? How long will it take me to explain to a new dev on my team how the build works, and how long will it take them to change or fix a build related problem?
• How many built in tasks are there with the tool? the more “custom” tasks i need to create to make my build work, the less productive I am
• How easy is it to create a custom task? in what language? how easy is it to integrate a custom task into a build flow?
• Can a script run other scripts ?
• How easy is it to read the log generated by the build tool?
• How well does it play with my existing toolset? (if I use TFS – can I easily invoke it from TFS? can it invoke TFS related things like add new TFS task items?)
• Is there a large community or product support in case of a commercial product?
• How much does it cost? will it cost me more to use a “free” tool given the questions above?

Bonus question:

• Am I choosing this tool blindly because someone I respect is using it, or did I check other options and this choice is based on the actual pros and cons of the tool?