Pros of Automation

If you have to run a set of tests repeatedly, automation is a huge win for you It gives you the ability to run automation against code that frequently changes to catch regressions in a timely manner It gives you the ability to run automation in mainstream scenarios to catch regressions in a timely manner (see What is a Nightly) Aids in testing a large test matrix (different languages on different OS platforms). Automated tests can be run at the same time on different machines, whereas the manual tests would have to be run sequentially.

Cons of Automation

It costs more to automate. Writing the test cases and writing or configuring the automate framework youre using costs more initially than running the test manually. Cant automate visual references, for example, if you cant tell the font color via code or the automation tool, it is a manual test.

Pros of Manual

If the test case only runs twice a coding milestone, it most likely should be a manual test. Less cost than automating it. It allows the tester to perform more ad-hoc (random testing). In my experiences, more bugs are found via ad-hoc than via automation. And, the more time a tester spends playing with the feature, the greater the odds of finding real user bugs.

Cons of Manual

Running tests manually can be very time consuming Each time there is a new build, the tester must rerun all required tests which after a while would become very mundane and tiresome.

Other deciding factors

What you automate depends on the tools you use. If the tools have any limitations, those tests are manual. Is the return on investment worth automating? Is what you get out of automation worth the cost of setting up and supporting the test cases, the automation framework, and the system that runs the test cases?

Criteria for automating There are two sets of questions to determine whether automation is right for your test case: Is this test scenario automatable? 1. 2. 3. Yes, and it will cost a little Yes, but it will cost a lot No, it is no possible to automate

How important is this test scenario? 1. 2. 3. I must absolutely test this scenario whenever possible I need to test this scenario regularly I only need to test this scenario once in a while

If you answered #1 to both questions definitely automate that test If you answered #1 or #2 to both questions you should automate that test If you answered #2 to both questions you need to consider if it is really worth the investment to automate What happens if you cant automate? Lets say that you have a test that you absolutely need to run whenever possible, but it isnt possible to automate. Your options are

Reevaluate do I really need to run this test this often? Whats the cost of doing this test manually? Look for new testing tools Consider test hooks