Selenium

The first of my 5-minute lightning talks has been posted to the Java Posse YouTube channel. Thanks to Carl Quinn for the videotaping and processing.



Comment on this video

Summary

Selenium is a web site testing tool that lets you build repeatable tests to ensure that all the important functionality of your web application still works after you've been altering your code base. The master copy of a test is saved as an HTML file, which can be edited easily in the Firefox plugin called the Selenium IDE. From there it can be saved as Java code to execute in a JUnit test, or in other languages like PHP, C#, Ruby, Perl, Python.

The Good News

Most interesting web applications depend on JavaScript, properly named hidden form inputs, and the integration of many systems working together in real time. The best way I've seen to test an entire system in a real browser is to create a collection of Selenium tests and run them automatically as a JUnit test suite. If you habitually write and maintain Selenium tests for all new and altered functionality on your web site, and run those tests a few times a day using a continuous integration system like Hudson, you'll have a solid test bed that tells you as soon as someone on your team breaks something unexpectedly.

The Bad News

Selenium is young and finicky. It doesn't want to eat its brussels sprouts and it sometimes cries for a glass of water in the middle of the night. Because it runs in a real browser, it is subject to various meaningless error states that I've seen over the past 2 years of using it, including:

• Random timeouts waiting for pages to load
• Inability to use secure pop-up windows in IE
• Overloaded CPU
• Failure to start if Firefox is already running
• A browser alert hangs the entire test suite indefinitely

Conclusion

Despite these monkey wrenches, I adore Selenium for its assurance that my web site works correctly before each release. It doesn't catch everything that a human QA team can find, but it does catch problems that a QA team doesn't have time to quadruple check every single week. You will probably need to write enough test utilities to restore the system to a known, deterministic state for each test. You'll also need to get your team in the habit of fixing a few tests for each significant code change, and learn to live with the fact that you won't always know who broke a test. Just fix it anyway. Once all that is in place, then you begin to have strong confidence that all your assumptions about your web site's functionality are still true six months later.