Every product we test at TextWise presents its own unique challenges. We recently released a new version of foof, our Firefox add-on that replaces ads with relevant content. Testing foof required us to consider issues such as Firefox versions, operating systems, the wide variety of sites and pages on the web, along with other Firefox extensions that users may have installed.
We’ve written previously about testing on multiple combinations of browsers and operating systems in our blog post “Will this work using PlanetWeb 2.6 on my Dreamcast?” Testing a Firefox add-on makes this a little simpler because we don’t have to consider non-Firefox browsers, but we do still have to consider multiple Firefox versions and multiple operating systems. In addition, we have to consider the potential impact of other Firefox add-ons. It’s not feasible to test every potential combination of add-ons, so we test with the most popular extensions that have the greatest potential to impact our product. For foof, that means paying particular attention to popular add-ons that touch the rendering of the page. Fortunately, the Mozilla site allows sorting of add-ons by popularity and by category making it relatively simple to identify add-ons like Stylish or Greasemonkey that use scripts to alter the look of the page.
Testing of any browser-based application is going to need to take into account the wide variety of sites and pages that are available on the web. We first use our internally hosted test pages and then supplement by browsing to external sites. Our internal test pages include both our own hand built pages which cover a variety of test cases and local versions of pages that have caused problems in the past. A tool like wget comes in handy for creating local versions of these problem pages, and having them in-house means we are not repeatedly hitting someone else’s server during testing. Only after we’ve tested successfully on our internal pages do we test on external pages focusing on the web’s most popular sites. Alexa is one source for web traffic data that can be used to identify sites that should be tested to ensure the browser extension will serve the needs of typical users. Firefox’s error console and the Firebug add-on allow testers to monitor for problems that may not be immediately visible on the test page.
Testers need to remember that there are two basic types of installation that could occur when the browser extension is released: a fresh installation and an upgrade. In addition, there can be multiple paths for installation and upgrade depending, for example, if a user goes to our foof site or to the add-ons on the Mozilla site. All of these installation options need to be tested. An added complication with Firefox extensions is that it’s not possible to test installation and upgrade from the Mozilla site until after release. We test installation and upgrading using our internal site and then do additional testing once the application hits the Mozilla download site. Of course, we also look at all the foof configuration options as well. And what gets installed also needs to be uninstalled so we verify that foof will uninstall cleanly (not that anyone would want to do that!).
Firefox add-on testing, as with any other product, does not end with the release. We know that we cannot test every possible combination of foof settings, web pages, operating system, Firefox version, and additional plugins. A forum on the foof site and email links both allow users to contact us with problems. Any reported problems can then feed back into our test cases which are living documents and regularly updated.
Tags: firefox, qa, quality assurance