# How To Run A Beta Test... Or Not? Originally published on 13 September 2004 at http://www.namesuppressed.com/syneryder/2008/treo-650-mp3-problem.shtml ---------------------------------------------------------------------- I had high hopes for this beta test. It was a 0.1 upgrade release of a product I'd been selling for 3 years. Most of the bugs had already been squashed, so I intended to spend most of my time surveying testers and soliciting ideas for improvements. I thought I could build word-of-mouth by using a large beta testing team. The expectation was that this would be a very quick and simple beta. Oh man. How wrong could I have been? Several aspects of the test went okay. But I decided to make changes to how I run my beta tests, and that resulted in problems. These are some lessons I learned from the latest test. ## Build a huge database of beta candidates One of the things namesuppressed has always done right is build a database of beta testing candidates. It tracks contact info, demographic data, hardware and software, and other metadata. I'm glad for that - it's clear you need a *lot* of candidates just to find a good testing team to choose from. Asking candidates to complete an application form also helps to weed out testers who are only interested in freebies. 35% of candidates in the database were deemed ineligible for the test. Eligibility criteria included compatible software/hardware, a history of answering emails, and a history of providing feedback. That left 65% of the database to choose from. Of the invitations I sent to potential candidates: * 22% of the emails sent to candidates bounced * 2.5% of candidates declined the invitation * Only 40% of invitations sent resulted in acceptances * The rest (more than 35%) never replied. ## Never roll your own if you don't have to I decided to use the free mailing list software provided by my webhost, even though it had some drawbacks. I thought I could overcome the drawbacks by writing my own software to compensate for them. In fact, I already sell software that does this. I thought I'd make a couple of modifications to the program I sell and all would be fine.... Uh, no. The extra software & modifications I wrote caused major problems. People started getting two copies of every email, it messed with HTML emails, and even modified them in ways that triggered spam filters. Hotmail users never received any emails until the problem was fixed - and then they were surprised by a sudden flood of messages. Fixing those problems was one of the most stressful parts of the whole beta. I should have just started a free discussion group on Yahoo Groups, or purchased an account with Topica. Both are proven solutions. By writing my own software, I wasted a lot of time that could have been better spent. But on the bright side, I fixed some bugs in my own software. ## Make your testers opt-in themselves Before the test began, I asked all testers personally if they would like to join an email discussion group for the beta test. Amazingly, everyone agreed - even I hadn't expected that. However, it soon deteriorated. As soon as the test began, some testers unsubscribed from the list straight away. More testers complained later and unsubscribed (or asked to be unsubscribed). * About 10% of beta testers unsubscribed as soon as the test began * Another 10% asked to unsubscribe during the test * Some testers couldn't find the unsubscribe link A problem was that some testers didn't recognize the emails when they first started receiving them. Even though the list was "confirmed opt-in" (we had written confirmation from each testers email address), it would be better if testers had performed some action to opt-in (eg clicking a weblink). This would help them realize they were joining an email list. Also, explaining how to identify the emails may be helpful (eg "The subject line of all beta test emails will begin with [betatest]" or something similar). ## Expect the unforeseen It's very rare for a beta test to go exactly as you plan. The whole idea of a beta test is to locate problems you couldn't find yourself. That approach needs to be applied to the beta testing process too. * About 10% of testers had personal issues that restricted their ability to test. * Expect the beta test to take about twice as long as planned... so if you think you can get through with just 40 hours work, expect to take 80 hours instead. ## Organize several beta testing groups Anyone who has run beta tests before will know that it can be hard to keep enthusiasm up for the duration of the testing period. The best way to curb this is to bring in new groups of testers at regular intervals (every second beta). I didn't have enough candidates to try that this time. However, it means I can show you a graph of enthusiasm levels by measuring the frequency of messages throughout the testing period: Chart showing frequency of beta tester feedback http://www.namesuppressed.com/syneryder/synegfx/beta-email.gif Notice that enthusiasm is highest at the very start, and is maintained for about a week. It is renewed slightly for the second beta, but doesn't last much further than that. This is not a criticism of the beta testers in any way, it's just something to be expected. ## Be exceptionally clear about expectations and etiquette Our testers were confused about the purpose of our beta-tester email list. Here's what some of them thought it was for: **Announcements Only** Some testers thought the list would only include announcements of new beta downloads. We explained in the beta invitations that it was a discussion group they could participate in, but apparently it wasn't clear to everyone. **Bug Reports Only** Some testers thought the list was for making bug reports only. They thought that telling everyone the bugs would reduce duplicate bug reports. It's a nice idea, but when you have hundreds of messages it's difficult for everyone to keep track. You either get lots of duplicate reports anyway, or lots of people who don't report their bugs because "someone else probably found that bug already". **Socializing** Lots of testers thought the list was so they could talk to other testers about anything they wanted. Actually, we encouraged this idea, thinking it would make testers feel comfortable. It didn't quite work - some felt comfortable, others felt alienated. **Tutorials** Some testers thought the purpose was to share images created using the program we were testing, and teach others how to create those images. I hadn't anticipated that, and it was too late to accommodate it. Tutorial lists involve lots of large attachments, and group members with dialup connections or small email in-boxes couldn't handle it. Also, some people like tutorial lists while other people really dislike them, causing a rift in the beta group. So, what was our list really meant to be? A combination of the above - we announced all new betas on the list, expected bugs to be reported to the list, and expected some off topic chat... even the occasional picture post. But we didn't make this clear in our initial beta test invitations, so no one knew what to expect or what the boundaries were. I guess that's because we didn't know where to set the boundaries either. ## Manage Conflicts Within The Testing Group On the surface, everything was fine - the test group seemed friendly, very active and quite productive. Behind the scenes, things were falling apart. I received angry and upset emails from people who were frustrated by the volume of emails, frustrated at levels of "off topic" discussion, people who felt shy or intimidated by others testers, and even people who just didn't get on with the other testers. Some statistics: Chart showing beta tester emotions http://www.namesuppressed.com/syneryder/synegfx/beta-positive.gif * 66% of beta testers had some kind of negative experience. * 33% of beta testers said they felt intimidated. * 25% said they felt anger during the beta test. I'm not sure what I needed to do to fix this. Certainly I needed a better understanding of how to manage virtual communities, and how to develop a stable culture within the group. Perhaps I needed to set up two communities with different rules. ## Take a final beta test survey I was disappointed with the lack of response to my final beta tester survey. Testers were told that completing the survey was a necessary part of testing, even in the beta invitations we sent. However, we got a low survey response rate: * 36% responded to the first survey request * Another 18% responded when we sent a reminder. * Overall, just 55% responded to the final survey The surveys are extremely important. They provide measurable feedback on pricing, product features and general opinions of the product. We use the data to calculate maximal profit curves and select features to add in later versions. It's crucial information, so dropping the surveys isn't an option. Another reason for the surveys is to elicit feedback from quiet testers. There are always some testers who never send bug reports or talk on the mailing list. Many of them will respond to an anonymous web survey though. The feedback is useful and often brutally honest - exactly what you want. So, how to increase the response rate?: **Explain that completing the survey is required** You need to use the word "required" and make it stand out. Some testers told me that they'd thought the survey was optional. **Offer the survey early on** Perhaps if the surveys are given out when enthusiasm is at its highest, I would have had a better response rate. As it was, I handed out the surveys during Beta 3, the time of lowest enthusiasm. **Double the number of testers** I've had a 50-55% survey rate on at least two beta surveys. By doubling the number of testers, I hope that I would get twice the number of responses. **Only give an unlocking code once the survey is completed** That's probably the best solution, and should at least demonstrate how many testers are interested in the program. But depending how it's done, it may compromise the anonymity of the survey, and that may reduce the quality of answers that you get. ## In summary: what should we do in future? **Use proven software solutions.** If you need to use software in your testing, use a pre-written program. Don't write your own unless necessary. You should only be debugging one program in your tests. **Set clear expectations and boundaries.** Tell testers you expect them to give you feedback, and set a minimum frequency (eg at least one email a week). Let them know if chit-chat is okay, and how much is acceptable. Set maximum email attachment sizes & frequency. List anything else you can think of, and tweak the expectations as you get feedback/complaints from testers. **Recruit 10 times as many people as you need.** If you want a database with 30 really good beta testers in it, you'll need *at least* 300 people to complete your beta tester application form. And don't forget to start a database to keep track of them all. **Invite 6 times as many testers as you need.** Nope, this isn't the same as above... this is saying that when you choose the beta testers from your database and invite them into your beta team, invite 6 times as many as you need. If you want 10 survey responses, you should probably invite 60 to test. This is because only 40% accept the invitation, 10% have unforeseen problems, another 80% of that continue participating, and 50% of that provide feedback.... you need to cover all these. **Keep to a fixed schedule.** I planned to release a new beta every 7 days, and judging from the enthusiasm graph this would have been ideal. 2 week intervals would have been too long. **Introduce new testers every second beta.** By the time the 3rd beta came around, everyone was understandably jaded. **Act when enthusiasm is highest.** Get all the important jobs done very early on when enthusiasm is at its highest. **Survey your testers.** It's better to have some feedback than no feedback at all. Survey them when you deliver their 2nd beta. ## Project Statistics Name: Softener 1.20 Duration: 4 weeks [31 days, 120 hours work] Development Platforms: Windows 98SE, Windows XP Professional Release Platforms: Windows 95/98/98SE/ME/2000/XP Lines Of Code: 3239 (Softener = 1661, nsPSPlugin = 1578) Development Tools: Textpad 4.6.2, Borland C++ Builder 3, FilterMeister 0.4.21, Ghost Installer 3.7, Ghost Installer 4.1, PADGen, Resource Hacker, XVI Hex Editor, Microsoft Virtual PC 2004, MySQL 3.23.44, MySQL Admin, MySQL Front 2.5, Beyond Compare 2, Jasc Paint Shop Pro 7, Jasc Paint Shop Pro 8, Jasc Paint Shop Pro 9 Beta, Jasc Paint Shop Pro Studio Beta, Adobe Photoshop 6 Tryout, Adobe Photoshop CS Tryout, Megalux Ultimate X 1.3, Microsoft Wordpad, ezmlm, namesuppressed WebScriber, other bespoke namesuppressed software ---------------------------------------------------------------------- (C) 2004 Kohan Ikin / http://www.namesuppressed.com/syneryder/ ----------------------------------------------------------------------