Path: news1.ucsd.edu!ihnp4.ucsd.edu!agate!howland.reston.ans.net!newsfeed.internetmci.com!in2.uu.net!news.eng.convex.com!not-for-mail
From: faught@convex.com (Danny Faught)
Newsgroups: comp.software.testing,comp.software-eng,comp.answers,news.answers
Subject: comp.software.testing FAQ
Supersedes: <cst_818194981@convex.com>
Followup-To: comp.software.testing
Date: 19 Dec 1995 01:00:06 -0600
Organization: Engineering, Convex Computer Corporation, Richardson, Tx USA
Lines: 595
Approved: news-answers-request@MIT.EDU
Expires: 17 Jan 1996 07:00:04 GMT
Message-ID: <cst_819356404@convex.com>
NNTP-Posting-Host: zeppelin.convex.com
Summary: read me before posting to comp.software.testing
Keywords: FAQ archives conferences books periodicals products organizations
Xref: news1.ucsd.edu comp.software.testing:6316 comp.software-eng:28585 comp.answers:13168 news.answers:51401
Archive-name: software-eng/testing-faq
Posting-Frequency: semi-monthly
Last-modified: 1995/12/16
comp.software.testing FAQ
and information resource
------------------------------
Subject: 1. Introduction
This is the Frequently Asked Questions list for comp.software.testing.
Well, it's really more of an information resource for
comp.software.testing folks and software testers in general. If we
need to address any questions that get asked too often, I'll add a
section for that.
The FAQ has been approved for the rtfm archives! This means it will
now be posted to comp.answers and news.answers. If your site it set
up properly, the most recent version of this FAQ should always be
available in these groups. Also, you'll be able to retrieve it from
the various rtfm ftp sites under ".../comp/software-eng/testing-faq",
if I'm guessing right. See the FAQ "Introduction to the *.answers
newsgroups" in news.announce.newusers for more information.
The FAQ is still under construction (and like most FAQs, it always
will be). There are several sections that still need feedback. I'm
maintaining this FAQ to collect a consensus from the
comp.software.testing community, so it depends on feedback for it to
take shape.
Herein, unless otherwise stated, "I" refers to Danny Faught,
faught@convex.com
------------------------------
Subject: 2. Table of Contents
1. Introduction
2. Table of Contents
3. Charter
4. Archives and mailing lists
5. Other FAQs
6. Product reviews
7. Netiquette
8. Conferences
9. Periodicals
10. Books
11. Bibliographic resources
12. Organizations
------------------------------
Subject: 3. Charter
If you're new to Usenet, please read through the FAQs in
news.announce.newusers and hang out in news.newusers.questions for a
while before you consider posting.
The original charter for comp.software.testing can be found at
<http://tsunami.jpl.nasa.gov/TEL/docs/cst-charter.html>. Here's an
excerpt:
CONTENTS:
For the purpose of this discussion, we will assume that a system is
comprised of hardware, software, people, and procedures. The
proposed group should be chartered to include discussions
characterized by the following list of subjects, compiled from
preliminary discussions:
system test automation
system test planning
system test optimization (e.g. minimize regression testing)
test tools
test metrics
measurement technology
nature of testing under various development models
(e.g. object-oriented, real time, etc...)
testing in a rapid prototyping environment (i.e. sans spec)
relationship of various classes/types of tests to requirements, ...
conference and symposium announcements
the definition of "software testing" and its relationship to
SQA and debugging.
requirements traceability
risk-based testing
the most useful form of specifications and functional
requirements from the tester's point of view.
testing techniques, e.g., structured testing using control
flowgraphs and basis path testing, equivalence class
partitioning, boundary value analysis, cause-effect graphing,
path predicate testing, data flow testing, program slices,
data object state and usage analysis, data flow anomaly
analysis, and sensitivity analysis.
test coverage criteria, e.g., statement (C1), path, branch (C2),
module (S0), and call-pair (S1) coverage criteria.
All levels of testing for both hardware and software should be
considered appropriate subject matter. While it is likely that the
group will take on a software flavor early on, we should try to
generate interest and input from people with hardware and system
testing experience and perspective. It is especially important that
we foster discussion of *system* level testing issues, since this
is the weakest area of our collective knowledge.
------------------------------
Subject: 4. Archives and mailing lists
A searchable archive of every message ever posted to c.s.t. continues to
operate at
<http://tel.jpl.nasa.gov/bin/browse>
Articles are also available a month at a time in "mbox" format (digest
coming soon...) from:
<ftp://tel.jpl.nasa.gov/pub/cst-archive>
There's a mailing list gateway for comp.software.testing. To subscribe,
send mail to majordomo@tel.jpl.nasa.gov with "subscribe cst" in the body
of the message. To receive the articles periodically in digest
format, use "subscribe cst-digest" instead.
I maintain a mailing list for people who like to talk about the
implementation details of OS testing and test tools. Mail
swtest-discuss-info@convex.com with "subscribe swtest-discuss" in the
body of the message to subscribe.
There is an ISO9000 mailing list. Mail to Listserv@VM1.nodak.edu with
"subscribe iso9000 (your name)" in the body of the message, or
"Signoff iso9000" to unsubscribe.
There is a mailing list for Microsoft Test. To join, mail to
majordomo@eskimo.com with "subscribe mt_info" in the body of the mail.
------------------------------
Subject: 5. Other FAQs
Brian Marick maintains the following FAQs and posts them periodically:
Testing Contractors and Consultants
Testing Courses
Testing Tool Suppliers
Please consult the appropriate FAQ before before asking questions
about the information already contained therein. The Testing Tool
Suppliers FAQ also contains a nice list of the available software test
tools.
These FAQs are also available at <http://www.stlabs.com/marick/root.htm>
and <ftp://cs.uiuc.edu/pub/testing/faqs/>. Brian is working on getting
his FAQs approved for news.answers and the standard repositories that
go with it.
Also, since software testing is a subdiscipline of software engineering,
you might find FAQs posted to comp.software-eng useful. They can
be found at <ftp://rtfm.mit.edu/pub/usenet/comp.software-eng/>.
------------------------------
Subject: 6. Product reviews
It has been proposed that we start an archive of product reviews, but
we're suffering from a lack of anything to archive. I've talked with
the USAF Software Technology Support Center and they also report that
people don't seem to have time to contribute product reviews. See the
Testing Tools Suppliers FAQ for information about existing sources for
product reviews.
------------------------------
Subject: 7. Netiquette
There are classes of postings that are of interest to some readers, but
others don't welcome them at all. For unsolicited advertisements and
commercial product announcements, I propose that they should start the
subject line with "AD:". Job postings could start with "JOB:".
Someone has also proposed identifying conference announcements. Maybe
we could use "ANNOUNCE:". But conference announcements are typically
cross-posted to several groups, so it would be harder to convince the
posters to follow the convention for our group.
Also, I'd like for vendor representatives to clearly identify when
they are representing their company. While some people mention that
they have no financial ties to a product when they give a
recommendation, I'd like to put the burden on the vendors to
identify when they *are* representing their company.
It's important to understand that these would be conventions, and
they'll never catch on unless most people agree with them. So please
post your comments on this proposal.
People have also expressed concern about postings that are totally
off-topic. These postings are typically "spam" postings that go to
most of the thousands of newsgroups that exist. There is very little
that a group can do proactively to prevent them, besides recruiting a
moderator (and I don't think that would be appropriate for
comp.software.testing). See news.admin.net-abuse.announce for more
information.
------------------------------
Subject: 8. Conferences
The Testing Techniques Newsletter publishes specific conference
information a few months at a time.
Comments in this section are by Boris Beizer.
Conferences: All of these conferences have published proceedings.
Generally, materials presented at these conferences and in their proceedings
lead the publication in regular journals (above) by about 2-3 years.
1. ISSTA (Annual), theory, academic, leading edge practitioners (see 3b
in the periodicals section.)
2. IEEE Conference on Software Engineering. Spring, world-wide.
Technical. Primary source after ISSTA for leading edge results.
3. Quality Week. Annual, San Francisco. Biggest Conference on Testing and
QA. Typically 700+. Many vendors. Good balance between
technical/theoretical and practitioners. Very broad base.
Workshops. Software Research Institute: 625 Third Street, San
Francisco, CA 94107-1997: qw@soft.com End of May.
4. Pacific Northwest Quality Conference. Annual, Portland Oregon, End
September. Definitely not a regional conference, despite the
name. Very broad from technical/research to practitioners. A
shade more academic and leading edge participation than quality
week. Terri Moore, Conference Coordinator. Pacific Agenda, 2425
NW Overton Str., (PO BOX 10142) Portland, OR 97210.
terri_moore@aortel.org
5. International Conference On Software Testing. Washington DC, Begin June.
13th annual coming up. Less technical and more practitioner/management
than Quality Week. Smaller, 400+ . June 10-13. Workshops. USPDI:
uspdi@clark.net
6. STAR. Software Quality Engineering. Also EuroSTAR. May, Various
locations. About the same target as the International Conference (3)
above. Comparable level and interest. EuroSTAR is it in Europe. SQE
1-800-423-8378. Workshops.
7. QAI International Conference. More of a tutorial/workshop than a
conference. Newbie orientation. Fall, Orlando. Quality
Assurance Institute, 407-363-1111.
8. IEEE Compsac, world-wide, various locations. Fall. Typical IEEE
technical conference with a substantial number of papers of direct
interest to testing and QA. Not as many as IEEE Software
Engineering, though.
There are about a dozen other annual conferences with a substantial
testing content: Korea, Japan, Germany, to name a few. In addition,
there are about two-dozen "conferences", privately sponsored by various
individuals and organizations. These are not real conferences in that
there is no formal review process. Speakers are invited by the
conference organizer. Often, a heavy tutorial content. These are
usually small ( under 100 participants, very few vendors). They range,
depending on the organizer, from superb to fair-value, to outright
rip-off. Sorry. I won't play 20 questions on this one.
(end of Beizer's comments)
Also of note:
The International Conference on Software Quality, sponsored by the
ASQC. The fifth ICSQ was held October 23-26, 1995, in Austin,
Texas.
------------------------------
Subject: 9. Periodicals
Comments in this section are by Boris Beizer
1. IEEE Transactions on Software Engineering. (monthly)
The most prestigious journal for testing stuff. Volume 1 number 1
published the landmark article by Goodenough and Gerhart. Almost
every issue since has had papers on testing and quality
assurance. I have all the issues back to V1 #1. Much of that
stuff is still relevant.
2. ACM TOSEM (Transactions on Software Engineering Methodology)-Quarterly.
Relatively new journal (1992). Has prestigious editorial
board. Somewhat more theoretical than IEEE SE. In all ways,
equivalent.
3a. ACM SIGSOFT Software Engineering Notes. (Special Interest group on
Software Engineering) (Monthly)
Goes back to May 1976. This is the place where people publish
stuff to stake out claims. It is not as strictly refereed as IEEE
SE or TOSEM, but some overview is provide to keep out the obvious
crap. Important because there is a two or three year waiting list
for publication in IEEE SE or ACM TOSEM. People who know this
field read ACM SIGSOFT for advanced information on what will
be published "officially" in IEEE SE and ACM TOSEM. You read an
article and contact the author directly for more information.
3b. Annual ISSTA conference proceedings. ISSTA is the annual theoretical/
Academic conference devoted to testing. This is the most
prestigious place to publish new results in testing theory and
software reliability theory.
4. IEEE Software. (Six times a year).
Rarely the latest stuff. More like surveys and overviews once a
subfield has become established. Refereed, generally high
standards. Mostly overviews, but occasionally new stuff.
Additional comments from Scott Killops (Scott_B_Killops@ccm.jf.intel.com):
"You might mention in your blurb about "IEEE Software" the
regular "Quality Time" feature. This is wide ranging and
generally quite useful. I've often thought that a collection
of all of the "Quality Time" articles published to date would
make a worthwhile book."
5. Software Testing, Verification, and Reliability. Quarterly.
International journal published by John Wiley and Sons.
Prestigious international editorial board. High quality original
articles and excellent surveys. Comparable in scope to IEEE SE
and ACM TOSEM, but sharply focused on testing, verification, and
reliability.
6. ACM Computing Surveys. (Quarterly).
Not specific to testing and QA, but contains the most prestigious
survey articles in the field. Typically, only when a subfield is
well established. The authors are usually authorities. Articles
are long and comprehensive. When an ACM Survey on a topic
appears, it usually means that the field has matured to the point
where it is possible to write meaningful books.
7. Communications of the ACM. (Monthly)
Survey articles and overviews. Sometimes (rarely) original stuff.
More academic and foundational/theory oriented than IEEE Software,
but generally same level.
8. American Programmer. Monthly, Cutter Information Corp. 37 Broadway,
Suite 1, Arlingon MA 02174-5552 (617-648-8702).
Private journal with prestigious editorial board (Larry
Constantine, Bill Curtiss Tom DeMarco, Capers Jones, Ed Yourdon.
Non technical, philosophical and overviews. Management
orientation. Essentially a good newsletter.
9a. Software Maintenance News. (6 times a year) Software Maintenance News
Inc. B10, Suite 237, 4546 El Camino Real, Los Altos CA, 94022.
General on testing and quality assurance and not just
maintenance. Invited articles, usually in an overview/survey
style with minimal technical prerequisites by authors who may have
previously published more advanced versions of this material in
IEEE SE or TOSEM. Essentially a good newsletter.
9b. Software Management Technology Reference Guide (annual)
Editor same as Software Maintenance News--Nicholas Zvegintzov. A
superb guide to tools, vendors, resources, edited by the unique
Nicholas Zvegintzov in his inimitable style. Really packed.
Probably the highest signal to noise ratio in the industry.
73670.22227@compuserve.com
10. Software QA Quarterly. (Quarterly). Ridgetop Publishing, Ltd..
PO Box 379, Silverton, Oregon, 97381-0379. Phone/FAX 503-829-6806,
email: ridgetop@teleport.com.
QA, testing, Practitioners, management. Somewhat conservative.
Myers' The Art of Software Testing (1978) was reviewed in Winter
1994 issue.
11. The Software Practitioner (news letter, 6 times a year).
Computing Trends, 350 Dalkeith Avenue, Los Angeles Ca. 90049
310-440-9982. Excellent newsletter. Non technical, management,
overviews.
12. Crosstalk. Journal of Defense Software Engineering. Software Technology
Support Center, Ogden AFB UT 84056. 801-777-8057.
(6 times a year) Heavy DoD orientation. Practitioner/Management/
Philosophical. Broad based on software engineering, including T V&V.
13. Other Journals. I'm sure I've offended several editors. There are
Journals concerned with Testing, QA, and Software Engineering
published in Japan, Korea, Germany, and Australia. Other
Journals, such as IEEE-Transaction Computers, ACM SIGPLAN, ACM
SIGARCH, ACM SIGSIM, ACM SIGMETRICS, ACM OOPSLA, all publish
material of interest to testing, especially insofar as hardware
testing results and theory apply to software
testing; also language issues vis-a-vis testing.
(end of Beizer's comments)
commercial newsletters-
STL Report: A Newsletter about Quality Assurance for Market-Driven Software
bi-monthly, subscriptions are free
for subscriptions: Trina Primitivo (206) 682-5832, ext. 138,
trinap@stlabs.com
editorial concerns: Prue Cuper at (206) 682-5832, ext. 137,
pruec@stlabs.com
TESTING TECHNIQUES NEWSLETTER
TTN/Online published monthly, TTN hardcopy is biannual
(415) 957-1441, (800) 942-SOFT, FAX: (415) 957-0730
ttn@soft.com
------------------------------
Subject: 10. Books
Here's a book list based on one posted by Scott Killops quite a while
ago. I'm sure there are many books still missing from the list - please
point them out. It has been proposed that we add the following
information for each:
-Audience (i.e. beginner, intermediate, expert level needed for comprehension)
-Book topics (i.e. black box, white box, sub-system, oo, etc)
See also the Computer Literacy Book Shop at <http://www.clbooks.com/>
to search for books and see reviews of some.
Beizer, Boris. Software System Testing and Quality Assurance.
New York: Von Nostrand Reinhold, 1984, ISBN 0-442-21306-9.
Beizer, Boris. Software Testing Techniques, Second Edition.
New York: Von Nostrand Reinhold, 1990, ISBN 0-442-20672-0.
DeMillo, R.A., McCracken, W.M., Martin, R.J., Passafiume, J.F.
Software Testing and Evaluation. Menlo Park, CA: Benjamin/Cummings,
1987, ISBN 0-8053-2535-2.
Deutsch, M.S. Software Verification and Validation - Realistic
Project Approaches. Englewood Cliffs, NJ: Prentice Hall, 1982,
ISBN 0-13-822072-7.
Dunn, Robert H. Software quality: concepts and plans. Englewood
Cliffs, NJ: Prentice Hall, 1990, ISBN 0-13-820283-4.
Friedman, Michael and Jeffrey M. Voas. "Software Assessment:
Reliability, Safety, Testability". New York: Wiley Press, 1995,
ISBN 0-471-01009-X
Hetzel, B. The Complete Guide to Software Testing, Second Edition
Wellesley, MA: QED Information Sciences, 1988, ISBN 0-89435-242-3.
Ince, Darrel. An introduction to software quality assurance and its
implementation. London, New York: McGraw Hill, 1994, ISBN
0-07-707924-8.
Kaner, Cem., Falk, J., Nguyen, H.Q. Testing Computer Software,
Second Edition. New York: Von Nostrand Reinhold, 1993, ISBN
0-442-01361-2.
Marick, Brian. "The Craft of Software Testing". Englewood Cliffs,
NJ: Prentice Hall, 1995. ISBN 0-13-177411-5.
Marks, D.M. Testing Very Big Systems. New York: McGraw Hill,
1992, ISBN 0-07-040433-X.
McConell, Steve. Code Complete: a practical handbook of software
construction. Redmond, Washington: Microsoft Press, 1993, ISBN
1-55-615484-4.
Mosley, D.J. The Handbook of MIS Application Software Testing.
Englewood Cliffs, NJ: Prentice Hall, 1993, ISBN 0-13-907007-9.
Myers, Glenford J. The Art of Software Testing. New York: John Wiley
and Sons, 1979, ISBN 0-471-04328-1.
Ould, M.A., Unwin, C., eds. Testing in Software Development.
Cambridge: Cambridge University Press, 1986, ISBN 0-521-33786-0.
Parrington, Normal and Marc Roper. Understanding Software Testing. Ellis
Horwood Limited, 1989. ISBN 0-7458-0533-7 (Ellis Horwood Limited) ISBN
0-470-21462-7 (Halsted Press).
Royer, T.C. Software Testing Management - Life on the Critical
Path. Englewood Cliffs, NJ: Prentice Hall, 1993, ISBN 0-13-532987-6.
Wilson, Rodney. UNIX Test Tools and Benchmarks. Prentice Hall, 1995.
ISBN 0-13-125634-3.
------------------------------
Subject: 11. Bibliographic resources
Comments by Boris Beizer.
1. ACM Reviews -- Montly. the primary review journal. Most good papers on
testing and quality assurance are eventually reviewed here. Extensive
evaluations and criticisms by excellent reviewer pool.
2. ACM Guide to Computing Literature. Annual. Does not contain reviews.
Most comprehensive published literature guide for authors, titles, and
subject.
3. Key Abstracts in Software Engineering. Published by INSPEC, but
available through IEEE membership. Monthly. About halfway between ACM
Reviews and ACM Guide to the literature. Thorough, best articles. However,
only author abstracts and no evaluation or criticism.
------------------------------
Subject: 12. Organizations
American Society for Quality Control
611 E. Wisconsin Ave.
P.O. Box 3005
Milwaukee, WI 53201-3005
800-248-1946, (414) 272-8575
<http:/www.asqc.org>
The ASQC provides CQE certification, which is apparently slanted
more toward manufacturing than software. The ASQC press has many
titles that may be of interest to software testers. Stolen from a
conference announcement:
> Founded in 1946, the American Society for Quality Control (ASQC)
> provides a variety of professional, educational, and informational
> programs reflecting the changing needs of business and industry.
> Headquartered in Milwaukee, WI, ASQC has been the leading quality
> improvement organization in the United States for almost 50 years.
> The Society is composed of more than 135,000 individual members and
> 1,000 sustaining members worldwide. ASQC's vision is to be the
> world's recognized champion and leading authority on all issues
> related to quality. The mission of ASQC is to facilitate
> continuous improvement and increased customer satisfaction by
> identifying, communicating, and promoting the use of quality
> principles, concepts, and technologies.
>
> ASQC's Software Division
>
> ASQC's Software Division is comprised of more than 5,000 members
> including software quality professionals and software engineers
> interested in applying quality principles to the field of software
> development. The Division develops a software engineer
> certification program, publishes a quarterly newsletter, works with
> the Software Quality System Registration Committee on establishing
> an ISO 9000 Software Registration Program in the United States,
> interacts with other professional software organizations such as
> the IEEE and the ACM (Association for Computing Machinery), and is
> involved in many other activities.
Quality Assurance Institute
7575 Dr. Phillips Blvd., Suite 350
Orlando Florida 32819
Phone: (407) 363-1111
Fax: (407) 363-1112
The QAI publishes The Journal of the Quality Assurance Institute
monthly.
They provide CQA certification. Steve Driscoll (sdriscol@oclc.org)
wrote:
> The CQA is someone who has demonstrated a mastery of the Common
> Body of Knowledge(CBOK) for ensuring the quality of systems (which
> often include software). While ISO-9000 is considered part of the
> CBOK, it's actually independent of the CQA designation (there is a
> certification program for ISO auditors).
> ... while my life is still pretty much the same as it was prior to
> obtaining the CQA designation, I found that the reading and
> studying to prepare for the exam extremely beneficial in my duties
> as a Quality Analyst. I recommend it to anyone who is interested
> in producing quality software (which includes developers as well
> as quality analysts).
The Software Quality Institute At UT
(stolen from a conference announcement)
> The Software Quality Institute (SQI) at The University of Texas at
> Austin is a multidisciplinary partnership between UT-Austin and the
> software and information systems industries in Texas. It is
> recognized as a leading authority on and champion for software
> quality. SQI was founded in 1993 for the purpose of strengthening
> organizations to compete more successfully in global markets
> through sponsorship of seminars and conferences. An advisory group
> of 24 industry and academic representatives guides the Institute.
> Its popular programs include: a 13 week long Software Project
> Management Certificate Program, one- and two-day seminars offered
> to the public and as in-company contract courses, our "Software
> Quality Matters" quarterly newsletter that addresses issues of
> concern relative to quality issues, sponsorship of monthly forums
> for discussion of software issues (including the Software Process
> Improvement Network (A-SPIN), the Austin Software Executives' Group
> (ASEG), and the Austin Forum for Object-Oriented Technology
> (AFOOT), and on-line services which include a Worldwide Web
> Home Page and topical newsgroups.
The web page is <http://ceswww.utexas.edu/sqi/>. The newsgroups
seem to be reachable only at UT. For information on newsletter
subscriptions, contact info@sqi.utexas.edu.
Central Ohio Quality Assurance Association
Australian Software Quality Research Institute
Griffith University, Brisbane, Australia
<http://www-sqi.cit.gu.edu.au/>