Search the FAQ Archives

3 - A - B - C - D - E - F - G - H - I - J - K - L - M
N - O - P - Q - R - S - T - U - V - W - X - Y - Z
faqs.org - Internet FAQ Archives

comp.software.testing Frequently Asked Questions (FAQ)

(MultiPage )
[ Usenet FAQs | Web FAQs | Documents | RFC Index | Restaurant inspections ]
Archive-name: software-eng/testing-faq
Posting-Frequency: monthly
Last-modified: 2002/08/06
Version: $Revision: 3.9 $
Maintainer: cst-faq@crim.ca
URL: http://www.faqs.org/faqs/software-eng/testing-faq/
http://www.crim.ca/ctl/cst.FAQ.html
http://www.cigital.com/c.s.t.faq.html

See reader questions & answers on this topic! - Help others by sharing your knowledge
                      comp.software.testing FAQ
                      and information resource

This is the Frequently Asked Questions list for comp.software.testing.
This FAQ is cross-posted to comp.software-eng.  Please don't hesitate 
to send corrections, additions, and other comments to 
cst-faq@crim.ca.

Be sure to use the tag of FAQ: in the subject line, for example,
"Subject: FAQ: Modification request...."

The latest version of this FAQ may be retrieved using Usenet, the World
Wide Web, ftp, email, gopher, or even telnet.  See the FAQ "Introduction
to the *.answers newsgroups" in news.announce.newusers or
<ftp://rtfm.mit.edu/pub/usenet/news.answers/news-answers/introduction>
for details.  Please check the date above - if this file is more than a
month old, it is obsolete.  The main ftp site for the FAQ is
<ftp://rtfm.mit.edu/pub/usenet/news.answers/software-eng/testing-faq>.

The folks at CRIM Software Test Center sponsor a web site
for the FAQ at http://www.crim.ca/ctl/cst.FAQ.html.
The folks at Cigital sponsor a web site for the FAQ, at
<http://www.cigital.com/c.s.t.faq.html>.

This file is Copyright 2001, 2002  by Raymond Rivest.  Danny R. Faught is the
original author.  Permission is granted to distribute this file unedited
and in its entirety, provided the "Date" header is no more than one month
earlier than the date of distribution.  This information is provided
without any express or implied warranty.


Subject: 1. Table of Contents 1. Table of Contents 2. Change Log 3. What is this newsgroup about? 4. Netiquette 5. Beta testing 6. What other FAQs are relevant? 7. What other resources are available? 8. World Wide Web resources 9. What's the difference between QA and testing? 10. I'm looking for a test tool... 11. How do I find information about testing object-oriented programs? 12. How do I test web-related applications? 13. What is black box/white box testing? 14. What are unit, component and integration testing? 15. What's the difference between load and stress testing ? 16. Should we discuss bug tracking tools in this newsgroup? 17. What kind of salary should a tester make? 18. Where can I find sample test plans? 19. What is the best tester to developer ratio? 20. Conferences 21. Periodicals 22. Books 23. Bibliographic resources 24. Organizations 25. Contributors
Subject: 2. Change Log This section will highlight notable additions, deletions, and changes to the FAQ. Changed in version 3.: o Some minor cosmetic changes. o Added CRIM STC in section 24. Changed in version 3.8: o Added Extreme Programming mini-FAQ link in section 6. Changed in version 3.7: o Finally cleaned up Danny's e-mail to tejasconsulting :) o Removed "The Outlook" is no longer available from McCabe? (subject 22). o Added a commercial newsletter to subject 22: The Payne Report, http://www.cigital.com/paynereport/. o Changed the comp.software.config-mgmt url to http://www.daveeaton.com/scm/PMTools.html. o Added http://www.QALinks.com and http://www.TestingFAQs.org in subject 8. o Added Die Software Testing Website http://www.softwaretesting.de in suject 8. o Added LogiGear web page at http://www.qacity.com/ in subject 8. o Added Tejas Software Consulting Newsletter in subject 21. o Added a new subject tag in section 4 (netiquette) for job searcher (CV:). o Added a link to http://www.xprogramming.com/software.htm in subject 7 about eXtreme programming and testing. o Added two references about security testing in subject 7. o Added Quality Web Systems by Elfriede A. Dustin/Douglas McDiarmid in subject 12. o Updated a broken link for http://www.asq.org/pub/ in subject 21 o Added a quick reference defining what is a "smoke test" in subject 7. Changed in version 3.6: o Changed reference from www.dejanews.com to groups.google.com. Google acquired dejanews ! Changed in version 3.5: o Inserted a new Section 15 and moved the rest downward. What's the difference between load and stress testing ? Changed in version 3.4: o Reordered the Table of Content. Section 7 to 4, 10 to 5 and moved the rest downward to explain the purpose of comp.software.testing from the start. o Changed references of Danny Faught's FAQs from his home page to http://www.testingfaqs.org/ in Section 10 (a forgotten link). o Added TESTCOM conference to Section 19. o Added reference to the defect tracking tool FAQ in Section 15. o Added StickyMinds.com in Section 8. o Added a book reference in Section 12. o Added the StickyMinds.com template section in Section 17 o Added a reference to comp.software.measurement newsgroup in Section 7. Changed in version 3.3: o Added the mention of subject tag for cst-faq alias because of spams. o Changed references of Danny Faught's FAQs from his home page to http://www.testingfaqs.org/ o Added reference to BetaSoft's QAForums in section 5. Changed in version 3.2: o Changed references of Danny Faught's FAQs to his home page. o Changed Danny Faught's email. o Added LatinStar 2001 conference. o Changed STQE booklist to stickyminds. o Added a new URL for the html FAQ. Changed in version 3.1: o Changed references of cst-faq@cigital.com to cst-faq@crim.ca. Changed in version 3.0: o Head FAQ maintainer responsibilities transferred from Danny Faught to Raymond Rivest. Have fun with it, Raymond! Changed in version 2.7: o Delegated some items to Cigital's SRM Hotlist, so I don't have to maintain them in two different places. This includes the list of mailing lists (subject 5), web pointers and dates for conferences (subject 19), and the entire book list (subject 21). This will also help to make the FAQ somewhat less bulky. o Removed the Triangle Quality Council, which is being phased out. Also removed Association for Software Testing Australia, which no longer seems to exist. (subject 23) Changed in version 2.6: o Reliable Software Technologies is now Cigital, Inc. Changed all RST references to Cigital. o Reflected the change from American Programmer to Cutter IT Journal. (subject 20) o Removed information about the cst-improve list, which is no longer in use. (subject 5) o Updated contact information for Software Practitioner. (subject 20)
Subject: 3. What is this newsgroup about? If you're new to Usenet, please read through the FAQs in news.announce.newusers and hang out in news.newusers.questions for a while before you consider posting. If you do not know how to subscribe to comp.software.testing, you first need to get Usenet access from an Internet service provider or your company, and you need newsreader software. Different service providers support different newsreaders, so there is no general answer. Ask your service provider or local system administrator to help. If all else fails, use a web browser to access Usenet at <http://groups.google.com/groups ?hl=en&lr=&safe=off&group=comp.software.testing>. The original charter for comp.software.testing can be found at <http://tsunami.jpl.nasa.gov/TEL/docs/cst-charter.html>. Here's an excerpt (with corrections): CONTENTS: For the purpose of this discussion, we will assume that a system is composed of hardware, software, people, and procedures. The proposed group should be chartered to include discussions characterized by the following list of subjects, compiled from preliminary discussions: system test automation system test planning system test optimization (e.g. minimize regression testing) test tools test metrics measurement technology nature of testing under various development models (e.g. object-oriented, real time, etc...) testing in a rapid prototyping environment (i.e. sans spec) relationship of various classes/types of tests to requirements, ... conference and symposium announcements the definition of "software testing" and its relationship to SQA and debugging. requirements traceability risk-based testing the most useful form of specifications and functional requirements from the tester's point of view. testing techniques, e.g., structured testing using control flowgraphs and basis path testing, equivalence class partitioning, boundary value analysis, cause-effect graphing, path predicate testing, data flow testing, program slices, data object state and usage analysis, data flow anomaly analysis, and sensitivity analysis. test coverage criteria, e.g., statement (C0), path, branch (C1), module (S0), and call-pair (S1) coverage criteria. All levels of testing for both hardware and software should be considered appropriate subject matter. While it is likely that the group will take on a software flavor early on, we should try to generate interest and input from people with hardware and system testing experience and perspective. It is especially important that we foster discussion of *system* level testing issues, since this is the weakest area of our collective knowledge.
Subject: 4. Netiquette Did I mention that you should read the FAQs in news.announce.newusers if you haven't done so lately? Also, you can reduce the chances of embarrassing yourself by reading comp.software.testing for a while before posting. At the very least, scan all the articles in the group currently on your server to make sure your question hasn't already been asked and answered. Please help the newsgroup's participants find articles in the categories they're interested in. If your posting falls into the categories below, please put the three-letter tag at the beginning of your subject line. This will also increase the likelihood that those wanting to read your post will find it. For example, your subject line might read - "Subject: ADV: Widget Tester 6.0 released". JOB: Help wanted ads ANN: Announcements for new books, publications CFP: Call for papers or participants CNF: Conference announcements ADV: Other advertisements and commercial product announcements CV: Curriculum Vitae, looking for a job Job postings that are cross-posted to the various job-related newsgroups should not be posted to comp.software.testing, since the subject conventions for those groups do not allow them to be easily identified as job postings, and the misc.jobs FAQ (<ftp://rtfm.mit.edu/pub/usenet/news.answers/jobs/welcome-faq>) specifies that job postings should not be cross-posted between the job-related newsgroups and non job-related newsgroups. Some recruiters refuse to follow the job posting policy. If you don't want to read job postings, consider using a kill file to systematically kill articles from these recruiters. Many questions about a particular test tool only interest those readers who use that tool. Please make sure to put the name of the test tool in the subject line so readers can select the articles they read by looking at the subject. In general, please use a descriptive subject line. If you post an article and later decide you shouldn't have posted it, please cancel it. It is much better to correct the problem than to send *another* post to apologize for the first one. Situations where this would be appropriate include posting empty articles, multiple copies of the same article, and any goof where you'd like to correct something you posted shortly after it was sent. Look in your newsreader documentation for help with cancelling an article; some also have a "supercede" feature which makes it easy to correct a posting. While the cancel feature has been common for quite some time, some newer newsreaders unfortunately do not offer it. If this is the case, complain to the author, and consider using a more reasonable newsreader. Here's a manual method to cancel an article - start to follow up to the errant posting. Change the subject to "cmsg cancel <message-id>", where message-id is taken from the References: header or the attribution. The body of the message is not important. Post it, and the news system should interpret it as a cancel request rather than a real posting. You can only do this for an article you posted yourself. People have also expressed concern about postings that are totally off-topic. These postings are typically "spam" postings that go to most of the thousands of newsgroups that exist. There is very little that an unmoderated group can do proactively to prevent them. See news.admin.net-abuse.usenet and <http://www.tezcat.com/~gbyshenk/ive.been.spammed.html> for more information. Please don't post test messages to comp.software.testing. This group is for discussing software testing, not for testing your news software. If you want several automatic confirmations that your post worked, post a test message to misc.test. There are other test groups at various levels of the news hierarchy that all end in ".test". It's best to try a local one first.
Subject: 5. Beta testing Discussions about managing a beta test program are within the scope of this newsgroup. However, companies looking for beta testers, and aspiring beta testers looking for something to test, should go to comp.sources.testers instead. A possible exception is test tool vendors looking for beta testers. If you are looking for a career in software testing, comp.software.testing will welcome you. Just don't call it "beta testing". Beta testing is typically conducted by end users of a software product who are not paid a salary for their efforts.
Subject: 6. What other FAQs are relevant? Danny Faught maintains the following FAQs, originally created by Brian Marick: Testing Contractors and Consultants List Testing Courses List Testing Tool Supplier List These FAQs are available at <http://www.testingfaqs.org/>. Please consult the appropriate FAQ before asking questions about the information already contained therein. The Testing Tool Suppliers List also contains a nice list of the available software test tools. If you see anything that needs to be changed or added to these FAQs, please contact Danny Faught at faught@tejasconsulting.com so that they may continue to be useful to everyone. Since software testing is a subdiscipline of software engineering, you will probably find the FAQs posted to comp.software-eng useful. They can be found at <ftp://rtfm.mit.edu/pub/usenet/comp.software-eng/> or <http://www.qucis.queensu.ca/Software-Engineering/> The Problem Management Tools (bug tracking) FAQ is posted to comp.software.config-mgmt, and is also available on news.answers and the rtfm archives. The URL is <http://www.daveeaton.com/scm/PMTools.html>. Rick Hower maintains the "Software QA/Test Resource Center" at <http://www.softwareqatest.com>. It covers a broad range of questions about software testing. Related to new trends, eXtreme Programming, here's the mini-FAQ: <http://www.jera.com/techinfo/xpfaq.html>.
Subject: 7. What other resources are available? The public comp.software.testing archives and the mailing list gateway are no longer available. As an alternative, try Google groups at <comp.software.testing">http://groups.google.com/groups?hl=en&lr=&safe=off&group=comp.software.testing> Comp.software.testing articles are archived back to March, 1995. If you're interested in quality measures (metrics), than <news:comp.software.measurement>will interest you. However, I didn't find any FAQ for this newsgroup. If eXtreme testing is of interest, than <http://www.xprogramming.com/software.htm> will interest you. See also Ronald E. Jeffries's article at <http://www.xprogramming.com/publications/SP99%20Extreme%20for%20Web.pdf> http://www.cigital.com/hotlist/publications-ml.html>. BetaSoft hosts Web forums on most popular testing tools and techniques at <http://www.qaforums.com/>. A hot topic is network security. Here are two pretty good references for testing. <http://www.ideahamster.org/> <http://www.itprc.com/security.htm> A quick reference explaining what is a "smoke test" can be found at <http://www.construx.com/stevemcc/bp04.htm>.
Subject: 8. World Wide Web resources Here are a few web sites that each try to connect you to much of the software testing and quality information that is available on the web. The Cigital Software Risk Management Hotlist <http://www.cigital.com/hotlist/> http://www.mtsu.edu/~storm> http://www.soft.com/Institute/HotList/> Kerry Zallar's software testing page <http://www.testingstuff.com/> Bret Pettichord's Software Testing Hotlist <http://www.io.com/~wazmo/qa/> StickyMinds.com Software Testing Area <http://www.StickyMinds.com/> Betasoft QA Links page <http://www.QALinks.com> Danny Faught's testing faqs page <http://www.TestingFAQs.org> Die Software Testing Website <http://www.softwaretesting.de> This website has lots of articles, book reviews, software testing related software for download and a discussion forum. Most of the website is in german language. All contents can be accessed free of charge. LogiGear's QACity.Com : Resources for Busy Testers <http://www.qacity.com/>
Subject: 9. What's the difference between QA and testing? Sharon Codrington asks: Please enlighten this young QA person as to the difference between QA & Testing. Am I right in thinking that QA is more a preventive thing, ensuring quality in the company and therefore the product rather than just testing the product for software bugs? And Bret Pettichord answers: DING. DING. DING. You are correct. Most testing jobs I see are nevertheless advertised as "QA". Some people suggest that QC is a better set of initials to describe testing. Don Mills writes: In my courses and my practice, I stick to the ANSI/IEEE definitions, which agree with what is generally held *outside* the software arena. The definitions boil down to: * TESTING means "quality control" * QUALITY CONTROL measures the quality of a product * QUALITY ASSURANCE measures the quality of processes used to create a quality product. Rick Hower addresses this question at <http://www.softwareqatest.com/qatfaq1.html>.
Subject: 10. I'm looking for a test tool... Did you read the Testing Tool Suppliers FAQ at <http://www.testingfaqs.org/>? It has a list of many of the available tools, plus a list of the commercial tools listings. While every question about test tools isn't answered there, it's a good start, and it'll at least allow you to ask a more directed question if you do post a question about test tools. When you do post a tools question, you should expect a few "check out my company's tool" responses from a few vendors. This rarely gives you any more information than is available in the tools FAQ, and which vendors respond at any given time varies widely. Often you'll find a vendor trying to drum up interest by posting the same thing in response to every current article that mentions tools. Keep in mind that your choices are much broader than those that happen to get posted.
Subject: 11. How do I find information about testing object-oriented programs? Based on information originally provided by Brian Marick: Books that include some discussion of testing OO programs: Binder, _Testing Object-Oriented Systems_ Jacobson, _Object-Oriented Software Engineering_ Marick, _The Craft of Software Testing_ McGregor, _Object-Oriented Software Development_ Siegel, _Object Oriented Software Testing_ There are a vast number of articles about testing OO programs. I suggest starting with the September 1994 Communications of the ACM, which is devoted to this topic. Chase references. The February 1996 issue of Object has a piece on system testing with use cases. Conferences like STAR and Quality Week and Pacific Northwest Software Quality Conference seem to always have papers on this topic. I suggest going to a conference and buttonholing people. Web sites with discussion: <http://www.cs.washington.edu/homes/gmurphy/testSTApp.html> <http://www.testing.com/> <http://www.toa.com> Courses on testing object-oriented code (one by Robert Binder, one by Ed Berard) are described in the Testing Courses FAQ. Also, a new addition is an offering from Software Quality Engineering.
Subject: 12. How do I test web-related applications? Rick Hower maintains a web page which includes a nice list of web testing tools, and some guidelines for testing web sites. <http://www.softwareqatest.com/qatweb1.html> Please contact him at rickhower@earthlink.net if you can offer additions or corrections to the list. One book that can be of help, "Testing Applications on the Web" by Hung Q. Nguyen and "The Web Testing Handbook" by Steven Splaine and Stefan P. Jaskiel. One recent addition is Quality Web Systems by Elfriede A. Dustin/Douglas McDiarmid. <http://www.StickyMinds.com/books.asp> It also contains a Tool Evaluation appendix, comparing the major tools (Mercury, Rational, Empirix, Compuware, Segue). This matrix consists of approx. 30 pages of detailed evaluation criteria.
Subject: 13. What is black box/white box testing? Black-box and white-box are test design methods. Black-box test design treats the system as a "black-box", so it doesn't explicitly use knowledge of the internal structure. Black-box test design is usually described as focusing on testing functional requirements. Synonyms for black-box include: behavioral, functional, opaque-box, and closed-box. White-box test design allows one to peek inside the "box", and it focuses specifically on using internal knowledge of the software to guide the selection of test data. Synonyms for white-box include: structural, glass-box and clear-box. While black-box and white-box are terms that are still in popular use, many people prefer the terms "behavioral" and "structural". Behavioral test design is slightly different from black-box test design because the use of internal knowledge isn't strictly forbidden, but it's still discouraged. In practice, it hasn't proven useful to use a single test design method. One has to use a mixture of different methods so that they aren't hindered by the limitations of a particular one. Some call this "gray-box" or "translucent-box" test design, but others wish we'd stop talking about boxes altogether. It is important to understand that these methods are used during the test design phase, and their influence is hard to see in the tests once they're implemented. Note that any level of testing (unit testing, system testing, etc.) can use any test design methods. Unit testing is usually associated with structural test design, but this is because testers usually don't have well-defined requirements at the unit level to validate. Definitions of these terms can be found in these references: Beizer, _Black Box Testing_, p. 8. Beizer, _Software Testing Techniques_, pp. 10-11. Daich, et al., _STSC Software Test Technologies Report_, pp. 198, 206. Kaner, et.al., _Testing Computer Software_, Second Edition, pp. 41-43. Mosley, _The Handbook of MIS Application Software Testing_, pp. 68-76. Myers, _The Art of Software Testing_, pp. 8-11. Wilson, _Unix Test Tools and Benchmarks_, pp. 298, 306
Subject: 14. What are unit, component and integration testing? The following definitions are from a posting by Boris Beizer on the topic of "integration testing" in the c.s.t. newsgroup. The definitions of integration tests are after Leung and White. Note that the definitions of unit, component, integration, and integration testing are recursive: Unit. The smallest compilable component. A unit typically is the work of one programmer (At least in principle). As defined, it does not include any called sub-components (for procedural languages) or communicating components in general. Unit Testing: in unit testing called components (or communicating components) are replaced with stubs, simulators, or trusted components. Calling components are replaced with drivers or trusted super-components. The unit is tested in isolation. component: a unit is a component. The integration of one or more components is a component. Note: The reason for "one or more" as contrasted to "Two or more" is to allow for components that call themselves recursively. component testing: the same as unit testing except that all stubs and simulators are replaced with the real thing. Two components (actually one or more) are said to be integrated when: a. They have been compiled, linked, and loaded together. b. They have successfully passed the integration tests at the interface between them. Thus, components A and B are integrated to create a new, larger, component (A,B). Note that this does not conflict with the idea of incremental integration -- it just means that A is a big component and B, the component added, is a small one. Integration testing: carrying out integration tests. Integration tests (After Leung and White) for procedural languages. This is easily generalized for OO languages by using the equivalent constructs for message passing. In the following, the word "call" is to be understood in the most general sense of a data flow and is not restricted to just formal subroutine calls and returns -- for example, passage of data through global data structures and/or the use of pointers. Let A and B be two components in which A calls B. Let Ta be the component level tests of A Let Tb be the component level tests of B Tab The tests in A's suite that cause A to call B. Tbsa The tests in B's suite for which it is possible to sensitize A -- the inputs are to A, not B. Tbsa + Tab == the integration test suite (+ = union). Note: Sensitize is a technical term. It means inputs that will cause a routine to go down a specified path. The inputs are to A. Not every input to A will cause A to traverse a path in which B is called. Tbsa is the set of tests which do cause A to follow a path in which B is called. The outcome of the test of B may or may not be affected. There have been variations on these definitions, but the key point is that it is pretty darn formal and there's a goodly hunk of testing theory, especially as concerns integration testing, OO testing, and regression testing, based on them. As to the difference between integration testing and system testing. System testing specifically goes after behaviors and bugs that are properties of the entire system as distinct from properties attributable to components (unless, of course, the component in question is the entire system). Examples of system testing issues: resource loss bugs, throughput bugs, performance, security, recovery, transaction synchronization bugs (often misnamed "timing bugs").
Subject: 15. What's the difference between load and stress testing ? Boris Beizer says: > One of the most common, but unfortunate misuse of terminology >is treating "load testing" and "stress testing" as synonymous. The >consequence of this ignorant semantic abuse is usually that the system >is neither properly "load tested" nor subjected to a meaningful stress >test. > >1. Stress testing is subjecting a system to an unreasonable load >while denying it the resources (e.g., RAM, disc, mips, interrupts, >etc.) needed to process that load. The idea is to stress a system to >the breaking point in order to find bugs that will make that break >potentially harmful. The system is not expected to process the >overload without adequate resources, but to behave (e.g., fail) in a >decent manner (e.g., not corrupting or losing data). Bugs and failure >modes discovered under stress testing may or may not be repaired >depending on the application, the failure mode, consequences, etc. >The load (incoming transaction stream) in stress testing is often >deliberately distorted so as to force the system into resource >depletion. > >2. Load testing is subjecting a system to a statistically >representative (usually) load. The two main reasons for using such >loads is in support of software reliability testing and in >performance testing. The term "load testing" by itself is too vague >and imprecise to warrant use. For example, do you mean representative >load," "overload," "high load," etc. In performance testing, load is >varied from a minimum (zero) to the maximum level the system can >sustain without running out of resources or having, transactions >suffer (application-specific) excessive delay. > >3. A third use of the term is as a test whose objective is to >determine the maximum sustainable load the system can handle. >In this usage, "load testing" is merely testing at the highest >transaction arrival rate in performance testing.
Subject: 16. Should we discuss bug tracking tools in this newsgroup? Discussing bug tracking tools is not within the original charter of comp.software.testing. The comp.software.config-mgmt newsgroup seems to have taken this topic under its wing. That group has a FAQ for problem management tools. See the "What other FAQs are relevant?" section for details. Danny Faught maintains a FAQ under is wing for defect-tracking tools at http://www.testingfaqs.org/t-track.htm.
Subject: 17. What kind of salary should a tester make? Salary information is available at <http://www.pencom.com/industry.html> and <http://www.computerjobs.com/salary_survey_search.cfm>. A nice summary of the various surveys that are available is at <http://www.ondaweb.com/sti/salary.htm>. John Tyson writes: > You could check the May '95 issue of Application Development Trends > magazine. The article "Testing moves from purgatory to profession" by > Linda Hayes [lghayes@metronet.com] has an excellent all-around > (non-technical) article on testing and does include some regional > salaries (albeit very general). John also encourages testing consultants to report their rates at Janet Ruhl's Computer Consultant's Resource Page - <http://www.realrates.com>. http://qualityprogress.asq.org/qp/nov97.html> > to order a back issue. Mark McWhinney writes: > You might not want to use the survey results. While they are helpful > for defining an organization's general pay scales, they are not very > useful on a case by case basis. There are too many factors involved > that may push a particular employee's base salary significantly above > or below the average or outside the range. > > For QA and test people, these surveys are even more a problem. QA and > test people are second-class citizens. They are paid less than their > counterparts in development. In part this is due to the fact that QA > and test people tend to be less senior and less experienced, therefore > deserving of less pay. I have no problem with that. However, > developers and testers with equivalent skill, education, and experience > are not paid the same. This is wrong....
Subject: 18. Where can I find sample test plans? See the following IEEE standards: 829-1983 IEEE Standard for Software Test Documentation 1008-1987 IEEE Standard for Software Unit Testing 1012-1986 IEEE Standard for Software Verification and Validation Plans 1059-1993 IEEE Guide for Software Verification and Validation Plans You can contact the IEEE in the US at 800-678-4333. Also, Requisite (800-732-4047) sells a Standards Pack with Microsoft Word Templates for 829, 830, and 1012 (part number 3100-05879). The Software Productivity Center (<http://www.spc.ca>, 604-662-8181) sells Word templates, including a test plan standard and a test plan template. Several people recommend Kaner's book, _Testing Computer Software_, for test plan samples. You'll find the ETET test plan in <ftp://ftp.ecs.soton.ac.uk/pub/etet/> under the path "doc/testplan". See <http://www.acomtech.com/testplan.html> for an adaptation of IEEE 829. And visit <http://members.tripod.com/~bazman/frame.html> for a sample system test plan. There is an outline of an IEEE based test plan on <http://www.itc.nrcs.usda.gov/sqa/tst_outline.html>. StickyMinds.com also have a template section for test documents at <http://www.stickyminds.com/swtest.asp?btntopic=tem&Function=Search &topcat=SWTST>. Information in this section was gleaned from posts by: Burt Gearhart, Wayne Woodruff, Brent Parsons, Castor Fu, Michele Mercer, Barry Dorgan, and others.
Subject: 19. What is the best tester to developer ratio? Reported tester:developer ratios range from 10:1 to 1:10. Jeremy L. Mordkoff writes: > There's no simple answer. It depends on so many things, I can't even > list them all. Amount of reused code, number and type of interfaces, > platform, quality goals, etc. > > It also can depend on the development model. The more specs, the less > testers. The roles can play a big part also. Does QA own beta? > Do you include process auditors or planning activities? Boris Beizer adds: > These figures can all vary very widely depending on how you define > "tester" and "developer". In some organizations, a "tester" is anyone > who happens to be testing software at the time -- such as their own. In > other organizations, a "tester" is only a member of an independent test > group. > > It is far, far, better to ask about the test labor content than it is > to ask about the tester/developer ratio. The test labor content, across > most applications is generally accepted as 50%, when people do honest > accounting. For life-critical software, this can go up to 80%.
Subject: 20. Conferences For further information about the conferences listed here and others, see the Conferences section of the SRM Hotlist at <http://www.cigital.com/hotlist/conferences.html>. Opinions in this section are by Boris Beizer. All of these conferences have published proceedings. Generally, materials presented at these conferences and in their proceedings lead the publication in regular journals (above) by about 2-3 years. 1. International Symposium on Software Testing and Analysis (ISSTA) theory, academic, leading edge practitioners (see 3b in the periodicals section.) Now transitioning to a summer conference that will alternate with ICSE. Sponsored by ACM and ACM's SIGSOFT. 2. International Conference on Software Engineering (ICSE). Spring, world-wide. Technical. Primary source after ISSTA for leading edge results. 3. Quality Week (QW). Annual, San Francisco Bay Area. Biggest Conference on Testing and QA. Typically 700+. Many vendors. Good balance between technical/theoretical and practitioners. Very broad base. Workshops. Sponsored by Software Research Institute, qw@soft.com, 800-942-SOFT, 415-957-1441. Quality Week Europe (QWE) is held in Brussels, Belgium in November. 4. Pacific Northwest Software Quality Conference (PNSQC). Annual, Portland Oregon, October. Definitely not a regional conference, despite the name. Very broad from technical/research to practitioners. A shade more academic and leading edge participation than quality week. 5. International Conference and Exposition on Testing Computer Software (TCS). Less technical and more practitioner/management than Quality Week. Smaller, 400+. Workshops. 17th annual conference was June 12-15, 2000, in Washington, D.C. Presented in cooperation with the ACM SIGSoft, ASQ Software Division, IEEE Reliability Society, and Software Technology Support Center. 6. Software Testing, Analysis, and Review (STAR). Software Quality Engineering. STAR East is in Orlando in May, and STAR West is on the east coast in the fall. About the same target as the International Conference (5) above. Comparable level and interest. Workshops. SQE 1-800-423-8378. 7. QAI International Software Testing Conference. More of a tutorial/workshop than a conference. Newbie orientation. Fall, Orlando, Florida. Quality Assurance Institute, 407-363-1111. 8. IEEE Compsac, world-wide, various locations. Fall. Typical IEEE technical conference with a substantial number of papers of direct interest to testing and QA. Not as many as IEEE Software Engineering, though. There are about a dozen other annual conferences with a substantial testing content: Korea, Japan, Germany, to name a few. In addition, there are about two-dozen "conferences", privately sponsored by various individuals and organizations. These are not real conferences in that there is no formal review process. Speakers are invited by the conference organizer. Often, a heavy tutorial content. These are usually small (under 100 participants, very few vendors). They range, depending on the organizer, from superb to fair-value, to outright rip-off. Sorry. I won't play 20 questions on this one. (end of Beizer's comments) Also of note: International Conference on Software Quality (ICSQ), sponsored by the Software Division of the ASQ. International Symposium on Software Reliability Engineering (ISSRE), sponsored by IEEE. International Software Assurance Certification Conference (ISACC), "The Software Risk Management Conference." ISACC is an international conference in an annual series to be devoted exclusively to the topic of software certification. International Conferences on Practical Software Quality / Testing Techniques (PSQT and PSTT). Two conferences a year, one in Minnesota and one further south. Managed by Software Dimensions. Software Testing Analysis & Review Europe (EuroSTAR). Software Testing Analysis & Review Latin American (LatinSTAR) in March 20-23, 2001. The IFIP 14th International Conference on Testing of Communicating Systems Berlin, Germany, March, 19th - 22nd, 2002 http://www.fokus.gmd.de/events/testcom2002
Subject: 21. Periodicals Comments in this section are by Boris Beizer 1. IEEE TSE (Transactions on Software Engineering). Monthly. The most prestigious journal for testing stuff. Volume 1 number 1 published the landmark article by Goodenough and Gerhart. Almost every issue since has had papers on testing and quality assurance. I have all the issues back to V1 #1. Much of that stuff is still relevant. 2. ACM TOSEM (Transactions on Software Engineering Methodology). Quarterly. Relatively new journal (1992). Has prestigious editorial board. Somewhat more theoretical than IEEE TSE. 3a. ACM SIGSOFT Software Engineering Notes. (from the Special Interest Group on Software Engineering). Monthly. Goes back to May 1976. This is the place where people publish stuff to stake out claims. It is not as strictly refereed as IEEE TSE or TOSEM, but some overview is provided to keep out the obvious crap. Important because there is a two or three year waiting list for publication in IEEE SE or ACM TOSEM. People who know this field read ACM SIGSOFT for advanced information on what will be published "officially" in IEEE SE and ACM TOSEM. You read an article and contact the author directly for more information. 3b. ISSTA conference proceedings. Annual. (Known as TAV in the past -- and possibly new name changes in the offing.) ISSTA is the theoretical/academic conference devoted to testing. This is the most prestigious place to publish new results in testing theory and software reliability theory. 4. IEEE Software. Six times a year. Rarely the latest stuff. More like surveys and overviews once a subfield has become established. Refereed, generally high standards. Mostly overviews, but occasionally new stuff. Additional comments from Scott Killops: You might mention in your blurb about 'IEEE Software' the regular "Quality Time" feature. This is wide ranging and generally quite useful. I've often thought that a collection of all of the "Quality Time" articles published to date would make a worthwhile book. 5. Software Testing, Verification, and Reliability. Quarterly. International journal published by John Wiley and Sons. Prestigious international editorial board. High quality original articles and excellent surveys. Comparable in scope to IEEE TSE and ACM TOSEM, but sharply focused on testing, verification, and reliability. ISSN 0960-0833. <http://www.csc.liv.ac.uk/users/mrw/stvr.main.html> <http://www.wiley.com/> info@qm.jwiley.com, cs-journals@wiley.co.uk John Wiley & Sons, Inc., Professional, Reference and Trade Group 605 Third Avenue, New York, NY 10158-0012 6. ACM Computing Surveys. Quarterly. Not specific to testing and QA, but contains the most prestigious survey articles in the field, typically only when a subfield is well established. The authors are usually authorities. Articles are long and comprehensive. When an ACM Survey on a topic appears, it usually means that the field has matured to the point where it is possible to write meaningful books. Volume 29, #4, December 1997, contains the most comprehensive survey of testing research ever published--"Software Unit Test Coverage and Adequacy", Hong Zhu, Patrick A. V. Hall, and John H. R. May, pages 366-427. The bibliography contains 221 entries. Anyone interested in understanding the technical/research literature of software testing should start with this survey. 7. Communications of the ACM. Monthly. Survey articles and overviews. Sometimes (rarely) original stuff. More academic and foundational/theory oriented than IEEE Software, but generally the same level. Doesn't publish too much on testing. 8. Cutter IT Journal (formerly American Programmer). Monthly. Cutter Information Corp., 37 Broadway, Suite 1, Arlingon MA 02474, phone 781-641-5118. Private journal with prestigious editorial board. Non-technical, philosophical and overviews. Management orientation. Essentially a good newsletter. <http://www.cutter.com/itjournal> 9. Software Testing & Quality Engineering. Bi-monthly. Software Quality Engineering, sqeinfo@sqe.com, 800-423-8378, fax 904-268-0733, phone 503-829-6806, <http://www.stqemagazine.com/>. Formerly called Software QA magazine and Software QA Quarterly. 10. The Software Practitioner. Newsletter, 6 times a year. Computing Trends, 1416 Sare Rd., Bloomington IN 47401. Robert L. Glass - Editor-in-Chief, rlglass@acm.org. Non technical, management, overviews. Not too much on testing. http://SoftwarePractitioner.com/ 11. Crosstalk: The Journal of Defense Software Engineering. 6 times a year, free. Software Technology Support Center, Ogden AFB UT 84056. 801-777-8057. Heavy DoD orientation. Practitioner/Management/ Philosophical. Broad based on software engineering, including IV&V. http://www.stsc.hill.af.mil/CrossTalk/crostalk.html> custserv@software.hill.af.mil. 12. Other journals. I'm sure I've offended several editors. There are journals concerned with testing, QA, and software engineering published in Japan, Korea, Germany, and Australia. Other journals, such as IEEE Transactions on Computers, IEEE Transactions on Reliability, ACM SIGPLAN, ACM SIGARCH, ACM SIGSIM, ACM SIGMETRICS, ACM OOPSLA, all publish material of interest to testing from time to time, especially insofar as hardware testing results and theory apply to software testing; also language issues vis-a-vis testing. (end of Beizer's comments) The American Society for Quality has begun publication of Software Quality Professional. This journal will span the body of knowledge for the Certified Software Quality Engineer. <http://www.asq.org/pub/> Commercial newsletters - the newsletters below are available for free to potential customers and some provide very good technical content. Quality Techniques Newsletter (QTN), formerly Testing Techniques Newsletter (TTN). Monthly, in electronic form only. Software Research, Inc., 415-957-1441, 800-942-SOFT. Send "subscribe" in the body of a message to qtn@soft.com. <http://www.soft.com/News/QTN-Online/> The Payne Report, http://www.cigital.com/paynereport/ Tejas Software Consulting Newsletter, emailed monthly. See <http://tejasconsulting.com/#news> for archives and to get a free subscription.
Subject: 22. Books Books relevant to software testers are listed in the Books section of the SRM Hotlist at <http://www.cigital.com/hotlist/publications-books.html>. See also <http://www.StickyMinds.com/> under "Books" for Software Testing and Software Quality Engineering's book list, formerly known as the SingleSource Guide. See the misc.technical.books FAQ <http://www.faqs.org/faqs/books/technical/> for contact information for technical book publishers. Comp.software.testing readers have recommended Reiter's Scientific and Professional Books <http://www.reiters.com> and Fatbrain.com <http://www.fatbrain.com/>.
Subject: 23. Bibliographic resources Comments by Boris Beizer. 1. ACM Computing Reviews. Monthly. The primary review journal. Most good papers on testing and quality assurance are eventually reviewed here. Extensive evaluations and criticisms by excellent reviewer pool. <http://www.acm.org/reviews/> http://www.acm.org/catalog/journals/107.html> Also, some books have large bibliographies that are very useful, such as those in Software Testing Techniques and Testing Computer Software.
Subject: 24. Organizations The organizations below are devoted to software testing or quality, or have special interest groups with such a focus. National and International organziations are listed first, followed by local organizations. National/International ---------------------- American Society for Quality 611 E. Wisconsin Ave. P.O. Box 3005 Milwaukee, WI 53201-3005 800-248-1946, 414-272-8575 <http://www.asq.org> The ASQ (formerly ASQC) administers the CQE (Certified Quality Engineer) and CSQE (Certifified Software Quality Engineer) designations. The ASQ press has many titles that may be of interest to software testers. Stolen from a conference announcement: > Founded in 1946, the American Society for Quality (ASQ) > provides a variety of professional, educational, and informational > programs reflecting the changing needs of business and industry. > Headquartered in Milwaukee, WI, ASQ has been the leading quality > improvement organization in the United States for almost 50 years. > The Society is composed of more than 135,000 individual members and > 1,000 sustaining members worldwide. ASQ's vision is to be the > world's recognized champion and leading authority on all issues > related to quality. The mission of ASQ is to facilitate > continuous improvement and increased customer satisfaction by > identifying, communicating, and promoting the use of quality > principles, concepts, and technologies. ASQ's Software Division > ASQ's Software Division is comprised of [sic] more than 5,000 members > including software quality professionals and software engineers > interested in applying quality principles to the field of software > development. The Division develops a software engineer > certification program, publishes a quarterly newsletter, works with > the Software Quality System Registration Committee on establishing > an ISO 9000 Software Registration Program in the United States, > interacts with other professional software organizations such as > the IEEE and the ACM (Association for Computing Machinery), and is > involved in many other activities. <http://www.asq.org/about/divtech/softdiv/swqweb.html> October is National Quality Month (in the U.S.), as decreed by Ronald Reagan in 1984. Quality Assurance Institute 7575 Dr. Phillips Blvd., Suite 350 Orlando Florida 32819 Phone: 407-363-1111 Fax: 407-363-1112 <http://www.qaiusa.com/> The QAI publishes The Journal of the Quality Assurance Institute monthly. They provide CQA certification. Steve Driscoll wrote: > The CQA is someone who has demonstrated a mastery of the Common > Body of Knowledge(CBOK) for ensuring the quality of systems (which > often include software). While ISO-9000 is considered part of the > CBOK, it's actually independent of the CQA designation (there is a > certification program for ISO auditors). > ... while my life is still pretty much the same as it was prior to > obtaining the CQA designation, I found that the reading and > studying to prepare for the exam extremely beneficial in my duties > as a Quality Analyst. I recommend it to anyone who is interested > in producing quality software (which includes developers as well > as quality analysts). Society for Software Quality > Vision - To be recognized as the Society for those interested > in promoting "quality" as a universal goal for software. > > Mission - Promote "quality" as a desirable attribute of software. PO Box 86958 San Diego, CA 92138-6958 619-297-1544 <http://www.ssq.org/> Software Process Improvement Network > The Software Process Improvement Network is comprised of > individuals who want to improve software engineering practice. The > individuals are organized into regional groups called "SPINS" that > meet and share their experiences initiating and sustaining software > process improvement programs. They meet annually at the SEPG > (Software Engineering Process Group) Conference, which is > co-sponsored by the SEI and a regional SPIN. (from the web page) <http://www.sei.cmu.edu/collaborating/spins/spins.html> British Computer Society Software Testing BCS Specialist Group > The Group has a number of full-day meetings per annum and is host > to the international conference on software testing - EuroSTAR. It > is producing a standard on component testing and has a sub-group on > Statistical Testing Methods. A full library of testing-related > material is available at meetings. > > It promotes awareness of testing, competence in the field, and > encourages research. (from the web page) <http://www.bcs.org.uk/siggroup/sg55.htm> Chinese Association for Software Quality (CASQ) This association is formed to share information and experiences amongst its members and people on software quality engineering. <http://www.casq.org> casq@newcastle.ac.uk Swedish Association for Software Testing (SAST) SAST is a non-profit organisation with members from all different companies that are interested in Software Testing within Sweden. The purpose of SAST to improve status of Software Testers and Software Testing, through informal education and exchange of information. Since this is a Swedish organization most activities are in Swedish. If you are interested you are welcome to contact Lise-Lotte Karlsson Boman (lise-lotte.karlsson-boman@posten.se) or Jan Warg (jan@warg.se). <http://public-www.pi.se/~sast> TestNet TestNet is a Special Interest Group in Software Testing with over 400 members of over 150 companies and organisations mainly in the Netherlands (although there are a few members from Belgium). The meetings and news-items are in Dutch. > The purpose of TestNet is the professionalization of testing IT products, > and an increase in the awareness and the importance of testing as a > profession in its own right. TestNet stimulates the exchange of > professional knowledge and practical experience amongst testers, and > stimulates research, from a scientific standpoint as well as from a > practical perspective. (from the web page) If interested or for information contact at secretaris@testnet.org <http://www.testnet.org> testnet@testnet.org CRIM Software Test Centre CRIM is a transfer and liaison centre with fifteen years' experience in leading-edge information technologies and computer application sectors. CRIM Software Test Centre (STC) helps big or small businesses and other kinds of organizations boost product quality, cut risk and optimize investments. Technologically, CRIM STC pinpoints the strengths and weaknesses of software applications based on pre-established criteria. CRIM STC may also recommend various ways of improving and optimizing the product testing and development processes. <http://www.crim.ca/en/index.epl?selec=a000&href=/en/ctl/index.htm> Local organizations ------------------- Most of the organizations above sponsor local organizations. Contact them to find local organizations near you. In this section you'll find local organizations that don't appear to have a national affiliation. The Software Quality Institute At UT (stolen from a conference announcement) > The Software Quality Institute (SQI) at The University of Texas at > Austin is a multidisciplinary partnership between UT-Austin and the > software and information systems industries in Texas. It is > recognized as a leading authority on and champion for software > quality. SQI was founded in 1993 for the purpose of strengthening > organizations to compete more successfully in global markets > through sponsorship of seminars and conferences. An advisory group > of 24 industry and academic representatives guides the Institute. > Its popular programs include: a 13 week long Software Project > Management Certificate Program, one- and two-day seminars offered > to the public and as in-company contract courses, our "Software > Quality Matters" quarterly newsletter that addresses issues of > concern relative to quality issues, sponsorship of monthly forums > for discussion of software issues (including the Software Process > Improvement Network (A-SPIN), the Austin Software Executives' Group > (ASEG), and the Austin Forum for Object-Oriented Technology > (AFOOT), and on-line services which include a Worldwide Web > Home Page and topical newsgroups. The web page is <http://www.utexas.edu/coe/sqi/>. The newsgroups seem to be reachable only at UT. For information on newsletter subscriptions, contact info@sqi.utexas.edu. Central Ohio Quality Assurance Association Mission Statement: To provide and promote continuous quality improvement information to software organizations in our communities. COQAA is organized to share state-of-the-art quality assurance methods, tools, and techniques among its members. P.O. Box 14191, Columbus, Ohio 43214-0191 contact: Jerry Fehribach, 614-447-3600, jfehribach@cas.org <http://www.coqaa.org> Australian Software Quality Research Institute Griffith University, Brisbane, Australia <http://www-sqi.cit.gu.edu.au/> Triangle Information Systems Quality Association (North Carolina) http://www.rtpnet.org/tisqa/ contacts - Dennis Brandel 72652.1146@compuserve.com David Wood DAVDWOOD@VNET.IBM.COM North Carolina Quality Leadership Foundation 800-207-5685 http://www.rtpnet.org/ncqlf/ New England Software Quality Assurance Forum (NESQAF) NESQAF is a regional non-profit society created by software quality professionals for software quality professionals. Each month, professionals from software companies and MIS organizations along with other industry experts exchange ideas, principles, and experiences on the latest quality assurance techniques and technologies. An email distribution list is available. Monthly meetings in Cambridge, MA. Contact - Alan Titelbaum, alan.titelbaum@tfn.com, 617-856-2467. <http://www.nesqaf.org/> Software Testing Institute Offers industry research, surveys, publications, online services and seminars to software development and testing professionals. A list of software testing product vendors, including web site and e-mail links, is available from our web site: <http://www.ondaweb.com/sti> contact - Susan Archer, Director (sarcher@metronet.com) 726 Loganwood Ave., Richardson, TX 75080 972-680-8507, fax 972-680-8905 Centre for Software Reliability The Centre for Software Reliability (CSR) is a research centre within the Department of Computing Science at the University of Newcastle upon Tyne; it conducts research on how to achieve improved levels of dependability from computing systems. csr@newcastle.ac.uk <http://www.csr.ncl.ac.uk/> Seattle Area Software Quality Assurance Group (SASQAG) Purpose: To Promote Professional Software Quality Practices <http://www.sasqag.org/>
Subject: 25. Contributors Thanks go to the people who have contributed to the FAQ in various ways, including the team of assistant FAQ maintainers - Chris Petrov, and Richard de Graaf, and Danny Faught. Email addresses for some of those mentioned herein are listed below. Boris Beizer - bbeizer@acm.org Bob Binder - rbinder@rbsc.com Richard de Graaf - r.dgraaf1@chello.nl Barry Dorgan - bazman@bigfoot.com Steve Driscoll - sdriscol@oclc.org Sigrid Eldh - Sigrid.Eldh@ehpt.com Danny Faught - faught@tejasconsulting.com Rick Hower - rickhower@earthlink.net Scott Killops - Scott_B_Killops@ccm.jf.intel.com Jennifer Larsen - Jennifer_Larsen@datatel.com Brian Marick - marick@testing.com Mark McWhinney - msm@powertest.com Don Mills - donmil@voyager.co.nz Jeremy L. Mordkoff - mordkoff@acm.org Charles Nichols - nichols@cse.ogi.edu Chris Petrov - petrovchris@usa.net Bret Pettichord - bret@pettichord.com Raymond Rivest - rrivest@crim.ca Laura Rose - laura@PACorp.com John Tyson - jmt@acm.org

User Contributions:

Comment about this article, ask questions, or add new information about this topic:

CAPTCHA




MultiPage

[ Usenet FAQs | Web FAQs | Documents | RFC Index ]

Send corrections/additions to the FAQ Maintainer:
raymond.rivest@crim.ca (Raymond Rivest)





Last Update March 27 2014 @ 02:12 PM